Switching Temporary Teachers for Semi-Supervised Semantic Segmentation
Authors: Jaemin Na
Jung-Woo Ha
Hyung Jin Chang
Dongyoon Han
Wonjun Hwang
Venue: NeurIPS 2023 poster (2023)
Keywords: Semi-supervised Learning
Semantic Segmentation

Abstract: The teacher-student framework, prevalent in semi-supervised semantic segmentation, mainly employs the exponential moving average (EMA) to update a single teacher's weights based on the student's. However, EMA updates raise a problem in that the weights of the teacher and student are getting coupled, causing a potential performance bottleneck. Furthermore, this problem may become more severe when training with more complicated labels such as segmentation masks but with few annotated data. This paper introduces Dual Teacher, a simple yet effective approach that employs dual temporary teachers aiming to alleviate the coupling problem for the student. The temporary teachers work in shifts and are progressively improved, so consistently prevent the teacher and student from becoming excessively close. Specifically, the temporary teachers periodically take turns generating pseudo-labels to train a student model and maintain the distinct characteristics of the student model for each epoch. Consequently, Dual Teacher achieves competitive performance on the PASCAL VOC, Cityscapes, and ADE20K benchmarks with remarkably shorter training times than state-of-the-art methods. Moreover, we demonstrate that our approach is model-agnostic and compatible with both CNN- and Transformer-based models. Code is available at https://github.com/naver-ai/dual-teacher.

Switching Temporary Teachers for Semi-Supervised Semantic Segmentation
Jaemin Na', Jung-Wooo Woo Ha", Hyung Jin Chang?, Dongyoon Han Wonjun Hwang Ajou University, Korea', NAVER AI Lab², University of Birmingham, UK3
osia1460ajou ou. ac.kr, jungwoo. ha@navercorp. com, h.j j. changebham ac uk dongyoon dongyoon. hww.oownooaaaaa..croo..o.oo com, wjhwangeajon ac. kr
Abstract
The teacher-student framework, prevalent in semi-supervised semantic segmenta- tion, mainly employs the exponential moving average (EMA) to update single teacher's 's weights based on the student's. However, EMA updates raise prob- lem in that the weights of the teacher and student getting coupled, causing a potential performance bottleneck. Furthermore, this problem may become more severe when training with more complicated labels such as segmentation masks but with few annotated data. This paper introduces Dual Teacher, simple yet effective approach that employs dual temporary teachers aiming alleviate the coupling problem for the student. The temporary teachers work in shifts and are progressively improved, so consistently prevent the teacher and student from becoming excessively close. Specifically, the temporary teachers periodically take turns generating pseudo-labels to train a student model and maintain the distinct characteristics of the student model for each epoch. Consequently, Dual Teacher achieves competitive performance on the PASCAL VOC, Cityscapes, and ADE20K benchmarks with remarkably shorter training times than state-of-th art methods. Moreover, we demonstrate that our approach compatible with both CNN- and Transformer- based models. Code is available at htops://githut com naver- ai/dual- teacher..
aggnosic and
Introduction
Semantic segmentation has progressed along with the advance of deep neural networks as critical component of computer vision tasks for visual understanding [8, 9, 55]. However, the prevailing approaches primarily based on supervised learning methods and still require laborious and time- consuming manual annotations at the pixel-level. To alleviate this intrinsic challenge, semi-supervised semantic segmentation has recently come into the limelight requiring only small amount of labels.
crucial challenge successful semi-supervised learning is how to acquire reliable and consistent labels from unlabeled data, particularly in more demanding tasks like semi-supervised segmentation. Pscudo-labeling [22], the most popular approach, assigns the off-the-shelf model's class predictions as labels for unlabeled data for training. Another seminal work, a teacher-student framework [36], is widely adopted as it provides robust results numerous semi-supervised learning tasks [41, 26, 13]. Despite the tremendous success, it is problematic that the weights of the teacher and student models are inevitably coupled as the teacher weights are updated using an exponential moving average (EMA) of the student model weights. This notable issue called the peacher-studens coupling problem [17] and causes the performance bottleneck for the student model because the EMA teacher exhibits similar performance and perspective to the student model in traditional teacher-student framework [36).
Furthermore, when the student model produces biased predictions over a few samples, training
'Corresponding authors.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Student
Studint
Student
JEMA
Label
Label
LEMA
EMA
X
Xw
Teacher
Student #2
Teacher #1
Teacher #2
(a) Teacher-Student
(b) Dual Student
(c) Dual Teacher (ours)
Figure I: Semi-supervised semantic segmentation frameworks. (a) A teacher model generates pseudo-labels train student model and is updated as exponential moving average (EMA) of the student madel weights. (b) Dual Student involves two distinct student models that also serve as teachers to each other. (c) Our Dual Teacher employs dual temporary teachers that switch alternately during training.
with a single EMA teacher is prone to accumulate errors and provide incorrect pscudo-labels to the student model, which causes the irremdiiable misclassification (i.e, confirmation bias [1]. One straightforward approach [17, 16, 6] to mitigate the issues is to employ explicit ensemble models with different initializations instead of counting
single student model. However, incorporating another deep neural network is not preferred due to the heavy computational demands and the inherent challenge of achieving scalability.
In this paper, we propose a novel Dual Teacher framework that comprises a single-student model and dual temporary teacher models to alternatively handle the coupling problem between the teacher and student network. Our framework, which is depicted in Figure 1, discourages the teacher and student models from converging and becoming similar to each other as training progresses. We feed the student model with different strongly augmented inputs for each epoch in training, thereby encouraging the construction of a student model with distinct characteristics at every epoch. At the same time, as we change the augmentation applied to the student model, we also switch the temporary teacher models assigned to it. This ensures that the temporary teacher models keep retaining the characteristics of the student model for each epoch. Additionally, we improve the robustness of the student model by enforcing consistent predictions between fractional models of the student model and the full teacher model. These sub-models contribute to an implicit ensemble, which is differentiated from the explicit ensemble.
Ultimately, our approach achieves competitive performance with state-of-the-ee art methods [41, 15, 24] while requiring much shorter training times and fewer parameters. We demonstrate the superiority of Dual Teacher under the semi-supervised semantic segmentation protocol using public benchmarks, including PASCAL VOC [9] and Cityscapes [8]. We further prove the scalability of method using the large-scale dataset, ADE20K [55], and have made our experimental protocols available to the public. We conduct extensive experiments only with ResNet-500-101 but also with Transformer- based SegFormer [43].
2 Related Work
Semi-supervised Learning. Semi-supervised learning (SSL) methods benefit significantly from consistency regularization [2, 31], which encourages a model to produce the same prediction when the input is perturbed, and pseudo-labeling [22] which uses model's prediction
label to train against.
Recently, semi-supervised learning methods [2, 3, 32] have shown that consistency regularization and pseudo-labeling can work together harmoniously. Most of all, FixMatch [32] generates predicted labels from augmened unlabeled images and exploits them as pscudo-labels when strongly augmented versions of the same images are fed into the identical model. This simple yet effective approach has been widely used in recent studies when leveraging unlabeled data in SSL.
Semi-supervised Semantic Segmentation. Recent success in semi-supervised semantic segmen- tation approaches can be attributed to the effective incorporation of consistent regularization and pseudo-labeling For example, CPS [6] and GCT [16] construct two parallel segmentation networks with the same structure and impose consistency on two networks with different initializations for the same input. Instead of using two different networks, which is burdensome to train, the latest works [41, 26, 15] adopt teacher-student framework which only the student model is trained,
and the teacher model is updated through the EMA of the student model. On top of this simple

---

framework, GTA-Seg [15], PL [41], and ReCo [24] introduce additional representation heads, and GTA-Seg further attaches a teacher assistant model the same size as the student model.
On the other hand, simple and effective data augmentations [49, 29, 50] that yield label-preserving perturbations have played critical role in consistency regularization for semi-supervised semantic segmentation [6, 41, 24, 46]. Moreover, AEL [13] and AugSeg [53] contribute to performance improvement by introducing more advanced adaptive data augmentation techniques. However, unlike the previous studies, we achieve comparable performance by leveraging basic augmentations within our novel stacher-ttudent framework without any lateral networks or sophisticated data augmentations. Meanwhile, PS-MT [26] presents an ensemble strategy that leverages multiple teachers to enhance the segmentation accuracy of unlabeled images. In contrast, rather than ensembling them, we employ the multiple teacher models in sequential and independent manner. This approach ensures that each teacher model, with its diverse characteristics, distinctly contributes to the student model.
Ensemble Learning, Network ensembling [19] gains popularity for improving model predictions by combining the outputs of diverse models. Recent works [21, 52, 42] have successfully improved performance by model ensembling; that is, the individual models are trained to maximize their diversity so that the ensemble models are less prone to overfitting. Although these straightforward explicit ensemble approaches have shown significant performance improvements, they share the downside of bearing burden of computation and memory cost. The impact of the explicit ensembling has also been indirectly exploited to train a single network through dropout [33], dropconnect [40], or stochastic depth [14] by activating only subset of the network, and thus the complete network can be seen as an implicit ensemble of trained sub-networks [34, 37]. Of particular interest is that Temporal Ensembling [20] forms consistent prediction of the unlabeled data using the outputs of the network being trained at different epochs under different regularization and input augmentations. Taking step further with this approach, Mean Teacher [36] updates the teacher model with EMA weights of the student model instead of sharing weights with the student model an ensembling perspective. Although this teacher-student framework has exhibited a significant impact, one critical problem is the coupling problem, wherein tightly coupled weights deteriorate performance. Alternatively, we address this problem by introducing dual temporary EMA teachers utilizing temporary and implicit ensemble perspectives.
3 Method
3.1 Preliminary
Teacher-student framework. The teacher-student framework [36] is a popular approach in semi- supervised learning that leverages unlabeled data to enhance model performance. As described in Figure (a), it comprises teacher and student model, where each model includes encoder and decoder. In supi-super semantic segmentation, the student model is trained with a few labeled
data as follows:
B:
H-W
Coup
Σ Cee Ceclpig. 1=1
B
H.
W
where B, denotes the number of labeled images æ!y with resolution of H. x W in training batch, and the Len is a pixel-wise cross-entropy loss applied to every pixel j on i-th labeled image. The ply indicates prediction for the labeled images of the student model, and v is ground-truth labels of corresponding labeled images. In this work, we include this supervised loss for the student model, and unsupervised loss for unlabeled data is addressed in Section 3.2.
The teacher model's parameters 0, are updated with an EMA of the student model's parameters 0g as:
0: - a02 + (1 a)0,,
(2)
where a is smoothing coefficient hyperparameter.
Semi-supervised semantic segmentation. semi-supervised semantic segmentation, sophisticated data augmentations and network architectures are essential for handling pixel-level labels with in- creased degrees of freedom. However, due to the limited number of training instances compared to the abundance of mask labels, more diversified student model should be involved than semi-supervised
3

---

classification. Furthermore, we argue that incorporating cumbersome data augmentations [13, 53] and additional heads [41, 15, 24] may impair efficiency, limiting the full potential of its benefits.
Revisiting Temporal Ensembling. Our work is inspired by Temporal Ensembling [20], which pursues to enjoy indirect ensemble (i.e., implicit ensemble) effect within a single model by ensembling predictions obtained from different regularizations for each epoch. This training approach accumulates the network's predictions into ensemble predictions after every training epoch. This process has been devalued for its slow training pace due to updating only once per epoch; however, we revalue it as a favorable method inject diversity into our student model to address the coupling problem.
3.2 Dual Temporary Teacher
In the context of semi-supervised semantic segmentation, our goal is to mitigate the coupling problem [17] caused by the single EMA update routine employed in the conventional teacher-student framework. Among the critical factors [17, 32] for improving the teacher model, one we mainly take notice of is the diversity of the student models, as noted in prior works [17, 25, 44]. We strive to prevent the teacher and student models from becoming too similar to each other training goes on by establishing a virtuous cycle in which the diversity of the student model enhances the teacher model. In turn, the improved teacher preserves students' diverse perspectives and effectively guides the student model. Since the student updates each EMA teacher, the teachers get enhanced as simultaneously diversifying the student due the ensemble effect [19, 36]. The strengthened teacher model can subsequently provide more refined supervisory signals to the student model.
From this perspective, we introduce simple yet effective framework Dual Teacher: an approach that temporary EMA teacher models alternate in generating pseudo-labels to guide a student model; concurrently, the student model updates the weights of the teacher models with exponential moving average of its own weights. We claim that introducing additional EMA teachers facilitates the diversification of students by providing distinct and varied supervision. As described in Figure 1 (c), Dual Teacher consists of a pair of teacher-student models, where the two temporary teacher models are switched every epoch during training to teach the single student model.
Similar to prior studies [32, 26, 41], we adopt groogly-augmentee images as inputs for the student model and weakly-augmentte images for the teacher models to ensure reliable pscudo-labels. How- ever, instead of the traditional teacher-student framework that relies on a single permanent teacher model, introduce dual temporary teachers alternately activated per epoch, providing diversi- fied guidance to the student model. The temporary EMA teachers capture the evolving temporal knowledge of the student model, performing like a temporal ensemble at different time steps.
Strong augmentation pool. To diversify the student model more, we go beyond leaning solely on the EMA teacher model, albeit with alternating dual teachers. To further ensure diversity, we provide variety by changing the type of strong augmentation applied to the student model when shifting the temporary teacher models per epoch. We achieve this by building predefined but non-deterministic pool of strong augmentations, including both fine class-level and coarse region-level augmentations (i.e., ClassMix [29] and CutMix [49])' Each training epoch randomly samples augmentation from the pool, constraining that the consecutive epoch does not use the same augmentation.
Updating student and teachers. In nutshell, the use of temporary teacher models helps to introduce diversity into the student model, while the diverse characteristies acquired by the student model contribute to the enhancement of the teacher models. Formally, the objective functions for the student model are defined as follows:
0, - 0g + ÖCoup + Nu Cu
B,
1
H-W
Lunsup
Σ j=1
Lee Cee {.))
(3)
00,
B
H W
where B,, denotes the number of unlabeled images in a training batch, an image size of H W. jth pixel on the ith image, the student model's prediction pž, with unlabeled input applied strong augmentation, and the corresponding pseudo-label 27, from the teacher models. The X, scalar hyper-parameter to adjust the unsupervised loss weight.
'The coarse Seiion-level augmentation helps the model exploit contextual information, while the fine class-level augmentation enables differentiation between objects based their unique properties.

---

One of the temporary teachers is activated alternately at each epoch and retains the student model's characteristies through the EMA weights. The k-th temporary teacher & {1, ....,,,, &n J is alternately switched each epoch; the parameters of k-th temporary teacher Of are updated through EMA based on the student parameters 0, by Eq. (2).
3.3 Implicit Consistency Learning
We introduce another implicit ensemble learning from the viewpoint of consistency regularization to enhance our student model. Motivated by [14, 38, 10]. we encourage a subset of layers to be active in the student model and all these sub-models to make consistent predictions. In contrast to the previous studies [38, 2] that enforce consistent predictions between the full model and sub-models within the same model, we impose the consistency between the sub-models of the student model and the full teacher models. We construct the sub-models within the student model via versatile stochastic depth [14], applicable from CNN models to transformer-basee models. Therefore, the student model is trained through the following objective functions:
8
H.W Σ
Loone
Lee (..)
H.W
where 0, denotes the parameters of sub-models of student model with drop rate т, and B is the number of images in a training batch. Note that we only apply weak augmentations to inputs for both teacher and student models, unlike in Eq. 3, where we feed strong augmented inputs to the student model and weak augmented inputs to the teacher model, respectively. Finally, this loss cooperates with the Lunsup to update the student model.
4 Experiment
In this section, we demonstrate the efficacy of our proposed method by conducting comprehensive comparisons with state-of-the- approaches on various publicly available benchmarks. Furthermore, we provide additional ablation studies to justify the significance and impact of our method.
4.1 Setup
Datasets. We evaluate our method on three public benchmarks in semantic segmentation, including the PASCAL VOC [9], Cityscapes [8], and ADE20K [55] datasets.
PASCAL VOC [9] is widely-used semantic segmentation benchmark that consists of 20 object categories in the foreground and a background category. It includes 1,464 and 1,449 images in the training and validation set, respectively. To evaluate our method, we follow recent protocols [46, 45, 41] and validate it on both the original high-quality set and the blended set, which consist of 10,582 training images. For the blended set, we follow the common practice [47, 13, 6] and make use of the SBD [1]] augmented set, comprising 9,118 training images with their corresponding annotations.
Cityscapes [8] is designed for semantic understanding of urban street scenes, including 30 classes, of which only 19 classes are used for scene parsing evaluation. There are 5,000 finely annotated images and 20,000 coarsely annotated images. The fine-annotated images split into training, validation, and test sets with 2,975, 500, and 1,525 images, respectively. Following the prior works [13, 45, 41], evaluate method 1/1e, 1/s, 1/4, and 1/2 label partitions.
ADE20K [55] is a large-scale scene parsing dataset that covers 150 object and stuff categorics. consists of 25,574, 2,000, and 3,000 images in the training, validation, and test set, respectively. Due to the lack of standardized label partitions for semi-super segmentation scenarios, we randomly split the whole training set and construct 1/32, 1/16, 1/8, 1/4, and 1/2 label partitions. We have publicly released these partitions on our GitHub repository.
Model architectures. We leverage various architectures to demonstrate the versatility of our model- agnostic approach. We adopt DeepLabv3+ [5] based on ResNet-50/-100 backbones [12] for PASCAL VOC, and ResNet-101 for Cityscapes. In addition, we employ DeepLabv2 [4] based on ResNet-101 for unsupervised domain adaptation task. Aside from the CNN backbones, we further validate our method with transformer-be basedd architecture SegFormer [43] with the MiT-B1.
5

---

Table 1: Comparison of mloU (%) with state-of-the-art methods PASCAL VOC 2012 under different partitions. Labeled images are sampled from the original high-quality training set. methods are based on DeepL.aby3+ with ResNet-50 backbone. We also report the total amount of trainable parameters in each method.
Method
#Params
1/16 (92)
1/8 (183)
1/4 (366)
1/2 (732)
Full (1,464)
Supervised-only
43.6M
44.03
52.26
61.65
66.72
72.94
PscudoSeg [56] (ICLR'21]
43.6M
54.89
61.88
64,85
70.42
71.00
PC Seg [54] [ICCV'21]
43.6M
56,90
64.63
67.62
70.90
72.26
AugSog [53] [CVPR 23]
43.6M
64.22
72.17
76.17
77.40
78.82
UniMatch [46] [CVPR'23)
43.6M
71.9
72.5
76.0
77.4
78.7
Ours
43.6M
70.76
74.53
76.43
77.68
78.15
Table 2:
Comparison mIoU (%) with state-of-the-art methods on PASCAL VOC 2012 under different
partitions. Labeled images are sampled from the blender training set, which consists of 10,582 samples in total. All methods are based on DeepLabv3+ with ResNet-101 101 backbone.
Method
#Params 1/16 (662)
1/8 (1,323)
1/4 (2,646)
1/2 (5,291)
Supervised-only
62.6M
67.87
71.55
75.80
77.13
MT [36] [NcurIPS' 17]
62.6M
70.51
71.53
73.02
76.58
CPS [6] [CVPR'21]
125.2M
74.48
76.44
77.68
78.64
AEL [13] [N
62.6M
77.20
77.57
78.06
80.29
U [41] [CVPR'22]
64.5M
77.21
79.01
79.30
80.50
GTA-Seg [15] [NeurIPS'22]
127.IM
77.82
80.47
80.57
81.01
PCR [45] [NeurIPS'22]
64.5M
78.60
80.71
80.78
80.91
Ours
62.6M
78.82
81.19
81.03
80.62
Implementation details. For the training PASCAL VOC, we employ the stochastic gradient descent (SGD) optimizer with an initial learning rate of 0.001 and weight decay of 0.001. We set batch size to 16 and run 80 epochs. Following the previous protocol [46], we resolution of 321 x x321 for the original high-quality training set while configuring the blender set to 513x513. For Cityscapes, we also use SGD optimizer with initial learning rate of 0.01, weight decay of le-4, batch size of 16, crop size of 769x769, and total training epochs of 200. For the training on the ADE20K, and poly LR scheduler with factor of 1.0. For a fair comparison, we validate the state-of-the-ar methods [6. 41, 15, 24] using their official training codes with model architectures of SegFormer without any modifications. We train the methods with batch size of 8 and crop size of 512 x 512 for the MiT-BI backbone. Following the previous work [41], we progressively increase EMA weights up to 0.99 across all experiments.
image
follow the recipe of SegFerrrr [43, 7] and set the learning rate of 6e-5
Evaluation. In all experiments, evaluate the performance of semantic segmentation. For Cityscapes, we apply online hard example mining loss the supervision loss and sliding window evaluation following the prior works [41, 53).
the mean of Intersection over Union (mloU) metric to
4.2 Comparison with State-of-the-Arts
On PASCAL VOC. In Table 1 and Table 2, we present the comparison results on validation set under different backbone networks (i.e., ResNet-50/101) to validate the performance of our method. Table 1 demonstrates the effectiveness of our approach, consistently surpassing the performance of the baseline with the ResNet-50 backbone. Compared to the current sfate-of-the-at art methods, AugSeg and UniMatch, our method yields superior results in three out of the five partitions. Except for the UniMatch [46], other studies have a higher training resolution than ours: 512 vs. 321. More details provided in the supplementary material.
Table 2 shows the comparison results with the state-of tfie--ofthe-asd art methods based on ResNet- 101. Com- pared to the baseline, our method achieves performance improvement of +10++0.99 +10.95% +9.64% +5.23%, and +3.49% for the label partitions !/16, 1/8, 1/4, and 1/2, respectively. While a few numbers achieve marginal improvements over the state-of-the-an approaches. it is noteworthy that our simple approach competitive with more complicated methods, such as GTA-Seg [15] and PCR [45], which rely on additional networks, and AEL [13], which employs more sophisticated augmentations.

---

Table 3:
Comparison of mloU (%) with state-of-the-art methods Cityscapes under different partitions. All methods are based on DeepLabv3+ with ResNet- backbone.
Method
#Params
1/16 (186)
1/8 (372)
1/4 (744)
1/2 (1,488)
Supervised-only
62.6M
65.74
72.53
74.43
77.83
CPS [6] [CVPR'2] 21]]
125.2M
74.72
77.62
79.21
80.21
AEL [13] [NcurIPS'21]
62.6M
75.83
77.9
79.01
80.28
PS-MT [26] [CVPR'22]
62.6M
76.9
77.6
79.09
U'PL [41] [CVPR
64.5M
74.9
76.48
78.51
79.12
PCR [45] [NeurIPS' 222]
64.5M
73.41
76.31
78.4
79.11
UniMatch [46] [CVPR'23]
62.6M
76.6
77.9
79.2
79.5
Ours
62.6M
76.81
78.4
79,46
80.52
Table 4: Comparison of mloU (%) with state-of tatte-of-the of-the-art methods on ADE20K under different partitions. All results are reported with single-scale inference SegFormer with MiT-B1 backbone. f: Results of our re- implementation according each official code.
Method
#aarams 1/32 (631) 1/16 (1,263)
(2,526)
1/4 (5,052)
1/2 (10,105)
Supervised
13.7M
15.15
22.32
25.90
29.53
32.57
CPSf [6] [CVPR'2] 121]
27.5M
18.79
22.27
27.89
32.41
36.89
U'PLI [41] [CVPR'22]
15.0M
18.54
23.64
28.11
33.52
36.01
GTA-Segi [15] [NeurIPS'22]
28.8M
19.52
23.09
26.86
27.85
29.99
ReCof [24] [ICLR'23 22]
14.4M
21.11
23.70
27.74
30.17
35.54
Ours
13.7M
22.60
26.01
30.60
33.74
37.30
On Cityscapes. Table 3 reports comparison results of our method against several existing methods on the Cityscapes validation set. Our method yields improvements of +f1.077%% 1..00.%%%..774. +5.87% +5.03%, and +2.69% over the baseline in the 1/16, 1/8, 1/4, and 1/2 label partitions, respectively, demonstrating its effectiveness, particularly the scarce-label setting. Notably, our method outperforms CPS [6] on 1/16 partition, achieving over 2% higher mloU despite using only half as many trainable parameters. In addition, we achieve superior performance across all the label partitions compared to AEL [13], method that introduces more advanced augmentations adaptive Copy-Paste and adaptive CutMix.
On ADE20K. We further validate the effectiveness of approach using SegFormer [43] in Table which unifies Transformer encoder and a lightweight multi-layer perception (MLP) decoder on ADE20K. Since there no standard label partitions of ADE20K for semi-supervised semantic segmentation, we construct partitions by sampling randomly from the entire training data. For fair comparison, we re-implement the current state-of-the methods according their official codes under the same network architecture (i.e., SegFormer with MiT-BI backbone). Our method consistently outperforms the supervised baseline by +4.86%, +3.69%, +4.70%, +4.21% ,, and +4. 73% in the 1/52, 1/16, 1/s, 1/4, and 1/2 partitions, respectively. It is noteworthy that we achieve competitive performance despite using fewer trainable parameters compared to other methods.
4.3 Empirical Studies
Analysis of diversity. To demon-
strate the coupling problem, we report the prediction distance between the teacher and student models. Here, we
3.6
0.07
Single Teacher
Dual Naachr
The
Teacher #1
Peddtton dibace
0.65
0.04
compute the mean square error of pre- dictions between two models to mea- sure prediction distance. As depicted in Figure 2 (a), the predictions of the EMA teacher are very close those of the student model in the conven- tional teacher-student framework (de- noted Single Teacher). On the other hand, we observe that Dual Teacher consistently strives to maintain the disparity between the pre- dictions of the teacher and student models during training. We argue that this approach allows the
20
40
60
60
C0W
d5g
Eottile
cat
shoep
Trairinng apochs
Category
(a) Prediction distance
(b) Class-wise IoU
Figure 2: Results of Dual Teacher in training process on 1/16 parti- tions of the PASCAL VOC blender set using ResNet-50.
7

---

Table S: The performance relative the num- ber of teacher models and augmentations on 1/1 splits of the PASCAL VOC blender set.
Table 6: Analyzing the diversity of supervisions using prediction distances between teacher-student models. A larger STD denotes a more diversified teacher's 's supervision.
Method
Single
Dual
Triple
Epoch|
1-5
6-
11-15
16-20 20
Aug 1
73.63
75.34
75.30
Single
0.0018=000010 0.0016+0.0005
0.00124:.00044
0.0012+:0.004
Aug 2
74.21
77.00
76.56
Dual
0.0378=0.0209
0.0249#0.0095
0.0181+0.0074
0.0199=0.0059
Aug 3
74.49
76.25
75.94
Triple
0.0425=0.0224
0.0109+0.0020 0.019440.0040 0.0222=000024
Table Performance analysis of different teaching strategies on PASCAL VOC 2012 across various partitions. Laheled images are sampled from the blender set and all methods are based on Deepl DevpLabb3++ with ResNet-50..
Method
1/16 (186)
1/8 (372)
1/4 (744)
1/2 (1,488)
Switching teachers
77.00
79.23
78.94
80.18
Ensembling teachers
73.25
75.99
76.41
77.34
student model to become more generalized, preventing it from learning towards a particular bias. This result supports our conjecture that the EMA teacher model is tightly coupled with the student model, which may incur performance degradation.
In Figure 2 (b), we investigate the distinct characteristics of the two temporary teacher models in Dual Teacher. Here, we select the top-5 categories with the most significant differences in class-wise IoU between dual temporary teacher models at the 10th epoch. Although the temporary teacher models have similar mloU of 66.48% and 66.01 in the middle of the training process, we observe significant performance differences across each category. This observation indicates that our dual temporary teachers produce distinct supervisions and teach the student model complementarily.
Impact of the number of teachers and augmentations. In this study, we investigate our method by extending the number of temporary teachers from Dual Teacher to Triple Teacher while also expanding the number of augmentations. In Table 5, the first row shows the performance according to the number of teacher models by applying only ClassMix [29]. which is specialized for semantic segmentation. We observe that employing two or more temporary teachers outperforms a single permanent teacher. This finding highlights the effectiveness of our approach that leverages teacher switching. In the second row, we begin to add CutMix [49] to our augmentation pool and utilize two different augmentations. We observe that the Dual Teacher achieves 2.79% improvement over the Single Teacher, and the Triple Teacher shows saturated performance compared to the Dual Teacher. confirms that enriching strong augmentation types in addition to teacher switching is beneficial. In the last row, we incorporate three different augmentations by appending MixUp [50]. In this case, the results do not significantly outperform the counterparts of using two augmentations, regardless of the number of teachers. Ultimately, draw a conclusion that leveraging both dual teachers and augmentations in our framework is the most effective.
Dual teachers vs. Triple teachers. In Table 6, we present comparative analysis of the prediction distances between teacher and student models across several frameworks. Notably, the Dual Teacher exhibits more pronounced fluctuations in distances, as evidenced by its higher standard deviations (STDs). Although the trend shows a similar decline for both Dual Teacher and Triple Teacher across epochs, we observe consistently lower STD for Triple Teacher. This suggests that Dual Teacher demonstrates greater variability prediction distance across epochs, often varying more than double that of the Triple Teacher. We posit that since the three augmentation methods we used are probably not orthogonal, teacher models may tend to transfer similar characteristics to the student model due to the diminished diversity among teacher models. We presume the diversity inherent in teacher models is intrinsically connected to student model's generalization capability, and our empirical backup supports it (see Table 5). Given the noticeable variations in prediction distances, we contend that teacher diversity is instrumental in amplifying the efficacy of Dual Teacher. As a result, teacher diversity is highly likely to contribute enhanced performance for Dual Teacher, preventing the student model from learning towards a specific teacher bias.
Switching teachers vs. Ensembling teachers. One prevalent approach to leveraging dual teachers involves the ensembling of teacher models, which then simultaneously guide student model [26]. In this teacher ensembling strategy, both teacher models consistently guide the student model throughout
8

---

Table 9: Comparison of mIoU (%) with the previous unsupervised domain adaptation methods Synthetic-
to- Ro-R: GTA5 -itysscpss All methods are evaluated on Deeplabb22 with ResNet-101.
Method
Road SW Build. Wall Fence Polc
TL
Vog.
Torr.
Sky
PR
Rider
Car Truck
Bus Train Motor Bike jmloU
Source-only 63.31 15.65 59.39
8.56
15.17
18.31
26.94
15
80.46
15,25
72.97
51
.40
17.67
59.68
28.
19
33.07
3:53
23.21
16.73
32.85
LTIR [18]
92.9 55.0
85.3
34.2
31.1
34.9
40.7
34,0
85.2
40.1
87.1
61.0
31.1
82.5
32.3
42.9
0.3
36.4
46.1
50.2
FDA [48]
92.5
53.3
82.4
26.5
27.6
36.4
40.6
38.9
82.3
39.8
78.0
62.6
34,4
84.9
34.1
53.1
16.9
27.7
46.4
50.45
PIT [27]
87.5
43.4
78.8
31.2
30.2
36.3
39,9
42.0
79.2
37.1
79.3
65.3
37.5
81.2
36.0
45.6
25.7
23.5
49.9
50.6
IAST [28]
23.8
57.8
85.1
39.5
26.7
26.2
43.1
34.7
84.9
32.9
88.0
62.6
29.0
87.3
39.2
49.6
23.2
34.7
39.6
51.5
Single Teacher(93.34 56.81
56.81
86.28
34.63
32.27
32,99
39.21
43.99
86.72
45.11
85.81
63.05
21,28
86.76
38.66
50.05
0.0
13.8
33.7
[49.71
Dual Teacher 93,93 56.88
86.47 35:49
31.57
35.12
37.78
43.43
86.02
43.54
88.06
62.14 28.49
88.59
50.14
54.2
0.18
35.75
49.82
(3303
all epochs. Therefore, this approach could limit the diversity of the student model, leading to weaker teacher models updated via EMA from the student model. On the other hand, our approach is differentiated by using a single switchable teacher model to instruct the student model in each epoch with more diversity. This ensures that our method produces uniquely distinct student model for every epoch. To guarantee diversity among our dual temporary teachers, we update the teacher model from student model trained with different augmentations for each epoch. Finally, Table shows ensembling two teacher models reduces the diversity and significantly degrades performance in our method. We ensemble the outputs of the two teacher models in equal proportions, following the previous protocol [26] in this experiment.
Impact of implicit consistency learning. Table 8 shows the results of our study on implicit consistency learning with diverse decay strategies to determine the drop rate in our student model. Following the original work [14], we investigate linear and uniform decay rules with different drop rates. We use the fixed training regime to train ResNet-101 on Cityscapes under 1/16 label partitions. Compared to the baseline model (75.85 mloU) trained without implicit consistency, the linear decay yields mostly stable performance improvements: but the performance degradation occurs at a drop rate of 0.4. On the other hand, uniform decay brings a notable enhance- ment at drop rate of 0.1. Nevertheless, as the drop rate exceeds 0. performance gets diminished. Uniform decay drops early layers more than linear decay, which imposes more significant challenges. We claim this challenging aspect enhances an implicit ensemble learning effect. (also evidenced by the highest number, 76.81 mloU); thereby, we adopt the uniform decay rule across all the experiments
Table 8: The performance under different decay strategies and drop rate 7 for implicit consistency.
Decay Strategy
0.1
0.2
0.3
0.4
Linear decay
75.87
76.55
76.63
75.74
Uniform decay
76.81
72.02
72.30
71.69
Performance vs. Training time. Here, we showcase the striking training efficiency of our method by comparing the trade-off between training time and mIoU against the current state-of of-the-art methods on ADE20K. For a fair
25.0
ADEZOK miou (1)
22.5
20.0
comparison, we run 200 epochs across all the methods, As described in Figure 3, our approach achieves competitive performance with less training time than other methods. Particularly, our method outperforms GTA-Seg, achieving 4.01% higher mIoU, requiring slightly less training time. In addition, we attain notable improvement of 5.86% in mloU with training time more than twice as fast as U Surprisingly, our method attains comparable mloU even with 12x faster than ReCo, which is based on pixel-level contrastive learning with sampling technique.
17.5
Method
GTA-Seg
15.0
ReCo
UºPL
12.5
Ours
10.0
1000
2000
3000
4000
5000
Training time (min)
Figure 3: mIoU (%) vs. training time. Ours shows clear efficiency over the competing methods with much improved final mloU.
Application unsupervised domain adaptation. We conduct extra experiments on unsupervised domain adaptation (UDA) task [39] for semantic segmentation. We apply our method on synthetic- to-real UDA scenarios using GTA5 [30] - Cityscapes benchmarks. In this experiment, we adopt DeepLabv2 [4] with ResNet- 101, which is pretrained on MS COCO [23] following the previous UDA setting [39]. As reported in Table 9, our approach achieves comparable performance to the prior UDA methods and outperforms the conventional single-teacher approach. While our method does not demonstrate performance improvement in the "train"" class, which suffers from a class-imbalance problem [35, 51], the overall performance supports the effectiveness of our method the UDA task.
9

---

(a) Input
(b) Supervised-only (c) Single Teacher
(d) Dual Teacher
(c) Ground Truth
Figure 4: Visualization of semantic segmentation results on PASCAL VOC. We qualitatively compare the segmentation results of our Dual Teacher with other methods, All methods are evaluated with Deepl.bbVa V33 on top of ResNct-101 under 1/1a label partitions.
Qualitative results. We visualize the semantic segmentation results compare our method with various methods in Figure 4. Column (b) shows the supervised-only method results, trained merely with limited amount of labeled data, produce the least plausible segmentation results but make incorrect predictions for many pixels. For example, in the last row, "sheep" pixels are completely misclassified "bird" in every pixel. Even though the Single Teacher yields slightly enhanced segmentation results, inaccurate pixels are still noticeable, as shown in column (c). Notably, in the first row, we can observe that Single Teacher produces inadequate results for pixels that contain different adjacent classes, such as person versus horse pixels. On the other hand, our Dual Teacher provides more sophisticated and accurate predictions compared to the Single Teacher.
5 Conclusion
In this paper, we have introduced novel Dual Teacher framework to address the coupling problem in the widely used teacher-student framework caused by EMA updates. Instead existing heavyweight solutions that rely on explicit ensemble, we have proposed simple yet efficient approach from the perspective of implicit ensemble learning. Our approach not only achieved competitive performance compared with state-of oftte-of-thearr methods but also required much less training time and parameters. We have demonstrated the superiority of our method over other competing approaches through extensive quantitative and qualitative comparisons. We hope that our method could serve valuable foundation for leveraging abundant unlabeled data in semantic segmentation tasks.
Limitation. Our focus was primarily on studying a more complicated segmentation task, but our method would apply to various other tasks, including classification detection.
Broader impact. Advanced semantic segmentation methods might have a potential risk when applied to malicious image manipulation such DeepFake. However, this work can contribute to more valuable applications the real world.
Acknowledgement. This work was partially supported by the Institute of Information & communi- cations Technology Planning & Evaluation (IITP) grant funded by the Korean government (MSIT) (RS-2023-00236225, Development of Perception/Planning Al SW for Seamless Autonomous Driving in Adverse Weather/Unsttructured Environment) and (UTP-2223-NNRRS-222-0022555 Intelligence Convergence Innovation Human Resources Development).
Artificial
10

---

References
[1] Eric Arazo, Diego Ortego, Paul Alhert, Noel O'Connor, and Kevin McGuinness. Pseudo-labeling and confirmation bias deep semi-superrvied learning. 2020 International Joint Conference on Neural Networks (JCNN), pages 1-8. IEEE, 2020.
[2] David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin A Raffel. Mixmatch: A holistic approach to semi-supervised learning. Advances in neural information processing systems, 32, 2019.
[3] David Berthelot, Rebecca Roelofs, Kihyuk Sohn, Nicholas Carlini, and Alex Kurakin. Adamatch: A unified approach to semi-supervised learning and domain adaptation. arXiv preprint arXiv:2106.045: 04732, 2021.
[4] Liang-Chieke Chen, George Papandreou, lasonas Kokkinos. Kevin Murphy. and Alan L Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE transactions on patter analysis and machine intelligence, 40(4):834-84) 2017.
[5] Liang- Chich Chen, Yukun Zhu, George Papandreou, Florian Schroff. and Hartwig Adam. decoder with atrous separable convolution for semantic image segmentation. In Proceedings of the European conference on computer vision (ECCV), pages 801-818, 2018
[6] Xiaokang Chen, Yuhui Yuan, Gang Zeng, and Jingdong Wang. Semi-supervised semantic segmentation with cross pseudo supervision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2612-2622 2021.
[7] MMSegmentation Contributors. MMSegmenntttinn: Openmmlab semantic segmentation toolbox and benchmark. https:///ithub: com/ heem//pen---ae mml wwt.mmma//mmmmmmmmaaatttttteemta
2020.
[8] Marius Cordts, Mohamed Omran. Sebastian Ramos, Timo Rehfeld, Markus Enzweiler. Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 3213-3223, 2016.
[9] Mark Everingham, Lue Van Gool. Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International journaa of computer vision, 88:303-338, 2010.
[10] Angela Fan, Edouard Grave, and Armand Joulin. Reducing transformer depth on demand with structured dropout. arXiv preprint arXiv: I 1909. 1556, 2019.
[11] Bharath Hariharan, Pablo Arheláez, Lubomir Bourdev, Subhransu Maji, and Jitendra Malik. Semantic contours from inverse detectors. In 201/ international conference computer vision, pages 991-998. IEEE, 2011.
[12] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 778778 2016.
[13] Hanzhe Hu. Fangyun Wei, Han Hu, Qiwei Ye. Jinshi Cui, and Liwei Wang. Semi-supervised semantic segmentation via adaptive equalization learning. Advances in Neural Information Processing Systems, 34:22106 22118, 2021.
[14] Gao Huang. Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Weinberger. Deep networks with stochastic depth. In Computer Vision-E.CC 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part 14, pages 646-661. Springer, 2016
[15] Ying Jin, Jiaqi Wang. and Dahua Lin. Semi-supervised semantic segmentation via gentle teaching assistant. Advances in Neural Information Processing Systems, 35:2803-2816, 2022.
[16] Zhanghan Ke, Di Qiu, Kaican Li, Qiong Yan, and Rynson WH Lau. Guided collaborative training for pixel- wise semi-supervised learning. Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XIII 16, pages 429-445. Springer, 2020.
[17] Zhanghan Ke, Daoye Wang, Qiong Yan. Jimmy Ren, and Rynson WH Lau. Dual student: Breaking the limits of the teacher in semi-supervised learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6728-6736, 2019.
[18] Mycongjin Kim and Hyeran Byun. Learning texture invariant representation for domain adaptation of semantic segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12975 12984, 2020.
11

---

[19] Anders Krogh and Jesper Vedelsby. Neural network ensembles, cross validation, and active learning. Advances in neural information processing systems, 7. 1994.
[20] Samuli Laine and Timo Aila. Temporal ensembling for semi-supervised learning. arXiv preprint arXiv.660.002222, 2016.
[21] Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems, 30, 2017.
[22] Dong-Hyun Lee et al. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Workshop on challenges representation learning, ICML. page 896, 2013.
[23] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740-755. Springer. 2014.
[24] Shikun Liu, Shuaifeng Zhi, Edward Johns, and Andrew J Davison. Bootstrapping semantic segmentation with regional contrast. arXiv preprint orXiv:2104.04465, 2021.
[25] Yen-Cheng Liu, Chih- Ma, Zijian He, Chia-Wen Kuo, Kan Chen, Peizhao Zhang, Bichen Wu, Zsolt Kira, and Peter Vajda. Unbiased teacher for semi-supervised object detection. arXiv preprint arXiv:2 12200948, 2021.
[26] Yuyuan Liu, Yu Tian, Yuanhong Chen, Fengbei Liu, Vasileios Belagiannis, and Gustavo Careeiro. Perturbed and strict mean teachers for semi-supervised semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4258-4267, 2022.
[27] Fengmao Lv, Tao Liang, Xiang Chen, and Guosheng Lin. Cross-domain semantic segmentation via interactive relation transfer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4334-4343, 2020.
[28] Ke Mei, Chuang Zhu, Jiaqi Zou, and Shanghang Zhang. Instance adaptive self self-training for unsupervised domain adaptation. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXVI 16, pages 415-430. Springer, 2020.
[29] Viktor Olsson, Wilhelm Tranheden, Juliano Pinto, and Lennart Svensson. Classmix: Segmentation- based data augmentation for semi-supervised learning. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1369-1378, 2021.
[30] Stephan R Richter, Vibhav Vineet, Stefan Roth, and Vladlen Koltun. Playing for data: Ground truth from computer games. In Computer Vision-ECCC 2016: 14th Europesa Conference, Amsterdam, The Netherlands, October 1-14, 2016, Proceedings, Part II 14, pages 102-118. Springer, 2016
[31] Mchdi Sajjadi, Mchran Javanmardi, and Tolga Tasdizen. Regularization with stochastic transformations and perturbations for deep semi-s eemissuperiied learning. Advances in neural information processing systems, 29, 2016.
[32] Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. Advances in neural information processing systems, 33:596-608, 2020.
[33] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1):1929-199 1958, 2014.
[34] Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference machine learning, pages 6105-6114 14. PMIRR 2019.
[35] Shuhan Tan, Xingchao Peng, and Kate Saeıko. Class-inbalanced domain adaptation: an empirical odyssey. In Computer Vision ECCV 2020 Workshops: Glasgow, UK, August 23 28, 2020, Proceedings, Part I 16, pages 585-602 Springer, 2020.
[36] Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight- consistency targets improve semi- suni-vupe deep learning results. Advances in neural information processing systems. 30, 2017.
12

---

[37] Hugo Touvron. Matthieu Cord, Mathhjs Douze, Francisco Massa, Alexandre Sablayrolles, and Herve Jegou. Training data-efficient image transformers &distillation through attention. In International Conference on Machine Learning, pages 10347-10357. PMLR, 2021.
[38] Hugo Touvron, Matthieu Cord, Maxime Oquab, Piotr Bojanowski. Jakob Verbeek. and Hervé Jégou. Co-training 2 submodels for visual recognition. arXiv preprint arXiv:2212.0488. 2022.
[39] Wilhelm Tranheden, Viktor Olsson, Juliano Pinto, and Lennart Svensson. Dacs: Domain adaptation via cross-domain mixed sampling. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1379-1389, 2021.
[40] Li Wan, Matthew Zeiler, Sixin Zhang, Yann Le Cun, and Rob Fergus. Regularization of neural networks using dropconnect. In International conference on machine learning, pages
PMLR, 2013.
[41] Yuchao Wang, Haochen Wang. Yujun Shen, Jingjing Fei, Wei Li, Guoqiang Jin, Liwei Wu, Rui Zhao, and Xinyi Le. Semi-supervised semantic segmentation using unreliable pseudo-labels. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4248-44-8 4257,,2 2022.
[42] Yanzhao Wu, Ling Liu, Zhongwei Xie, Ka-Ho Chow, and Wenqi Wei. Boosting ensemble accuracy by revisiting ensemble diversity metrics. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16469-16477, 2021.
[43] Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. Segformer: Simple and efficient design for semantic segmentation with transformers. Advances in Neural Information P Processing Systems, 34: 24777-1110 2021.
[44] Qizhe Xie, Minh Minh-Thang Thang Luong, Eduard Hovy, and Quoe V Le. Self-training with noisy student improves imagenet classification. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10687-10698, 2020.
[45] Haiming Xu, Lingqiao Liu, Qiuchen Bian, and Zhen Yang. Semi-supervine semantic segmentation with prototype-basee based consistency regularization. Advances in Neural Information Processing Systems, 55:26007-26005 2022.
[46] Lihe Yang, Lei Qi, Litong Feng. Wayne Zhang, and Yinghuan Shi. Revisiting weak-to-strong consistency in semi-supervised semantic segmentation. In Proceedings of IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages
2023.
[47] Lihe Yang. Wei Zhuo, Lei Qi. Yinghuan Shi, and Yang Gao. St++: Make self-training work better for semi superviidd semantic segmentation. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 4268-4277, 2022.
[48] Yanchao Yang and Stefano Soatto. Fda: Fourier domain adaptation for semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4085-4099 2020.
[49] Sangdoo Yun, Dongyoon Han, Seong Joon Oh. Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE/CVF international conference on computer vision, pages 6023-6032, 2019.
[50] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint arXiv: 1710.09412. 2017.
[51] Qiming Zhang, Jing Zhang, Wei Liu, and Dacheng Tao. Category anchor- anchor-guided unsupervised domain adaptation for semantic segmentation. Advances in neural information processing systems, 32, 2019.
[52] Shaofeng Zhang, Meng Liu, and Junchi Yan. The diversified ensemble neural network. Advances in Neural Information Processing Systems, 33:16001-16011 2020.
[53] Zhen Zhao, Lihe Yang. Sifan Long. Jimin Pi, Luping Zhou, and Jingdong Wang. Augmentation matters: A aupple-ee-efflectiv approach to semi-supervised semantic segmentation. arXiv preprint 27X75.2212.0497. 2022.
[54] Yuanyi Zhong, Bodi Yuan, Hong Wu, Zhiqiang Yuan, Jian Peng, and Yu-Xion Xiong Wang. Pixel contrastive- consistent semi superviee semantic segmentation. In Proceedings of the IEEE/CVF International Confer- Computer Vision, pages 7273-7282, 2021.
[55] Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler. Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In Proceedings of the IEEE conference computer vision and pattern recognition. pages 633-641, 2017.
13

---

[56] Yuliang Zou, Zizhao Zhang, Han Zhang, Chun-Liang Li, Xiao Bian. Jia-Bin Huang, and Tomas Pfister. Pseudoseg: Designing pseudo labels for semantic segmentation. arXiv preprint urXiv:2010.09713. 2020.
14


Reviews:
[{"review_id": "vnOgOGWh4f", "reviewer": "Reviewer_vns8", "summary": "This work proposes a method for semi-supervised semantic segmentation of images.\nTo mitigate the coupling problem of the student-teacher framework, the author proposes Dual teacher framework and  Implicit Consistency Learning.\nDual teacher framework consists of a pair of teacher-student models, where the two temporary teacher models are switched every epoch during training to teach the single student model.\nIn addition to this,  Implicit Consistency Learning is proposed, where the layer of the student model is stochastically dropped out (so becomes the so-called sub-model) and trains such sub-model by enforcing the prediction of the sub-model and the teacher model are same. \nThis approach shows the state-of-the-art performance for all dataset in every settings.", "questions": "1. What is the difference between PS-MT and the proposed one? Please give me a detailed comparison between them.\n\n2. Can you visualize (or numerically show) how much the proposed method diversifies teacher models' output compared to PS-MT or the typical mean-teacher model?", "limitations": "See weaknesses.", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 1, "strengths": "1. This paper shows the state-of-the-art performance compared to the previous and supervised methods.\n2. This paper conducts thorough experiments, showing their component's effectiveness.\n3. This paper is easy-to-follow and clearly motivated.", "weaknesses": "\nThe major concern of the reviewer in this paper is its technical contribution.\n\nThe main idea of this paper is: to introduce two teachers and students, and each of them is updated alternately activated per epoch, e.g., for epoch t, teacher model 1 is updated by student model 1 (and t2 is updated by s2), and for epoch t+1, teacher model 1 is updated by student model 2 (and t2 is updated by s1), and so on.\n\nHowever, this approach is very similar to PS-MT [1]. In PS-MT, it introduced two teacher model that is updated by one single student model.  And in PS-MT, such teacher models are updated 'alternatively' per epoch; that is, for epoch t, only one teacher network is updated while the other is frozen, and for epoch t+1, the frozen one is now updated, and the updated one is now frozen.\n\nThis training strategy is introduced to diversify the teacher model's output, which is exactly the same as the proposed method's objective.\n\nSo hereby the reviewer thinks that the proposed method is just a variant of PS-MT's method, and have a question about why the author of this paper did not compare their method with PS-MT's one. \n\nIn addition to this, the reviewer thinks that Implicit Consistency Learning is just a variant of the previous methods as the author already mentioned in their paper, and the only difference is that the source of consistency has been changed like this: from (sub-model of the student <--> full model of the student: previous method ) to (sub-model of the student <--> full model of the teacher: proposed method). \n\nSo the reviewer thinks the proposed components lack of their technical contributions at the current state of the paper.\n\n\nRef.\n\n[1] Yuyuan Liu, Yu Tian, Yuanhong Chen, Fengbei Liu, Vasileios Belagiannis, and Gustavo Carneiro. Perturbed and strict mean teachers for semi-supervised semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4258–4267, 2022.\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your constructive feedback. We address all the concerns raised by the reviewer and provide some new experimental results to support our contributions.\n\n>Q1. What are the main differences from PS-MT?\n\nA1. We would like to emphasize the notable differences. \n\n1. Our approach differs from PS-MT in that only one switchable teacher model teaches a student model per epoch, and this difference allows our method to obtain a more distinct student model for each epoch. \n2. To ensure diversity of our dual temporary teachers, we update the teacher model from a student model trained with different augmentations for each epoch.\n\nSpecifically, in PS-MT, two teachers always contribute to training the student model in all epochs. They use the ensembled outputs of the two teacher models in every epoch to provide better pseudo-labels to the student model. However, this training approach is limited to obtaining a student model that varies over epochs and results in teacher models that are updated via EMA from the student model not diversifying enough. \n\nOn the other hand, in our method, only one of the temporary teacher models involves teaching the student model in each epoch. This approach was intended to diversify the teacher models: our dual teachers are trained to have different perspectives using different types of augmentations, and only one of the teacher models independently teaches the student model with a distinct perspective at each epoch. We argue that ensembling teacher models ensures consistent supervision for our method but diminishes diversified guidance for the student model, which potentially hinders the student model from becoming more generalized in the end. Furthermore, we provide the numerical comparison table on this concern in the table of A4.\n\nTo provide the backup for our claim, we perform the following experiments to demonstrate that ensembling the outputs of the two teacher models negatively affects obtaining diverse teacher models in our method. Here, we ensemble the outputs of the two teacher models in equal proportions, according to the ensemble method of PS-MT. Please check the results in the table below.\n\n|Method | 1/16 (186) | 1/8 (372) |  1/4 (744) | 1/2 (1,488) |\n|:-|:-:|:-:|:-:|:-:|\n|**Ours + Switching temporary teaching**|77.0|79.23|78.94|80.18|\n|**Ours + Ensemble teaching**|73.25|75.99|76.41|77.34|\n\n>Q2. Why does the author not compare their method with PS-MT?\n\nA2. We already compared our method with PS-MT on both PASCAL VOC and Cityscapes datasets and verified our method has achieved better performances compared with PS-MT. Please check Tables 1 and 3 in our submitted paper.\n\n>Q3. Is implicit consistency learning a variant of the previous methods?\n\nA3. The aspect of consistent learning between the student's sub-model and the teacher's full-model within the Mean Teacher framework has not been previously tackled. Through extensive analysis and experimental validation on our method, we believe this can make a valuable contribution in this field.\n\n>Q4. Can you show how much the proposed method diversifies teacher models' output compared to PS-MT or the typical mean-teacher model?\n\nA4. We compared our method with the typical mean-teacher model in Figure 2 (a) in the paper. Here we further compare with PS-MT in the following tables for teacher diversity. Further details will follow:\n\n1. Comparison with typical mean-teacher:\n\nFigure 2. (a) shows the comparison results in prediction distance between typical mean-teacher (i.e., Single Teacher) and our dual teacher models. Our Dual Teacher consistently maintains a larger distance between the student and teachers than the typical mean-teacher model during training.\n\n2. Comparison with PS-MT:\n\nWe compare the output predictions of two teachers of ours and PS-MT using class-wise IoU metric, emphasizing noticeable differences. We highlight the top-5 categories with \"the most significant differences ($\\Delta$Diff)\" in class-wise IoU between the teacher models. For a fair comparison, we compute class-wise IoU when both teachers have similar mIoU values (e.g., Ours: 66.48 / 66.01 / PS-MT: 67.09 / 67.37). The results demonstrate a larger distance and a clear distinction between teachers, indicating that our method produces more diverse teacher models than PS-MT, which ensembles teacher outputs every epoch.\n\n- Proposed method (top-5 categories with the most significant differences):\n|Category | cow | dog |  cat | bottle | sheep |\n|:-:|:-:|:-:|:-:|:-:|:-:|\n|**Teacher #1**|58.73|61.24|61.66|63.16|79.55|\n|**Teacher #2**|77.70|52.98|69.81|56.91|75.97|\n|**$\\Delta$Diff**|18.97|8.26|8.15|6.20|3.58|\n\n- PS-MT (top-5 categories with the most significant differences):\n| Category | bird | plant | table | chair | motorbike |\n|:-:|:-:|:-:|:-:|:-:|:-:|\n|**Teacher #1**|79.68|49.19|47.38|20.63|73.78|\n|**Teacher #2**|84.33|45.62|49.09|22.31|75.34|\n|**$\\Delta$Diff**|4.65|3.57|1.71|1.68|1.67|"}, {"review_id": "hZ801ot7sm", "reviewer": "Reviewer_3Udz", "summary": "This work proposes the dual teacher knowledge distillation algorithm. For teachers trained with EMA during KD, there exists the issue of conformation bias which has been tackled in previous works. The motivation of this work is to diversify the training strategy using the dual-teacher student algorithm, in a way that it reduces the conformation bias as much as possible. This is done by making sure the teacher and student don't converge to the same representation in the feature space, using the proposed algorithm, which is a combination of augmentations to the inputs as well as switching teachers to train a student. The results obtained are quite promising compared to some recent KD schemes.", "questions": "1. Is there any study to show that the performance improvements from this approach are not attributed only to the input augmentation scheme?\n2. How do you tune the terms in equation 3? \n3. Does the training of the student require starting from a frozen checkpoint?", "limitations": "Yes", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Presentation:\nThe ideas are presented well. The concept is clear and explained in a way which is understandable to the reader. The reviewer appreciates the fact that figures and tables are self-explanatory, which improves the papers clarity. It reads well in a single pass.\n\nConcept:\nThe idea seems novel and well-motivated by using observations from prior art[1].\n\nExperiments:\nThe results are promising. It seems the main advantage of this method is that it converges faster than other related teacher-student EMA based algorithms published recently.\n\nTransparency:\nThe authors have made the code available and also addressed limitations and societal impacts of their work.\n\n[1] Pseudo-Labeling and Confirmation Bias in Deep Semi-Supervised Learning", "weaknesses": "One major concern is, as the authors have augmented their inputs to train the student, have they also augmented the data similarly while training other methods to compare with their proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your valuable comments and positive reviews. We address your concerns and questions below in detail.\n\n>Q1. Did the authors apply similar augmentations to other methods?\n\nA1. Yes, we utilize basic augmentations such as CutMix and ClassMix, which is a similar level of augmentation compared to recent methods. For instance, CPS, PS-MT, and U$^2$PL utilized CutMix, ReCo used ClassMix, and AEL and AugSeg applied advanced augmentations such as adaptive copy-paste CutMix and adaptive label-injecting CutMix.\n\n>Q2. Is there any study to show that the performance improvements from this approach are not attributed only to the input augmentation scheme?  \n\nA2. Yes. In Table 5, we reported that not only augmentation but also our training protocols that switch the temporary teachers make a significant contribution to performance improvement. For example, our strategy of switching temporary teachers shows a performance improvement of about 2% mIoU, regardless of the number of augmentations.\n\n>Q3. How do you tune the terms in Equation 3?\n\nA3. In all experiments, we set the hyperparameter $\\lambda_u$ as 1 to train the supervised and unsupervised losses in equal proportions.\n\n>Q4. Does the training of the student require starting from a frozen checkpoint? \n\nA4. Yes. In our experiment, we use ResNet-50/101 pre-trained on ImageNet as the backbones and ResNet-101 pre-trained on MS COCO for the domain adaptation task in Table 7. We employ the identical backbones utilized in the previous works.\n\n"}, {"review_id": "b34nhJnjM3", "reviewer": "Reviewer_gkhQ", "summary": "This paper addresses the task of semi-supervised semantic segmentation. It proposes dual teachers that are alternatively updated to supervise a single student. Improvements are obtained compared with previous methods.", "questions": "I see the authors upload the source code. Since several other works are reproduced for ADE20K, it will be more convincing and greatly appreciated if the authors can provide comprehensive experimental logs as well.", "limitations": "Yes, they have been discussed.", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "- Improvements over previous best methods on Pascal VOC and Cityscapes.\n- The method is also validated in unsupervised domain adaptation.", "weaknesses": "- The practice of dual teachers has already been explored in PS-MT [1]. The teachers in PS-MT are also alternatively updated and then supervise the student by ensembling their predictions.\n- Some latest methods should be compared in Table 2 and Table 3, such as AugSeg [2] and UniMatch [3]. The results are inferior to these methods in some splits. Besides, the authors should report the high-quality splits (92, 183, 366... labels) on Pascal. The Table 1 and Table 2 adopt the U2PL splits (choose from the high-quality images first), which are unfair for some methods, such as CPS and ST++ (randomly choose from the blended set).\n- More ablation studies are required. For example, does the improvement come from more diverse spatial augmentations, i.e., CutMix and ClassMix?\n\n[1] Perturbed and Strict Mean Teachers for Semi-supervised Semantic Segmentation, CVPR 2022.\n\n[2] Augmentation Matters: A Simple-yet-Effective Approach to Semi-supervised Semantic Segmentation, CVPR 2023.\n\n[3] Revisiting Weak-to-Strong Consistency in Semi-Supervised Semantic Segmentation, CVPR 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the constructive reviews with suggestions. We address all the concerns and add new experiments that you suggested.\n\n>Q1. The practice of dual teachers has already been explored in PS-MT. The teachers in PS-MT are also alternatively updated and then supervise the student by ensembling their predictions.\n\nA1. We would like to highlight the following differences. \n\n1. Our approach differs from PS-MT in that only one switchable teacher model teaches a student model per epoch, and this difference allows our method to obtain a more distinct student model for each epoch. \n2. To ensure the diversity of our dual temporary teachers, we update the teacher model from a student model trained with different augmentations for each epoch.\n\nIn PS-MT, two teachers contribute to training the student model in all epochs. They use the ensembled outputs of the two teachers in every epoch to provide better pseudo-labels to the student. This training approach is limited to obtaining a student that varies over epochs and results in teachers that are updated via EMA from the student not diversifying enough.\n\nHowever, in our method, only one of the temporary teachers involves teaching the student in each epoch. This approach was intended to diversify the teachers: our dual teachers are trained to have different perspectives using different types of augmentation, and only one of the teachers teaches the student with a distinct perspective at each epoch. We argue that ensembling teachers ensures consistent supervision for our method but diminishes diversified guidance for the student, which potentially hinders the student from becoming more generalized in the end.\n\nTo provide the backup for our claim, we perform the following experiments to demonstrate that ensembling the outputs of the two teachers negatively affects obtaining diverse teachers in our method. Here, we ensemble the outputs of the two teachers in equal proportions, according to the ensemble method of PS-MT. Please check the results in the table below.\n\n|Method | 1/16 (186) | 1/8 (372) |  1/4 (744) | 1/2 (1,488) |\n|:-|:-:|:-:|:-:|:-:|\n|**Ours + Switching temporary teaching**|77.0|79.23|78.94|80.18|\n|**Ours + Ensemble teaching**|73.25|75.99|76.41|77.34|\n\n>Q2. Some latest CVPR2023 methods should be compared in Table 2 and Table 3, such as AugSeg and UniMatch. \n\nA2. We show performance comparisons of our method in Tables 2 and 3 as well as Table 1, including AugSeg [CVPR2023] and UniMatch [CVPR2023], and our method shows comparable results with the competing AugSeg and UniMatch on PASCAL VOC. During the rebuttal period, we get better accuracy using the adaptive CutMix employed in AugSeg. We will append these results to our tables. +: means we use adaptive CutMix instead of the original CutMix.\n\n- Comparison results on PASCAL VOC.\n|Method (ResNet-50)|1/16|1/8|1/4|Method (ResNet-101)|1/16|1/8|1/4|\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|U$^2$PL|74.70|77.40|77.50|U$^2$PL|77.21|79.01|79.30|\n|AugSeg|77.28|78.27|78.24|AugSeg|79.29|81.46|80.5|\n|UniMatch|78.1|79.0|**79.1**|UniMatch|**80.9**|**81.9**|80.4|\n|**Ours**|77.0|79.23|78.94|**Ours**|78.82|81.19|**81.03**|\n|**Ours+**|**78.66**|**79.59**|78.52|**Ours+**|80.07|81.47|80.53|\n\n- Comparison results on Cityscapes.\n|Method | 1/16 (186) | 1/8 (372) |  1/4 (744) | 1/2 (1,488) |\n|:-:|:-:|:-:|:-:|:-:|\n|CPS|74.72|77.62|79.21|80.21|\n|AEL|75.83|77.9|79.01|80.28|\n|U$^2$PL|74.9|76.48|78.51|79.12|\n|PCR|73.41|76.31|78.4|79.11|\n|AugSeg|75.22|77.82|**79.56**|80.43|\n|UniMatch|76.6|77.9|79.2|79.5|\n|**Ours**|**76.81**|**78.4**|79.46|**80.52**|\n\n>Q3. Authors should report other splits (92, 183, 366... labels) on PASCAL VOC.\n\nA3. Thank you for the suggestion. In the submitted paper, we report performance following the experimental setup of U$^2$PL [CVPR-2022] but also provide the experimental results with the suggested splits on ResNet-50 here. We use the same image resolution in these experiments according to the standard training protocols (e.g., AugSeg).\n|Method | 1/16 (92) | 1/8 (183) |  1/4 (366) | 1/2 (732) | Full (1,464) |\n|:-:|:-:|:-:|:-:|:-:|:-:|\n|Supervised|44.03|52.26|61.65|66.72|73|\n|PseudoSeg|54.89|61.88|64.85|70.42|71|\n|PC$^2$Seg|56.9|64.63|67.62|70.9|72|\n|AugSeg|64.22|**72.17**|**76.17**|77.4|**78.82**|\n|**Ours**|**69.3**|71.02|76.09|**77.86**|78.28|\n\n>Q4. Authors adopt the U2PL splits (choose from the high-quality images first), which are unfair for some methods, such as CPS and ST++ (randomly choose from the blended set).\n\nA4. We follow the reviewer’s suggestion, and further perform experiments on ResNet-50 using the splits used in CPS and ST++ for a fair comparison.\n|Method | 1/16 (662) | 1/8 (1,323) |  1/4 (2,646) |\n|:-:|:-:|:-:|:-:|\n|CPS|71.98|73.67|74.90|\n|ST++|72.6|74.4|75.4|\n|PS-MT|72.83|75.7|76.43|\n|UniMatch|74.5|75.8|76.1|\n|AugSeg|**74.66**|75.99|77.16|\n|**Ours**|74.28|**76.15**|**77.42**|\n\n>Q5. Does the improvement come from more diverse spatial augmentations?\n\nA5. We hypothesized the truth of this question and now support it with new experimental results. We report our performance by adopting the adaptive CutMix proposed in AugSeg instead of the basic CutMix. We use PASCAL VOC 2012 with U2PL splits with ResNet-50. As shown below, we achieve mostly better results by changing the basic CutMix to a more sophisticated augmentation, adaptive CutMix. (In 1/4 partitions, we didn’t observe better performance using adaptive CutMix than basic CutMix, but tuning hyper-parameters would give better results.)\n\n|Method | 1/16 (662) | 1/8 (1,323) |  1/4 (2,646) |\n|:-:|:-:|:-:|:-:|\n|**Ours w/ CutMix**|77|79.23|**78.94**|\n|**Ours w/ Adaptive-CutMix**|**78.66**|**79.59**|78.52|\n\n>Q6. I see the authors upload the source code. Since several other works are reproduced for ADE20K, it will be more convincing and greatly appreciated if the authors can provide comprehensive experimental logs as well.\n\nA6. We followed AC's instructions and submitted the experimental logs to AC. Please check the logs from AC’s comment."}, {"review_id": "7Ne7Le6M0A", "reviewer": "Reviewer_WnDT", "summary": "The authors argue that the teacher-student coupling problem leads to the performance bottleneck in semi-supervised semantic segmentation. To this end, this paper proposes a novel Dual Teacher framework. Specifically, it introduces two teachers that are switched every epoch during training to supervise the student model. By doing so, it manages an implicit ensemble and the improved model diversity do contribute to better segmentation performances. Experiments under various evaluation protocols demonstrate the efficacy of the proposed method.", "questions": "I do not have any questions. Please refer to the weaknesses section.\n\n\n**Post Rebuttal Comments**\n\nThe reviewer appreciates the extra experiments a lot. Based on this evidence, it seems that the more diverse teachers we have, the better performance the student achieves, which is quite interesting. Also, as the teachers of PS-MT are much less diverse, the reviewer thinks that the student-teacher coupling problem has not been well addressed by PS-MT. Therefore, I am going to keep my initial rating: 6. The authors are supposed to revise their paper according to our dialog, especially the analysis of prediction distances.", "limitations": "The authors have discussed the limitations of this work in the submission.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper is well-written and easy to follow.\n2. The teacher-student coupling problem is novel and worth studying in semi-supervised semantic segmentation.\n3. Experiments are sufficient. Notably, the authors conduct experiments on ADE20k as well as using Transformer-based architectures to demonstrate the efficacy of the proposed method, which is not a common practice in the field of semi-supervised semantic segmentation.\n4. The code is provided in the supplementary material.\n", "weaknesses": "1. More explanations are needed to Table 5. Specifically, why does the Triple Teacher framework perform worse than the Dual Teacher alternative? As mentioned, to alleviate the teacher-student coupling problem, diversity among models is crucial. *It may be helpful to plot the prediction distance of the Triple Teacher framework in Figure 2(a).* I guess 3 teachers should be more diverse than 2. If it is supported by empirical evidence, it is better to explain the underlying reason for the performance degradation in Table 5.\n2. Missing ablations. It is relatively hard to evaluate the effectiveness of each component. It is better to start from the mean-teacher framework and add each component step by step.\n3. Figure 1(c) seems to be incorrect. From the method section, $x_w$ should be fed into the teacher while $x_{s_1}$ and $x_{s_2}$ should be fed into the student.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the insightful comments and positive reviews. We address all the concerns below and will revise them in the final version.\n\n>Q1. Why does the Triple Teacher framework perform worse than the Dual Teacher alternative in Table 5? As mentioned, to alleviate the teacher-student coupling problem, diversity among models is crucial.\n\nA1. Thank you for the constructive comments. In our experiments, we adopted different types of augmentation, such as class-level and coarse region-level augmentations (i.e., ClassMix, and CutMix) to train our Dual Teacher. In addition, we introduced Mixup as a third augmentation strategy to train Triple Teacher because Mixup is another well-known augmentation method recently. Within this setting, we speculate that Triple Teacher could not outperform Dual Teacher because Mixup has a similar effect to the augmentations already used for the Dual Teacher. However, we expect the Triple Teacher could surpass the Dual Teacher if we can provide diverse views to the model, which is sufficiently different from the two augmentations preemptively applied to the dual teacher. We will study better diversifying augmentations for Triple Teacher. \n\n>Q2. It is better to start evaluating each component from the mean-teacher framework step by step.\n\nA2. In Table 5, 'Single Teacher' refers to the conventional Mean Teacher framework. Hence, in the first column, we can observe the performance effect when appending different augmentations to the basic Mean Teacher framework. Meanwhile, the first row shows the impact of the number of temporary teachers from the Mean Teacher framework. We will provide more detailed and clear explanations in the captions of the table.\n\n>Q3. Figure 1(c) seems to be incorrect. From the method section, $x_w$ should be fed into the teacher while $x_{s1}$ and $x_{s2}$ should be fed into the student.\n\nA3. Thank you for pointing this out. We will modify it to accurately represent that the weak augmented inputs ($x_w$) are fed to the teacher models and the strong augmented inputs ($x_{s1}$ and $x_{s2}$) are supplied to the student model, respectively.\n"}]

Author rebuttal:

Meta-review: This paper received mixed reviews: two weak accepts, one borderline reject, and one reject. The reviewers appreciated strong performance, extensive experiments, and clarity of the draft. They at the same time raised concerns with limited technical contribution (vns8, gkhQ), lack of comparisons with latest work (gkhQ), unfair comparisons with some previous work (gkhQ), missing ablation studies (gkhQ, WnDT), and lack of reasoning for experimental results (WnDT).

The authors provided thorough rebuttals that address these comments; most of the concerns have been well assuaged, but a few remain unresolved, e.g., missing details of inference given two student models (vns8) and potential adverse effects by the specialization of teachers (vns8). Also, a reviewer expressed disappointment, questioning why the comparisons with latest methods (presented in CVPR 2023) were not included in the initial submission (gkhQ). Hence, even after a large body of discussions, the scores still diverge.

The AC found that the concerns on missing details of inference and potential adverse effects of the proposed decoupling strategy can be addressed by simple additional experiments or already have been addressed by existing results. Further, these concerns were raised after the author-reviewer discussion period and thus the authors did not have a chance to address them unfortunately. More importantly, comparisons with CVPR 2023 papers (i.e., UniMatch and AugSeg) are not mandatory since they were officially presented after the submission deadline of NeurIPS 2023; marginal improvement over or scores inferior to those of the CVPR 2023 papers should not be a factor for rejection.

Putting these together, the AC considers that these concerns are minor or inappropriate, and thus the strengths and the rebuttal outweigh the negative aspects of the paper. Hence, the AC recommends accepting the paper even though the average score is not sufficiently high; this decision has been extensively discussed between the AC and SAC. The authors are strongly encouraged to reflect the valuable comments and suggestions by the reviewers, and to include additional results given in the rebuttals, in the revision.
Decision: Accept (poster)