SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Evaluation. In all experiments, we use the mean of Intersection over Union (mIoU) as a metric to evaluate the performance of semantic segmentation. For Cityscapes, we apply online hard example mining loss as the supervision loss and sliding window evaluation following the prior works [41, 53].
On PASCAL VOC. In Table 1 and Table 2, we present the comparison results on validation set under different backbone networks (i.e., ResNet-50/101) to validate the performance of our method. Table 1 demonstrates the effectiveness of our approach, consistently surpassing the performance of the baseline with the ResNet-50 backbone. Compared to the current state-of-the-art methods, AugSeg and UniMatch, our method yields superior results in three out of the five partitions. Except for the UniMatch [46], other studies have a higher training resolution than ours: 512 vs. 321. More details are provided in the supplementary material.
Table 2 shows the comparison results with the state-of-the-art methods based on ResNet-101. Compared to the baseline, our method achieves a performance improvement of  $+10.95\%$ ,  $+9.64\%$ ,  $+5.23\%$ , and  $+3.49\%$  for the label partitions  $^{1/16}$ ,  $^{1/8}$ ,  $^{1/4}$ , and  $^{1/2}$ , respectively. While a few numbers achieve marginal improvements over the state-of-the-art approaches, it is noteworthy that our simple approach is competitive with more complicated methods, such as GTA-Seg [15] and PCR [45], which rely on additional networks, and AEL [13], which employs more sophisticated augmentations.
Table 3: Comparison of mIoU (%) with state-of-the-art methods on Cityscapes under different partitions. All methods are based on DeepLabv3+ with ResNet-101 backbone.
Method	#Params	1/16 (186)	1/8 (372)	1/4 (744)	1/2 (1,488)
Supervised-only	62.6M	65.74	72.53	74.43	77.83
CPS [6] [CVPR'21]	125.2M	74.72	77.62	79.21	80.21
AEL [13] [NeurIPS'21]	62.6M	75.83	77.9	79.01	80.28
PS-MT [26] [CVPR'22]	62.6M	-	76.9	77.6	79.09
U²PL [41] [CVPR'22]	64.5M	74.9	76.48	78.51	79.12
PCR [45] [NeurIPS'22]	64.5M	73.41	76.31	78.4	79.11
UniMatch [46] [CVPR'23]	62.6M	76.6	77.9	79.2	79.5
Ours	62.6M	76.81	78.4	79.46	80.52
Table 4: Comparison of mIoU (%) with state-of-the-art methods on ADE20K under different partitions. All results are reported with single-scale inference on SegFormer with MiT-B1 backbone. †: Results of our re-implementation according to each official code.
Method	#Params	1/32 (631)	1/16 (1,263)	1/8 (2,526)	1/4 (5,052)	1/2 (10,105)
Supervised-only	13.7M	15.15	22.32	25.90	29.53	32.57
CPS† [6] [CVPR'21]	27.5M	18.79	22.27	27.89	32.41	36.89
U²PL† [41] [CVPR'22]	15.0M	18.54	23.64	28.11	33.52	36.01
GTA-Seg† [15] [NeurIPS'22]	28.8M	19.52	23.09	26.86	27.85	29.99
ReCo† [24] [ICLR'22]	14.4M	21.11	23.70	27.74	30.17	35.54
Ours	13.7M	22.60	26.01	30.60	33.74	37.30
On Cityscapes. Table 3 reports comparison results of our method against several existing methods on the Cityscapes validation set. Our method yields improvements of  $+11.07\%$ ,  $+5.87\%$ ,  $+5.03\%$ , and  $+2.69\%$  over the baseline in the  $1/16, 1/8, 1/4,$  and  $1/2$  label partitions, respectively, demonstrating its effectiveness, particularly in the scarce-label setting. Notably, our method outperforms CPS [6] on  $1/16$  partition, achieving over  $2\%$  higher mIoU despite using only half as many trainable parameters. In addition, we achieve superior performance across all the label partitions compared to AEL [13], a method that introduces more advanced augmentations – adaptive Copy-Paste and adaptive CutMix.
On ADE20K. We further validate the effectiveness of our approach using SegFormer [43] in Table 4, which unifies Transformer encoder and a lightweight multi-layer perception (MLP) decoder on ADE20K. Since there are no standard label partitions of ADE20K for semi-supervised semantic segmentation, we construct partitions by sampling randomly from the entire training data. For a fair comparison, we re-implement the current state-of-the-art methods according to their official codes under the same network architecture (i.e., SegFormer with MiT-B1 backbone). Our method consistently outperforms the supervised baseline by  $+4.86\%$ ,  $+3.69\%$ ,  $+4.70\%$ ,  $+4.21\%$ , and  $+4.73\%$  in the  $1/32$ ,  $1/16$ ,  $1/8$ ,  $1/4$ , and  $1/2$  partitions, respectively. It is noteworthy that we achieve competitive performance despite using fewer trainable parameters compared to other methods.
Analysis of diversity. To demonstrate the coupling problem, we report the prediction distance between the teacher and student models. Here, we compute the mean square error of predictions between two models to measure prediction distance. As depicted in Figure 2 (a), the predictions of the EMA teacher are very close to those of the student model in the conventional teacher-student framework (denoted as Single Teacher). On the other
(a) Prediction distance
(b) Class-wise IoU
Figure 2: Results of Dual Teacher in training process on  $^{1} / _{16}$  partitions of the PASCAL VOC blender set using ResNet-50.
hand, we observe that Dual Teacher consistently strives to maintain the disparity between the predictions of the teacher and student models during training. We argue that this approach allows the
Table 5: The performance relative to the number of teacher models and augmentations on  $1/16$  splits of the PASCAL VOC blender set.
Method	Single	Dual	Triple
Aug × 1	73.63	75.34	75.30
Aug × 2	74.21	77.00	76.56
Aug × 3	74.49	76.25	75.94
Table 6: Analyzing the diversity of supervisions using prediction distances between teacher-student models. A larger STD denotes a more diversified teacher's supervision.
Epoch	1 - 5	6 - 10	11 - 15	16 - 20
Single	0.0018±0.0010	0.0016±0.0005	0.0012±0.0004	0.0012±0.0004
Dual	0.0378±0.0209	0.0249±0.0095	0.0181±0.0074	0.0199±0.0059
Triple	0.0425±0.0224	0.0169±0.0020	0.0194±0.0040	0.0222±0.0024
Table 7: Performance analysis of different teaching strategies on PASCAL VOC 2012 across various partitions. Labeled images are sampled from the blender set and all methods are based on DeepLabv3+ with ResNet-50.
Method	1/16 (186)	1/8 (372)	1/4 (744)	1/2 (1,488)
Switching teachers	77.00	79.23	78.94	80.18
Ensembling teachers	73.25	75.99	76.41	77.34
student model to become more generalized, preventing it from learning towards a particular bias. This result supports our conjecture that the EMA teacher model is tightly coupled with the student model, which may incur performance degradation.
In Figure 2 (b), we investigate the distinct characteristics of the two temporary teacher models in Dual Teacher. Here, we select the top-5 categories with the most significant differences in class-wise IoU between dual temporary teacher models at the 10th epoch. Although the temporary teacher models have similar mIoU of  $66.48\%$  and  $66.01\%$  in the middle of the training process, we observe significant performance differences across each category. This observation indicates that our dual temporary teachers produce distinct supervisions and teach the student model complementarily.
Impact of the number of teachers and augmentations. In this study, we investigate our method by extending the number of temporary teachers from Dual Teacher to Triple Teacher while also expanding the number of augmentations. In Table 5, the first row shows the performance according to the number of teacher models by applying only ClassMix [29], which is specialized for semantic segmentation. We observe that employing two or more temporary teachers outperforms a single permanent teacher. This finding highlights the effectiveness of our approach that leverages teacher switching. In the second row, we begin to add CutMix [49] to our augmentation pool and utilize two different augmentations. We observe that the Dual Teacher achieves  $2.79\%$  improvement over the Single Teacher, and the Triple Teacher shows saturated performance compared to the Dual Teacher. It confirms that enriching strong augmentation types in addition to teacher switching is beneficial. In the last row, we incorporate three different augmentations by appending MixUp [50]. In this case, the results do not significantly outperform the counterparts of using two augmentations, regardless of the number of teachers. Ultimately, we draw a conclusion that leveraging both dual teachers and augmentations in our framework is the most effective.
Dual teachers vs. Triple teachers. In Table 6, we present a comparative analysis of the prediction distances between teacher and student models across several frameworks. Notably, the Dual Teacher exhibits more pronounced fluctuations in distances, as evidenced by its higher standard deviations (STDs). Although the trend shows a similar decline for both Dual Teacher and Triple Teacher across epochs, we observe a consistently lower STD for Triple Teacher. This suggests that Dual Teacher demonstrates greater variability in prediction distance across epochs, often varying more than double that of the Triple Teacher. We posit that since the three augmentation methods we used are probably not orthogonal, teacher models may tend to transfer similar characteristics to the student model due to the diminished diversity among teacher models. We presume the diversity inherent in teacher models is intrinsically connected to a student model's generalization capability, and our empirical backup supports it (see Table 5). Given the noticeable variations in prediction distances, we contend that teacher diversity is instrumental in amplifying the efficacy of Dual Teacher. As a result, teacher diversity is highly likely to contribute to enhanced performance for Dual Teacher, preventing the student model from learning towards a specific teacher bias.
Switching teachers vs. Ensembling teachers. One prevalent approach to leveraging dual teachers involves the ensembling of teacher models, which then simultaneously guide a student model [26]. In this teacher ensembling strategy, both teacher models consistently guide the student model throughout
Table 9: Comparison of mIoU (%) with the previous unsupervised domain adaptation methods on Synthetic-to-Real: GTA5→Cityscapes. All methods are evaluated on DeepLabv2 with ResNet-101.
Method	Road	SW	Build.	Wall	Fence	Pole	TL	TS	Veg.	Terr.	Sky	PR	Rider	Car	Truck	Bus	Train	Motor	Bike	mIoU
Source-only	63.31	15.65	59.39	8.56	15.17	18.31	26.94	15	80.46	15.25	72.97	51.04	17.67	59.68	28.19	33.07	3.53	23.21	16.73	32.85
LTIR [18]	92.9	55.0	85.3	34.2	31.1	34.9	40.7	34.0	85.2	40.1	87.1	61.0	31.1	82.5	32.3	42.9	0.3	36.4	46.1	50.2
FDA [48]	92.5	53.3	82.4	26.5	27.6	36.4	40.6	38.9	82.3	39.8	78.0	62.6	34.4	84.9	34.1	53.1	16.9	27.7	46.4	50.45
PIT [27]	87.5	43.4	78.8	31.2	30.2	36.3	39.9	42.0	79.2	37.1	79.3	65.3	37.5	83.2	36.0	45.6	25.7	23.5	49.9	50.6
IAST [28]	93.8	57.8	85.1	39.5	26.7	26.2	43.1	34.7	84.9	32.9	88.0	62.6	29.0	87.3	39.2	49.6	23.2	34.7	39.6	51.5
Single Teacher	93.34	56.81	86.28	34.63	32.27	32.99	39.21	43.93	86.72	45.11	85.81	63.05	21.28	86.76	38.66	50.05	0.0	13.8	33.7	49.71
Dual Teacher	93.93	56.88	86.47	35.49	31.57	35.12	37.78	43.43	86.02	43.54	88.06	62.14	28.49	88.59	50.14	54.2	0.18	35.75	49.82	53.03
all epochs. Therefore, this approach could limit the diversity of the student model, leading to weaker teacher models updated via EMA from the student model. On the other hand, our approach is differentiated by using a single switchable teacher model to instruct the student model in each epoch with more diversity. This ensures that our method produces a uniquely distinct student model for every epoch. To guarantee diversity among our dual temporary teachers, we update the teacher model from a student model trained with different augmentations for each epoch. Finally, Table 7 shows assembling two teacher models reduces the diversity and significantly degrades performance in our method. We ensemble the outputs of the two teacher models in equal proportions, following the previous protocol [26] in this experiment.
Impact of implicit consistency learning. Table 8 shows the results of our study on implicit consistency learning with diverse decay strategies to determine the drop rate in our student model. Following the original work [14], we investigate linear and uniform decay rules with different drop rates. We use the fixed training regime to train ResNet-101 on Cityscapes
Table 8: The performance under different decay strategies and drop rate  $\tau$  for implicit consistency.
Decay Strategy	0.1	0.2	0.3	0.4
Linear decay	75.87	76.55	76.63	75.74
Uniform decay	76.81	72.02	72.30	71.69
under  $\frac{1}{16}$  label partitions. Compared to the baseline model (75.85 mIoU) trained without implicit consistency, the linear decay yields mostly stable performance improvements; but the performance degradation occurs at a drop rate of 0.4. On the other hand, uniform decay brings a notable enhancement at a drop rate of 0.1. Nevertheless, as the drop rate exceeds 0.1, performance gets diminished. Uniform decay drops early layers more than linear decay, which imposes more significant challenges. We claim this challenging aspect enhances an implicit ensemble learning effect. (also evidenced by the highest number, 76.81 mIoU); thereby, we adopt the uniform decay rule across all the experiments.
Performance vs. Training time. Here, we showcase the striking training efficiency of our method by comparing the trade-off between training time and mIoU against the current state-of-the-art methods on ADE20K. For a fair comparison, we run 200 epochs across all the methods. As described in Figure 3, our approach achieves competitive performance with less training time than other methods. Particularly, our method outperforms GTA-Seg, achieving  $4.01\%$  higher mIoU, requiring slightly less training time. In addition, we attain a notable improvement of  $5.86\%$  in mIoU with training time more than twice as fast as U $^2$ PL. Surprisingly, our method attains comparable mIoU even with 12x faster than ReCo, which is based on pixel-level contrastive learning with sampling technique.
Figure 3: mIoU  $(\%)$  vs. training time. Ours shows clear efficiency over the competing methods with a much improved final mIoU.
Application to unsupervised domain adaptation. We conduct extra experiments on unsupervised domain adaptation (UDA) task [39] for semantic segmentation. We apply our method on synthetic-to-real UDA scenarios using GTA5  $[30] \rightarrow$  Cityscapes benchmarks. In this experiment, we adopt DeepLabv2 [4] with ResNet-101, which is pretrained on MS COCO [23] following the previous UDA setting [39]. As reported in Table 9, our approach achieves comparable performance to the prior UDA methods and outperforms the conventional single-teacher approach. While our method does not demonstrate performance improvement in the "train" class, which suffers from a class-imbalance problem [35, 51], the overall performance supports the effectiveness of our method on the UDA task.
Figure 4: Visualization of semantic segmentation results on PASCAL VOC. We qualitatively compare the segmentation results of our Dual Teacher with other methods. All methods are evaluated with DeepLabV3+ on top of ResNet-101 under  $^{1} / _{16}$  label partitions.
Qualitative results. We visualize the semantic segmentation results to compare our method with various methods in Figure 4. Column (b) shows the supervised-only method results, trained merely with a limited amount of labeled data, produce the least plausible segmentation results but make incorrect predictions for many pixels. For example, in the last row, "sheep" pixels are completely misclassified as "bird" in every pixel. Even though the Single Teacher yields slightly enhanced segmentation results, inaccurate pixels are still noticeable, as shown in column (c). Notably, in the first row, we can observe that Single Teacher produces inadequate results for pixels that contain different adjacent classes, such as person versus horse pixels. On the other hand, our Dual Teacher provides more sophisticated and accurate predictions compared to the Single Teacher.
In this paper, we have introduced a novel Dual Teacher framework to address the coupling problem in the widely used teacher-student framework caused by EMA updates. Instead of existing heavyweight solutions that rely on explicit ensemble, we have proposed a simple yet efficient approach from the perspective of implicit ensemble learning. Our approach not only achieved competitive performance compared with state-of-the-art methods but also required much less training time and parameters. We have demonstrated the superiority of our method over other competing approaches through extensive quantitative and qualitative comparisons. We hope that our method could serve as a valuable foundation for leveraging abundant unlabeled data in semantic segmentation tasks.
**Limitation.** Our focus was primarily on studying a more complicated segmentation task, but our method would apply to various other tasks, including classification or detection.
Broader impact. Advanced semantic segmentation methods might have a potential risk when applied to malicious image manipulation such as DeepFake. However, this work can contribute to more valuable applications in the real world.
Acknowledgement. This work was partially supported by the Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korean government (MSIT) (RS-2023-00236245, Development of Perception/Planning AI SW for Seamless Autonomous Driving in Adverse Weather/Unstructured Environment) and (IITP-2023-No.RS-2023-00255968, Artificial Intelligence Convergence Innovation Human Resources Development).
[56] Yuliang Zou, Zizhao Zhang, Han Zhang, Chun-Liang Li, Xiao Bian, Jia-Bin Huang, and Tomas Pfister. Pseudoseg: Designing pseudo labels for semantic segmentation. arXiv preprint arXiv:2010.09713, 2020.