SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Jaemin Na $^{1}$ , Jung-Woo Ha $^{2}$ , Hyung Jin Chang $^{3}$ , Dongyoon Han $^{2\dagger}$ , Wonjun Hwang $^{1,2\dagger}$   
Ajou University, Korea $^{1}$ , Naver AI Lab $^{2}$ , University of Birmingham, UK $^{3}$   
osial46@ajou.ac.kr, jungwoo.ha@navercorp.com, h.j.chang@bham.ac.uk  
dongyoon.han@navercorp.com, wjhwang@ajou.ac.kr
The teacher-student framework, prevalent in semi-supervised semantic segmentation, mainly employs the exponential moving average (EMA) to update a single teacher's weights based on the student's. However, EMA updates raise a problem in that the weights of the teacher and student are getting coupled, causing a potential performance bottleneck. Furthermore, this problem may become more severe when training with more complicated labels such as segmentation masks but with few annotated data. This paper introduces Dual Teacher, a simple yet effective approach that employs dual temporary teachers aiming to alleviate the coupling problem for the student. The temporary teachers work in shifts and are progressively improved, so consistently prevent the teacher and student from becoming excessively close. Specifically, the temporary teachers periodically take turns generating pseudo-labels to train a student model and maintain the distinct characteristics of the student model for each epoch. Consequently, Dual Teacher achieves competitive performance on the PASCAL VOC, Cityscapes, and ADE20K benchmarks with remarkably shorter training times than state-of-the-art methods. Moreover, we demonstrate that our approach is model-agnostic and compatible with both CNN- and Transformer-based models. Code is available at https://github.com/naver-ai/dual-teacher.
Semantic segmentation has progressed along with the advance of deep neural networks as a critical component of computer vision tasks for visual understanding [8, 9, 55]. However, the prevailing approaches are primarily based on supervised learning methods and still require laborious and time-consuming manual annotations at the pixel-level. To alleviate this intrinsic challenge, semi-supervised semantic segmentation has recently come into the limelight requiring only a small amount of labels.
A crucial challenge in successful semi-supervised learning is how to acquire reliable and consistent labels from unlabeled data, particularly in more demanding tasks like semi-supervised segmentation. Pseudo-labeling [22], the most popular approach, assigns the off-the-shelf model's class predictions as labels for unlabeled data for training. Another seminal work, a teacher-student framework [36], is widely adopted as it provides robust results in numerous semi-supervised learning tasks [41, 26, 13]. Despite the tremendous success, it is problematic that the weights of the teacher and student models are inevitably coupled as the teacher weights are updated using an exponential moving average (EMA) of the student model weights. This notable issue is called the teacher-student coupling problem [17] and causes the performance bottleneck for the student model because the EMA teacher exhibits similar performance and perspective to the student model in traditional teacher-student framework [36]. Furthermore, when the student model produces biased predictions over a few samples, training
(a) Teacher-Student
(b) Dual Student
(c) Dual Teacher (ours)
Figure 1: Semi-supervised semantic segmentation frameworks. (a) A teacher model generates pseudo-labels to train a student model and is updated as an exponential moving average (EMA) of the student model weights. (b) Dual Student involves two distinct student models that also serve as teachers to each other. (c) Our Dual Teacher employs dual temporary teachers that switch alternately during training.
with a single EMA teacher is prone to accumulate errors and provide incorrect pseudo-labels to the student model, which causes the irremediable misclassification (i.e., confirmation bias [1]). One straightforward approach [17, 16, 6] to mitigate the issues is to employ explicit ensemble models with different initializations instead of counting on a single student model. However, incorporating another deep neural network is not preferred due to the heavy computational demands and the inherent challenge of achieving scalability.
In this paper, we propose a novel Dual Teacher framework that comprises a single-student model and dual temporary teacher models to alternatively handle the coupling problem between the teacher and student network. Our framework, which is depicted in Figure 1, discourages the teacher and student models from converging and becoming similar to each other as training progresses. We feed the student model with different strongly augmented inputs for each epoch in training, thereby encouraging the construction of a student model with distinct characteristics at every epoch. At the same time, as we change the augmentation applied to the student model, we also switch the temporary teacher models assigned to it. This ensures that the temporary teacher models keep retaining the characteristics of the student model for each epoch. Additionally, we improve the robustness of the student model by enforcing consistent predictions between fractional models of the student model and the full teacher model. These sub-models contribute to an implicit ensemble, which is differentiated from the explicit ensemble.
Ultimately, our approach achieves competitive performance with state-of-the-art methods [41, 15, 24] while requiring much shorter training times and fewer parameters. We demonstrate the superiority of Dual Teacher under the semi-supervised semantic segmentation protocol using public benchmarks, including PASCAL VOC [9] and Cityscapes [8]. We further prove the scalability of our method using the large-scale dataset, ADE20K [55], and have made our experimental protocols available to the public. We conduct extensive experiments not only with ResNet-50/-101 but also with Transformer-based SegFormer [43].
Semi-supervised Learning. Semi-supervised learning (SSL) methods benefit significantly from consistency regularization [2, 31], which encourages a model to produce the same prediction when the input is perturbed, and pseudo-labeling [22] which uses a model's prediction as a label to train against. Recently, semi-supervised learning methods [2, 3, 32] have shown that consistency regularization and pseudo-labeling can work together harmoniously. Most of all, FixMatch [32] generates predicted labels from weakly-augmented unlabeled images and exploits them as pseudo-labels when strongly augmented versions of the same images are fed into the identical model. This simple yet effective approach has been widely used in recent studies when leveraging unlabeled data in SSL.
Semi-supervised Semantic Segmentation. Recent success in semi-supervised semantic segmentation approaches can be attributed to the effective incorporation of consistent regularization and pseudo-labeling. For example, CPS [6] and GCT [16] construct two parallel segmentation networks with the same structure and impose consistency on two networks with different initializations for the same input. Instead of using two different networks, which is burdensome to train, the latest works [41, 26, 15] adopt a teacher-student framework in which only the student model is trained, and the teacher model is updated through the EMA of the student model. On top of this simple
framework, GTA-Seg [15],  $\mathrm{U}^2\mathrm{PL}$  [41], and ReCo [24] introduce additional representation heads, and GTA-Seg further attaches a teacher assistant model the same size as the student model.
On the other hand, simple and effective data augmentations [49, 29, 50] that yield label-preserving perturbations have played a critical role in consistency regularization for semi-supervised semantic segmentation [6, 41, 24, 46]. Moreover, AEL [13] and AugSeg [53] contribute to performance improvement by introducing more advanced adaptive data augmentation techniques. However, unlike the previous studies, we achieve comparable performance by leveraging basic augmentations within our novel teacher-student framework without any lateral networks or sophisticated data augmentations. Meanwhile, PS-MT [26] presents an ensemble strategy that leverages multiple teachers to enhance the segmentation accuracy of unlabeled images. In contrast, rather than ensembling them, we employ the multiple teacher models in a sequential and independent manner. This approach ensures that each teacher model, with its diverse characteristics, distinctly contributes to the student model.
Ensemble Learning. Network ensembling [19] gains popularity for improving model predictions by combining the outputs of diverse models. Recent works [21, 52, 42] have successfully improved performance by model ensembling; that is, the individual models are trained to maximize their diversity so that the ensemble models are less prone to overfitting. Although these straightforward explicit ensemble approaches have shown significant performance improvements, they share the downside of bearing a burden of computation and memory cost. The impact of the explicit ensembling has also been indirectly exploited to train a single network through dropout [33], dropconnect [40], or stochastic depth [14] by activating only a subset of the network, and thus the complete network can be seen as an implicit ensemble of trained sub-networks [34, 37]. Of particular interest is that Temporal Ensembling [20] forms a consistent prediction of the unlabeled data using the outputs of the network being trained at different epochs under different regularization and input augmentations. Taking a step further with this approach, Mean Teacher [36] updates the teacher model with EMA weights of the student model instead of sharing weights with the student model as an ensembling perspective. Although this teacher-student framework has exhibited a significant impact, one critical problem is the coupling problem, wherein tightly coupled weights deteriorate performance. Alternatively, we address this problem by introducing dual temporary EMA teachers utilizing temporary and implicit ensemble perspectives.
Teacher-student framework. The teacher-student framework [36] is a popular approach in semi-supervised learning that leverages unlabeled data to enhance model performance. As described in Figure 1 (a), it comprises a teacher and a student model, where each model includes an encoder and a decoder. In semi-supervised semantic segmentation, the student model is trained with a few labeled data as follows:
$$
\mathcal {L} _ {s u p} = \frac {1}{\mathcal {B} _ {l}} \sum_ {i = 1} ^ {\mathcal {B} _ {l}} \frac {1}{\mathcal {H} \cdot \mathcal {W}} \sum_ {j = 1} ^ {\mathcal {H} \cdot \mathcal {W}} \mathcal {L} _ {c e} \left(p _ {i j} ^ {l}, y _ {i j} ^ {l}\right), \tag {1}
$$
where  $\mathcal{B}_l$  denotes the number of labeled images  $x_{ij}^{l}$  with a resolution of  $\mathcal{H} \times \mathcal{W}$  in a training batch, and the  $\mathcal{L}_{ce}$  is a pixel-wise cross-entropy loss applied to every pixel  $j$  on  $i$ -th labeled image. The  $p_{ij}^{l}$  indicates prediction for the labeled images of the student model, and  $y_{ij}^{l}$  is ground-truth labels of corresponding labeled images. In this work, we include this supervised loss for the student model, and unsupervised loss for unlabeled data is addressed in Section 3.2.
The teacher model's parameters  $\theta_{t}$  are updated with an EMA of the student model's parameters  $\theta_{s}$  as:
$$
\theta_ {t} \leftarrow \alpha \theta_ {t} + (1 - \alpha) \theta_ {s}, \tag {2}
$$
where  $\alpha$  is a smoothing coefficient hyperparameter.
Semi-supervised semantic segmentation. In semi-supervised semantic segmentation, sophisticated data augmentations and network architectures are essential for handling pixel-level labels with increased degrees of freedom. However, due to the limited number of training instances compared to the abundance of mask labels, a more diversified student model should be involved than semi-supervised
classification. Furthermore, we argue that incorporating cumbersome data augmentations [13, 53] and additional heads [41, 15, 24] may impair efficiency, limiting the full potential of its benefits.
Revisiting Temporal Ensembling. Our work is inspired by Temporal Ensembling [20], which pursues to enjoy an indirect ensemble (i.e., implicit ensemble) effect within a single model by ensembling predictions obtained from different regularizations for each epoch. This training approach accumulates the network's predictions into ensemble predictions after every training epoch. This process has been devalued for its slow training pace due to updating only once per epoch; however, we revalue it as a favorable method to inject diversity into our student model to address the coupling problem.
3.2 Dual Temporary Teacher
In the context of semi-supervised semantic segmentation, our goal is to mitigate the coupling problem [17] caused by the single EMA update routine employed in the conventional teacher-student framework. Among the critical factors [17, 32] for improving the teacher model, one we mainly take notice of is the diversity of the student models, as noted in prior works [17, 25, 44]. We strive to prevent the teacher and student models from becoming too similar to each other as training goes on by establishing a virtuous cycle in which the diversity of the student model enhances the teacher model. In turn, the improved teacher preserves students' diverse perspectives and effectively guides the student model. Since the student updates each EMA teacher, the teachers get enhanced as simultaneously diversifying the student due to the ensemble effect [19, 36]. The strengthened teacher model can subsequently provide more refined supervisory signals to the student model.
From this perspective, we introduce a simple yet effective framework - Dual Teacher: an approach that temporary EMA teacher models alternate in generating pseudo-labels to guide a student model; concurrently, the student model updates the weights of the teacher models with an exponential moving average of its own weights. We claim that introducing additional EMA teachers facilitates the diversification of students by providing distinct and varied supervision. As described in Figure 1 (c), Dual Teacher consists of a pair of teacher-student models, where the two temporary teacher models are switched every epoch during training to teach the single student model.
Similar to prior studies [32, 26, 41], we adopt strongly-augmented images as inputs for the student model and weakly-augmented images for the teacher models to ensure reliable pseudo-labels. However, instead of the traditional teacher-student framework that relies on a single permanent teacher model, we introduce dual temporary teachers alternately activated per epoch, providing diversified guidance to the student model. The temporary EMA teachers capture the evolving temporal knowledge of the student model, performing like a temporal ensemble at different time steps.
Strong augmentation pool. To diversify the student model more, we go beyond leaning solely on the EMA teacher model, albeit with alternating dual teachers. To further ensure diversity, we provide variety by changing the type of strong augmentation applied to the student model when shifting the temporary teacher models per epoch. We achieve this by building a predefined but non-deterministic pool of strong augmentations, including both fine class-level and coarse region-level augmentations (i.e., ClassMix [29] and CutMix [49]). Each training epoch randomly samples one augmentation from the pool, constraining that the consecutive epoch does not use the same augmentation.
Updating student and teachers. In a nutshell, the use of temporary teacher models helps to introduce diversity into the student model, while the diverse characteristics acquired by the student model contribute to the enhancement of the teacher models. Formally, the objective functions for the student model are defined as follows:
$$
\theta_ {s} \leftarrow \theta_ {s} + \eta \frac {\partial \left(\mathcal {L} _ {s u p} + \lambda_ {u} \mathcal {L} _ {u n s u p}\right)}{\partial \theta_ {s}}, \quad \mathcal {L} _ {u n s u p} = \frac {1}{\mathcal {B} _ {u}} \sum_ {i = 1} ^ {\mathcal {B} _ {u}} \frac {1}{\mathcal {H} \cdot \mathcal {W}} \sum_ {j = 1} ^ {\mathcal {H} \cdot \mathcal {W}} \mathcal {L} _ {c e} \left(p _ {i j} ^ {u}, \hat {y} _ {i j} ^ {u}\right), \tag {3}
$$
where  $\mathcal{B}_u$  denotes the number of unlabeled images in a training batch, an image size of  $\mathcal{H} \times \mathcal{W}$ ,  $j^{th}$  pixel on the  $i^{th}$  image, the student model's prediction  $p_{ij}^u$  with unlabeled input applied strong augmentation, and the corresponding pseudo-label  $\hat{y}_{ij}^u$  from the teacher models. The  $\lambda_u$  is a scalar hyper-parameter to adjust the unsupervised loss weight.
<sup>1</sup>The coarse region-level augmentation helps the model exploit contextual information, while the fine class-level augmentation enables differentiation between objects based on their unique properties.
One of the temporary teachers is activated alternately at each epoch and retains the student model's characteristics through the EMA weights. The  $k$ -th temporary teacher  $k \in \{1, \dots, t_n\}$  is alternately switched at each epoch; the parameters of  $k$ -th temporary teacher  $\theta_t^k$  are updated through EMA based on the student parameters  $\theta_s$  by Eq. (2).
3.3 Implicit Consistency Learning
We introduce another implicit ensemble learning from the viewpoint of consistency regularization to enhance our student model. Motivated by [14, 38, 10], we encourage a subset of layers to be active in the student model and all these sub-models to make consistent predictions. In contrast to the previous studies [38, 2] that enforce consistent predictions between the full model and sub-models within the same model, we impose the consistency between the sub-models of the student model and the full teacher models. We construct the sub-models within the student model via versatile stochastic depth [14], applicable from CNN models to transformer-based models. Therefore, the student model is trained through the following objective functions:
$$
\mathcal {L} _ {\text {c o n s}} = \frac {1}{\mathcal {B}} \sum_ {i = 1} ^ {\mathcal {B}} \frac {1}{\mathcal {H} \cdot \mathcal {W}} \sum_ {j = 1} ^ {\mathcal {H} \cdot \mathcal {W}} \mathcal {L} _ {\text {c e}} \left(p _ {i j} \left(\tilde {\theta} _ {s}\right), \hat {y} _ {i j} \left(\theta_ {t}\right)\right), \tag {4}
$$
where  $\tilde{\theta}_s$  denotes the parameters of sub-models of student model with a drop rate  $\tau$ , and  $\mathcal{B}$  is the number of images in a training batch. Note that we only apply weak augmentations to inputs for both teacher and student models, unlike in Eq. 3, where we feed strong augmented inputs to the student model and weak augmented inputs to the teacher model, respectively. Finally, this loss cooperates with the  $\mathcal{L}_{unsup}$  to update the student model.
In this section, we demonstrate the efficacy of our proposed method by conducting comprehensive comparisons with state-of-the-art approaches on various publicly available benchmarks. Furthermore, we provide additional ablation studies to justify the significance and impact of our method.
Datasets. We evaluate our method on three public benchmarks in semantic segmentation, including the PASCAL VOC [9], Cityscapes [8], and ADE20K [55] datasets.
PASCAL VOC [9] is a widely-used semantic segmentation benchmark that consists of 20 object categories in the foreground and a background category. It includes 1,464 and 1,449 images in the training and validation set, respectively. To evaluate our method, we follow recent protocols [46, 45, 41] and validate it on both the original high-quality set and the blended set, which consist of 10,582 training images. For the blended set, we follow the common practice [47, 13, 6] and make use of the SBD [11] augmented set, comprising 9,118 training images with their corresponding annotations.
Cityscapes [8] is designed for semantic understanding of urban street scenes, including 30 classes, of which only 19 classes are used for scene parsing evaluation. There are 5,000 finely annotated images and 20,000 coarsely annotated images. The fine-annotated images are split into training, validation, and test sets with 2,975, 500, and 1,525 images, respectively. Following the prior works [13, 45, 41], we evaluate our method on  $^{1/16}$ ,  $^{1/8}$ ,  $^{1/4}$ , and  $^{1/2}$  label partitions.
ADE20K [55] is a large-scale scene parsing dataset that covers 150 object and stuff categories. It consists of 25,574, 2,000, and 3,000 images in the training, validation, and test set, respectively. Due to the lack of standardized label partitions for semi-supervised segmentation scenarios, we randomly split the whole training set and construct  $\frac{1}{32}, \frac{1}{16}, \frac{1}{8}, \frac{1}{4}$ , and  $\frac{1}{2}$  label partitions. We have publicly released these partitions on our GitHub repository.
Model architectures. We leverage various architectures to demonstrate the versatility of our model-agnostic approach. We adopt DeepLabv3+ [5] based on ResNet-50/-101 backbones [12] for PASCAL VOC, and ResNet-101 for Cityscapes. In addition, we employ DeepLabv2 [4] based on ResNet-101 for unsupervised domain adaptation task. Aside from the CNN backbones, we further validate our method with transformer-based architecture SegFormer [43] with the MiT-B1.
Table 1: Comparison of mIoU (%) with state-of-the-art methods on PASCAL VOC 2012 under different partitions. Labeled images are sampled from the original high-quality training set. All methods are based on DeepLabv3+ with ResNet-50 backbone. We also report the total amount of trainable parameters in each method.
Method	#Params	1/16 (92)	1/8 (183)	1/4 (366)	1/2 (732)	Full (1,464)
Supervised-only	43.6M	44.03	52.26	61.65	66.72	72.94
PseudoSeg [56] [ICLR'21]	43.6M	54.89	61.88	64.85	70.42	71.00
PC2Seg [54] [ICCV'21]	43.6M	56.90	64.63	67.62	70.90	72.26
AugSeg [53] [CVPR'23]	43.6M	64.22	72.17	76.17	77.40	78.82
UniMatch [46] [CVPR'23]	43.6M	71.9	72.5	76.0	77.4	78.7
Ours	43.6M	70.76	74.53	76.43	77.68	78.15
Table 2: Comparison of mIoU (%) with state-of-the-art methods on PASCAL VOC 2012 under different partitions. Labeled images are sampled from the blender training set, which consists of 10,582 samples in total. All methods are based on DeepLabv3+ with ResNet-101 backbone.
Method	#Params	1/16 (662)	1/8 (1,323)	1/4 (2,646)	1/2 (5,291)
Supervised-only	62.6M	67.87	71.55	75.80	77.13
MT [36] [NeurIPS'17]	62.6M	70.51	71.53	73.02	76.58
CPS [6] [CVPR'21]	125.2M	74.48	76.44	77.68	78.64
AEL [13] [NeurIPS'21]	62.6M	77.20	77.57	78.06	80.29
U²PL [41] [CVPR'22]	64.5M	77.21	79.01	79.30	80.50
GTA-Seg [15] [NeurIPS'22]	127.1M	77.82	80.47	80.57	81.01
PCR [45] [NeurIPS'22]	64.5M	78.60	80.71	80.78	80.91
Ours	62.6M	78.82	81.19	81.03	80.62
Implementation details. For the training on PASCAL VOC, we employ the stochastic gradient descent (SGD) optimizer with an initial learning rate of 0.001 and weight decay of 0.001. We set a batch size to 16 and run 80 epochs. Following the previous protocol [46], we use an image resolution of  $321 \times 321$  for the original high-quality training set while configuring the blender set to  $513 \times 513$ . For Cityscapes, we also use SGD optimizer with an initial learning rate of 0.01, weight decay of 1e-4, batch size of 16, crop size of  $769 \times 769$ , and total training epochs of 200. For the training on the ADE20K, we follow the recipe of SegFormer [43, 7] and set the learning rate of 6e-5 and poly LR scheduler with a factor of 1.0. For a fair comparison, we validate the state-of-the-art methods [6, 41, 15, 24] using their official training codes with model architectures of SegFormer without any modifications. We train the methods with a batch size of 8 and crop size of  $512 \times 512$  for the MiT-B1 backbone. Following the previous work [41], we progressively increase EMA weights up to 0.99 across all experiments.