SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where  $\hat{\mathbf{y}}_{ij}^{u} = \mathrm{argmax}(\mathrm{DA}(\mathbf{y}_{ij}^{u}))$ . In this way, the distribution-aware weighting function is rewarded with better generalization, benefiting from more equal learning of labeled and unlabeled data, mitigating the issue of distribution imbalance, and enjoying the synergy. The algorithm flow is shown in the supplementary material.
Datasets: (1) PASCAL VOC 2012 [29] is an object-centric semantic segmentation dataset, containing 21 classes with 1,464 and 1,449 finely annotated images for training and validation, respectively. Some researches [30, 19] augment the original training set (e.g., classic) by incorporating the coarsely annotated images in SBD [31], obtaining a training set (e.g., blender) with 10,582 labeled samples. (2) Cityscapes [32] is an urban scene understanding dataset with 2,975 images for training and 500 images for validation.
Implementation Details: For a fair comparison, we follow the common practice and use ResNet [33] as our backbone and DeepLabv3+[34] as the decoder. We set the crop size as  $513 \times 513$  for PASCAL and  $801 \times 801$  for Cityscapes, respectively. For both datasets, we adopt SGD as the optimizer with the same batch size of 16 and different initial learning rate, which is set as 0.001 and 0.005 for PASCAL and Cityscapes. We use the polynomial policy to dynamically decay the learning rate along the whole
Table 2: Quantitative results of different SSL methods on Pascal classic and blender set. We report mIoU (%) under various partition protocols and show the improvements over Sup.-only baseline. The best is highlighted in bold.
Method	Class	Blender
1/16(92)	1/8(183)	1/4(366)	1/2(732)	Full(1464)	1/16(662)	1/8(1323)	1/4(2646)
ResNet-50	Sup.-only	44.0	52.3	61.7	66.7	72.9	62.4	68.2	72.3
Pseudo-Label[ICML'13] [10]	55.7	60.2	65.6	69.7	74.8	66.3	70.8	74.5
FixMatch[NeurIPS'20] [18]	60.1	67.3	71.4	73.7	76.9	70.6	73.9	75.1
iMAS[CVPR'23] [24]	-	-	-	-	-	74.8	76.5	77.0
AugSeg[CVPR'23] [25]	64.2	72.2	76.2	77.4	78.8	74.7	76.0	77.2
ResNet-101	DAW (Ours)	68.5	73.1	76.3	78.6	79.7	76.2	77.6	77.4
Δ↑	+24.5	+20.8	+14.6	+11.9	+6.8	+13.8	+9.4	+5.1
Sup.-only	45.1	55.3	64.8	69.7	73.5	67.5	71.1	74.2
Pseudo-Label[ICML'13] [10]	57.3	64.1	69.4	73.3	77.2	69.1	73.8	76.7
FixMatch[NeurIPS'20] [18]	63.9	73.0	75.5	77.8	79.2	74.3	76.3	76.9
CPS[CVPR'21] [26]	64.1	67.4	71.7	75.9	-	74.5	76.4	77.7
AEL[NeurIPS'21] [27]	-	-	-	-	-	77.2	77.6	78.1
iMAS[CVPR'23] [24]	68.8	74.4	78.5	79.5	81.2	76.5	77.9	78.1
AugSeg[CVPR'23] [25]	71.1	75.5	78.8	80.3	81.4	77.0	77.3	78.8
CCVC[CVPR'23] [28]	70.2	74.4	77.4	79.1	80.5	77.2	78.4	79.0
ResNet-101	DAW (Ours)	74.8	77.4	79.5	80.6	81.5	78.5	78.9	79.6
Δ↑	+29.7	+22.1	+14.7	+10.9	+8.0	+11.0	+7.8	+5.4
Table 3: Quantitative results of different SSL methods on Cityscapes. We report mIoU (\%) under various partition protocols and show the improvements over Sup.-only baseline. The best is highlighted in bold.
Method	ResNet-50	ResNet-101
1/16(186)	1/8(372)	1/4(744)	1/2(1488)	1/16(186)	1/8(372)	1/4(744)	1/2(1488)
Sup.-only	63.3	70.2	73.1	76.6	66.3	72.8	75.0	78.0
Pseudo-Label[ICML'13] [10]	67.2	72.4	74.9	77.4	68.9	74.3	76.8	78.6
FixMatch[NeurIPS'20] [18]	72.6	75.7	76.8	78.2	74.2	76.2	77.2	78.4
AEL[NeurIPS'21] [27]	74.0	75.8	76.2	-	75.8	77.9	79.0	80.3
PCR[NeurIPS'22] [2]	-	-	-	-	73.4	76.3	78.4	79.1
GTA-Seg[NeurIPS'23] [3]	63.0	69.4	72.0	76.1	69.4	72.0	76.1	-
iMAS[CVPR'23] [24]	74.3	77.4	78.1	79.3	-	-	-	-
AugSeg[CVPR'23] [25]	73.7	76.5	78.8	79.3	75.2	77.8	79.6	80.4
DAW (Ours)	75.2	77.5	79.1	79.5	76.6	78.4	79.8	80.6
Δ↑	+11.9	+7.3	+6.0	+2.9	+10.3	+5.6	+4.8	+2.6
training and assemble the channel dropout perturbation [22] to improve the generalization ability of the model. We train the model for 80 epochs on PASCAL and 240 epochs on Cityscapes, using  $8 \times$  NVIDIA GeForce RTX 3090 GPUs.
3.2 Comparison with State-of-the-art Methods
We conduct experiments on two popular benchmarks including PASCAL VOC 2012 and Cityscapes and make a fair comparison with SOTA semi-supervised semantic segmentation methods. We consistently observe that our DAW outperforms all other methods under all partition protocols on all datasets with different backbones, which strongly proves the effectiveness of our method.
Results on PASCAL VOC 2012 Dataset. Table 2 shows the comparison of our method with the SOTA methods on PASCAL classic and blender set. Specifically, on the PASCAL classic set, our method outperforms the supervised-only (Sup.-only) model by  $29.7\%$ ,  $22.1\%$ ,  $14.7\%$ ,  $10.9\%$  under the partition protocols of 1/16, 1/8, 1/4 and 1/2, respectively with ResNet-101. Our method also significantly outperforms the existing semi-supervised SOTA methods under all data partition protocols. Taking the recently proposed method AugSeg [25] as an example, the performance gain of our approach reaches to  $+4.3\%$  under 1/16 partition protocol with ResNet-50. The same superiority of our method can also be observed on the PASCAL blender set.
Results on Cityscapes Dataset. Table 3 compares DAW with SOTA methods on the Cityscapes dataset. DAW achieves consistent performance gains over the Sup.-only baseline, obtaining improvements of  $11.9\%$ ,  $7.3\%$ ,  $6.0\%$  and  $2.9\%$  under 1/16, 1/8, 1/4 and 1/2 partition protocols with ResNet-50, respectively. We can also see that over all protocols, DAW outperforms the SOTA methods, e.g., DAW excels to iMAS [24] by  $1.1\%$  under the 1/16 partition with ResNet-101.
Table 4: Ablation studies of different components. Note that "Fixed" denotes the result of UniMatch.
None	Fixed	DAWF	DA	mIoU
✓	62.7
✓	66.9
✓	68.0
✓	✓	68.5
Table 5: Ablation studies of different momentum on PASCAL classic 92.
m	mIoU
0.99	68.5
0.999	68.1
0.9999	67.9
Figure 2: The curve of Pos.&Neg. distribution and  $t^*$  during training.
Pseudo-Label
FixMatch
Figure 3: Qualitative comparison with different methods. Note that significant improvements are marked with yellow boxes.
AEL
Qualitative Results. We compare qualitative results of our DAW with different SOTA methods. As shown in Figure 3, DAW also shows more powerful segmentation performance in fine-grained details (see the first and second row in Figure 3). With the help of the optimal weighting function, DAW demonstrates superior abilities in most scenarios.
3.3 Ablation Study and Analysis
To look deeper into our method, we perform a series of ablation studies on PASCAL classic set under 92 partition protocol with ResNet-50 to analyze each component of our DAW, including the Distribution-Aware Weighting Function (DAWF) and the Distribution Alignment (DA). The baseline method is UniMatch [22].
Effectiveness of Components. In Table 4, "None" denotes there is no threshold for pseudo-label during the training (i.e., Pseudo-Label [10]) while "Fixed" indicates that a fixed threshold is set (i.e., UiMatch [22]). A certain performance lift compared with the baseline can be observed owing to
Figure 4: Comparison between distributions of different methods at different training epochs (e.g., epoch 1 vs. epoch 79).
w/o DA
Figure 5: Viz. of Dist. w&w/o DA.
EM Image
Ground Truth
Figure 6: Visualization of Lucchi dataset.
Table 6: Quantitative results of different SSL methods on Lucchi dataset. We report mIoU (%) under various partition protocols. The best is highlighted in bold.
Method	1/32(5)	1/16(10)	1/8(20)
Sup.-only	45.7	57.4	61.8
MT [35]	71.8	72.4	75.4
CCT [36]	84.7	85.4	85.8
CPS [30]	84.5	84.6	85.8
DAW (Ours)	85.9	86.6	87.6
the introduction of Distribution-Aware Weighting Function and Distribution Alignment. (1) The utilization of DAWF brings a  $1.1\%$  improvement of mIoU, demonstrating that the negative impact raised by the  $E1 + E2$  trade-off is effectively alleviated. (2) DA brings further accuracy gains, indicating that the existence of the discrepancy between the distributions of labeled and unlabeled data may cause a bottleneck in learning. For better visualization, we employ the unused annotations of "unlabeled data" to calculate the ground-truth distribution on the unlabeled data. And as shown in Figure 5, there is a relatively large gap between the distributions, and DA can effectively relieve it, further unlocking the potential of the weighting function and enjoy the synergy.
Hyperparameter Evaluations. As shown in Table 5, it can be observed that the performance is optimal with  $m = 0.99$ .
Scalability for Other Scenarios. We further conduct extra experiments on Lucchi [37-42] to evaluate the scalability of our method. Figure 6 shows the image and ground-truth of Lucchi dataset, presenting a common problem in electron microscope images that the instances are very small and scattered. This calls for more reliable supervision in training under a semi-supervised setting. As shown in Table 6, DAW outperforms other competitive methods in the electron microscopy domain, indicating that our method can provide more reliable supervision.
Comparison of Distribution. As shown in Figure 4, the distributions of different methods are almost the same. As the learning goes on, the discrepancy between the positive and negative distributions of ours becomes larger (simultaneously shown in Figure 2) while the others almost no change. A large discrepancy between the positive and negative distributions means that we can filter out as many negative samples while recruiting as many positive samples as possible, which is conducive to model training. This is the fundamental reason behind why our method outperforms other methods.
Semi-Supervised Learning. Semantic segmentation is a fundamental task that has achieved conspicuous achievements credited to the recent advances in deep neural networks [43-48]. However, its data-driven nature makes it heavily dependent on massive pixel-level annotations, which are laborious and time-consuming to gather. To alleviate the data-hunger issue, considerable works have turned their attention to semi-supervised learning. Designing appropriate and effective supervision signals for unlabelled data is the core problem of semi-supervised learning. Previous research can be summarized into two learning schemes: self-training and consistency regularization. Self-training based methods [49, 12, 10, 50] aim to train the model based on the pseudo-labels generated by the up-to-date optimized model for the unlabelled data. Consistency regularization-based methods aim to obtain prediction invariance under various perturbations, including input perturbation [51, 52], feature perturbation [53] and network perturbation [54-57]. The recently semi-supervised methods, MixMathch [58] and FixMatch [18] combine these two techniques together and achieve state-of-the-art performance. Based on the paradigm of existing semi-supervised learning methods, our method explores a better weighting function for the pseudo-label scheme during training.
Semi-Supervised Semantic Segmentation. Semi-supervised semantic segmentation aims at pixel-level classification with limited labeled data. Recently, following the paradigm of semi-supervised learning, many semi-supervised semantic segmentation methods also focus on the design of self-training [26, 27, 5] and consistency regularization [59, 60, 53, 61] strategies.  $\mathrm{U}^2\mathrm{PL}$  [5] proposes to
make sufficient use of unreliable pseudo-labeled data. CCT [53] adopts a feature-level perturbation and enforces consistency among the predictions from different decoders. More recently, SOTA semi-supervised segmentation methods also integrate both technologies for better performance. PseudoSeg [7], AEL [27] and UCC [62] propose to use the pseudo-labels generated from weak augmented images to constrain the predictions of strong augmented images. In this paper, we shed light on semi-supervised semantic segmentation based on pseudo-labeling and strive to explore better strategies for using pseudo-labels.
In this paper, we propose DAW to systematically analyze the trade-off in previous methods that hinder the model's learning. We formally define the trade-off between inaccurate yet utilized pseudo-labels, and correct yet discarded pseudo-labels by explicitly modeling the confidence distribution of correct and inaccurate pseudo-labels, equipped with a unified weighting function. Experiments show the effectiveness.
This work was partially supported by the National Defense Basic Scientific Research Program (Grant JCKY2020903B002).
[1] Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3431-3440, 2015.
[2] Haiming Xu, Lingqiao Liu, Qiuchen Bian, and Zhen Yang. Semi-supervised semantic segmentation with prototype-based consistency regularization. Advances in Neural Information Processing Systems, 35:26007-26020, 2022.
[3] Ying Jin, Jiaqi Wang, and Dahua Lin. Semi-supervised semantic segmentation via gentle teaching assistant. Advances in Neural Information Processing Systems, 35:2803-2816, 2022.
[4] Lihe Yang, Wei Zhuo, Lei Qi, Yinghuan Shi, and Yang Gao. St++: Make self-training work better for semi-supervised semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4268-4277, 2022.
[5] Yuchao Wang, Haochen Wang, Yujun Shen, Jingjing Fei, Wei Li, Guoqiang Jin, Liwei Wu, Rui Zhao, and Xinyi Le. Semi-supervised semantic segmentation using unreliable pseudo-labels. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4248-4257, 2022.
[6] Jie Qin, Jie Wu, Ming Li, Xuefeng Xiao, Min Zheng, and Xingang Wang. Multi-granularity distillation scheme towards lightweight semi-supervised semantic segmentation. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXX, pages 481-498. Springer, 2022.
[7] Yuliang Zou, Zizhao Zhang, Han Zhang, Chun-Liang Li, Xiao Bian, Jia-Bin Huang, and Tomas Pfister. Pseudoseg: Designing pseudo labels for semantic segmentation. arXiv preprint arXiv:2010.09713, 2020.
[8] Mennatullah Siam, Mostafa Gamal, Moemen Abdel-Razek, Senthil Yogamani, Martin Jagersand, and Hong Zhang. A comparative study of real-time semantic segmentation for autonomous driving. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pages 587-597, 2018.
[9] Saeid Asgari Taghanaki, Kumar Abhishek, Joseph Paul Cohen, Julien Cohen-Adad, and Ghassan Hamarneh. Deep semantic segmentation of natural and medical images: a review. Artificial Intelligence Review, 54:137–178, 2021.
[59] Xin Lai, Zhuotao Tian, Li Jiang, Shu Liu, Hengshuang Zhao, Liwei Wang, and Jiaya Jia. Semi-supervised semantic segmentation with directional context-aware consistency. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1205-1214, 2021.
[60] Yuanyi Zhong, Bodi Yuan, Hong Wu, Zhiqiang Yuan, Jian Peng, and Yu-Xiong Wang. Pixel contrastive-consistent semi-supervised semantic segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7273-7282, 2021.
[61] Zenggui Chen and Zhouhui Lian. Semi-supervised semantic segmentation via prototypical contrastive learning. In Proceedings of the 30th ACM International Conference on Multimedia, pages 6696-6705, 2022.
[62] Jiashuo Fan, Bin Gao, Huan Jin, and Lihui Jiang. Ucc: Uncertainty guided cross-head co-training for semi-supervised semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9947-9956, 2022.