SlowGuess/NeurIPS_2023_OCR huggingface dataset:
- Pascal VOC [43] val. Under the SOLOv2 framework, our approach achieves  $37.1\%$  AP and  $38.4\%$  AP with 12 epochs and 36 epochs, respectively, outperforming other methods by  $1.4\% - 2.5\%$  mask AP with ResNet-50. With the Mask2Former framework, our approach also outperforms its counterparts. Furthermore, with the Swin-L backbone [55], our proposed approach achieves very promising performance,  $49.6\%$  mask AP with 50 epochs.
- COCO [49] val. Under the SOLOv2 framework, our approach achieves  $32.0\%$  AP and  $32.9\%$  AP with 12 epochs and 36 epochs, and surpasses its best counterpart by  $1.0\%$  AP and  $0.4\%$  AP using ResNet-50, respectively. Under the Mask2Former framework, our method still achieves the best performance with ResNet-50 backbone. Furthermore, equipped with stronger backbones, our approach obtains more robust performance, achieving  $38.0\%$  mask AP with ResNet-101, and  $41.0\%$  mask AP using Swin-L backbone.
Table 1: Quantitative results (§4.1) on Pascal VOC [43] and COCO val [49] with mask AP(%)
Method	Backbone	#Epoch	Pascal VOC	COCO
AP	\( AP_{50} \)	\( AP_{75} \)	AP	\( AP_{50} \)	\( AP_{75} \)
BBTP [NeuriIPS19] [16]	ResNet-101	12	23.1	54.1	17.1	21.1	45.5	17.2
BoxInst [CVPR21] [8]	ResNet-50	36	34.3	58.6	34.6	31.8	54.4	32.5
DiscoBox [ICCV21] [17]	ResNet-50	36	-	59.8	35.5	31.4	52.6	32.2
BoxLevelset [ECCV22] [9]	ResNet-50	36	36.3	64.2	35.9	31.4	53.7	31.8
SOLOv2 Framework
Pairwise Loss [CVPR21] [8]	ResNet-50	12	35.7	64.3	35.1	31.0	52.8	31.5
TreeEnergy Loss [CVPR22] [7]	ResNet-50	12	35.0	64.4	34.7	30.9	52.9	31.3
CRF Loss [CVPR23] [10]	ResNet-50	12	35.0	64.7	34.9	30.9	53.1	31.4
APro(0urs)	ResNet-50	12	37.1	65.1	37.0	32.0	53.4	32.9
Pairwise Loss [CVPR21] [8]	ResNet-50	36	36.5	63.4	38.1	32.4	54.5	33.4
TreeEnergy Loss [CVPR22] [7]	ResNet-50	36	36.1	63.5	36.1	31.4	54.0	31.2
CRF Loss [CVPR23] [10]	ResNet-50	36	35.9	64.0	35.7	32.5	54.9	33.2
APro(0urs)	ResNet-50	36	38.4	65.4	39.8	32.9	55.2	33.6
APro(0urs)	ResNet-101	36	40.5	67.9	42.6	34.3	57.0	35.3
Mask2Former Framework
Pairwise Loss [CVPR21] [8]	ResNet-50	12	35.2	62.9	33.9	33.8	57.1	34.0
TreeEnergy Loss [CVPR22] [7]	ResNet-50	12	36.0	65.0	34.3	33.5	56.7	33.7
CRF Loss [CVPR23] [10]	ResNet-50	12	35.7	64.3	35.2	33.5	57.5	33.8
APro(0urs)	ResNet-50	12	37.0	65.1	37.0	34.4	57.7	35.3
APro(0urs)	ResNet-50	50	42.3	70.6	44.5	36.1	62.0	36.7
APro(0urs)	ResNet-101	50	43.6	72.0	45.7	38.0	63.6	38.7
APro(0urs)	Swin-L	50	49.6	77.6	53.1	41.0	68.3	41.9
Qualitative Results. Fig. 3 illustrates the visual comparisons on affinity maps of our APro and other approaches, and Fig. 4 compares the segmentation results. One can clearly see that our method captures accurate pairwise affinity with object's topology and yields more fine-grained predictions.
4.2 Weakly-supervised Semantic Segmentation
Datasets. We conduct experiments on the widely-used Pascal VOC2012 dataset [51], which contains 20 object categories and a background class. As in [6, 7], the augmented Pascal VOC dataset is adopted here. The point [2] and scribble [5] annotations are employed for weakly point-supervised and scribble-supervised settings, respectively.
Implementation Details. As in [7], we adopt LTF [19] as the base segmentation model. The input size is  $512 \times 512$ . The SGD optimizer with momentum of 0.9 and weight decay of  $10^{-4}$  is used. The initial learning rate is 0.001, and there are 80k training iterations. The same data augmentations as in [7] are utilized. We employ the partial cross-entropy loss to make full use of the available
Table 2: Quantitative results (§4.2) on Pascal VOC2012 [51] val with mean IoU(%)
Method	Backbone	Supervision	CRF Post.	mIoU
†KernelCut Loss [ECCV18] [6]	DeepLabV2	Point	✓	57.0
*TEL [CVPR22] [7]	LTF	X	66.8
APro(0urs)	LTF	X	67.7
†NormCut Loss [CVPR18] [59]	DeepLabV2	Scribble	✓	74.5
†DenseCRF Loss [ECCV18] [6]	DeepLabV2	✓	75.0
†KernelCut Loss [ECCV18] [6]	DeepLabV2	✓	75.0
†GridCRF Loss [ICCV19] [36]	DeepLabV2	X	72.8
PSI [ICCV21] [36]	DeepLabV3	X	74.9
*TEL [CVPR22] [7]	LTF	X	76.2
APro(0urs)	LTF	X	76.6
†:adopting multi-stage training, *:our re-implementation.
point/scribble labels and constrain the unary term. ResNet-101 [54] pretrained on ImageNet [57] is adopted as backbone network for all methods.
Quantitative Results. As shown in Table 2, we compare our APro approach with the state-of-the-art methods on point-supervised and scribble-supervised semantic segmentation, respectively.
- Point-wise supervision. With DeepLabV2 [43], KernelCut Loss [6] achieves  $57.0\%$  mIoU. Equipped with LTF [19], TEL [7] achieves  $66.8\%$  mIoU. Our APro achieves  $67.7\%$  mIoU, outperforming the previous best method TEL [7] by  $0.9\%$  mIoU.
- Scribble-wise supervision. The scribble-supervised approaches are popular in weakly supervised semantic segmentation. We apply the proposed approach under the single-stage training framework without calling for CRF post-processing during testing. Compared with the state-of-the-art methods, our approach achieves better performance with  $76.6\%$  mIoU.
Figure 3: Visual comparisons of pairwise affinity maps based on the RGB image for a specific position (green star), including TreeFilter (TF) [19], local kernels with full image size (L#), and our presented LP and GP processes. The GP process can capture the long-range pairwise affinity with object's topology, while LP retrieves the local similarities. Our APro approach smoothes the noisy initial network predictions (init.) to obtain cleaner soft pseudo labels.
4.3 CLIP-guided Semantic Segmentation
Datasets. To more comprehensively evaluate our proposed approach, we conduct experiments on CLIP-guided annotation-free semantic segmentation with three widely used datasets:
Pascal VOC 2012 [51] introduced in Section 4.2.
- Pascal Context [60], which contains 59 foreground classes and a background class with 4,996 train images and 5,104 val images.
- COCO-Stuff [61], which has 171 common semantic object/stuff classes on 164K images, containing 118,287 train images and 5,000 val images.
Base Architectures and Backbones. We employ MaskCLIP+ [40] as our base architecture, which leverages the semantic priors of pretrained CLIP [38] model to achieve the annotation-free dense semantic segmentation. In the experiments, we couple MaskCLIP+ with our APro approach under ResNet-50, ResNet-  $50\times 16$  and ViT-B/16 [62]. The dense semantic predictions of MaskCLIP [40] are used as the unary term, and our proposed method can refine it and generate more accurate pseudo labels for training target networks.
Implementation Details. For fair comparison, we keep the same settings as MaskCLIP+ [40]. We keep the text encoder of CLIP unchanged and take prompts with target classes as the input. For text embedding, we feed prompt engineered texts into the text encoder of CLIP with 85 prompt templates, and average the results with the same class. For ViT-B/16, the bicubic interpolation is adopted for the pretrained positional embeddings. The initial learning rate is set to  $10^{-4}$ . We train all models with batch size 32 and 2k/4k/8k iterations. DeepLabv2-ResNet101 is used as the backbone.
Quantitative Results. Table 3 compares our approach with MaskCLIP+ [40] for annotaion-free semantic segmentation. We have the following observations.
- Pascal VOC2012 [51] val. With ResNet-50 as the image encoder in pretrained CLIP model, our approach
Table 3: Quantitative results (\$4.3) on Pascal VOC2012 [51] val, Pascal Context [60] val, and COCO-Stuff [61] val with mean IoU (%).
Method	CLIP Model	VOC2012	Context	COCO.
MaskCLIP+ [ECCV22] [40]	ResNet-50	58.0	23.9	13.6
APro(Ours)	61.6↑3.6	25.4↑1.5	14.6↑1.0
MaskCLIP+ [ECCV22] [40]	ResNet-50×16	67.5	25.2	17.3
APro(Ours)	70.4↑2.9	26.5↑1.3	18.2↑0.9
MaskCLIP+ [ECCV22] [40]	ViT-B/16	73.6	31.1	18.0
APro(Ours)	75.1↑1.5	32.6↑1.5	19.5↑1.5
outperforms MaskCLIP+ by  $3.6\%$  mIoU. With ResNet-  $50\times 16$  and ViT-B/16 as the image encoders, our method surpasses MaskCLIP+ by  $2.9\%$  and  $1.5\%$  mIoU, respectively.
- Pascal Context [60] val. Our proposed method outperforms MaskCLIP+ consistently with different image encoders (about +1.5% mIoU).
- COCO-Stuff [61] val. COCO-Stuff consists of hundreds of semantic categories. Our method still brings  $+1.0\%$ ,  $+0.9\%$  and  $+1.5\%$  performance gains over MaskCLIP+ with ResNet-50, ResNet-50×16 and ViT-B/16 image encoders, respectively.
only GP
only LP
APro
Figure 4: Qualitative results with different pairwise affinity terms on weakly supervised instance segmentation. Our method only with GP process preserves details without local consistency, while the model with only LP process encourages the local smoothness without topology-wise details. Our APro approach yields high-quality predictions with fine-grained details.
4.4 Diagnostic Experiments
For in-depth analysis, we conduct ablation studies on Pascal VOC [51] upon the weakly box-supervised instance segmentation task.
Unary and Pairwise Terms. Table 4 shows the evaluation results with different unary and pairwise terms. When using the unary term only, our method achieves  $25.9\%$  AP. When the global pairwise term is employed, our method achieves a much better performance
Table 4: Effects of unary and pairwise terms.
Unary	Global Pairwise	Local Pairwise	AP	\( AP_{50} \)	\( AP_{75} \)
✓	25.9	57.0	20.4
✓	✓	36.3	63.9	37.0
✓	✓	36.0	64.3	35.6
✓	✓	✓	38.4	65.4	39.8
of  $36.3\%$  AP. Using the local pairwise term only, our method obtains  $36.0\%$  AP. When both the global and local pairwise terms are adopted, our method achieves the best performance of  $38.4\%$  AP.
Tree-based Long-range Affinity Modeling. The previous works [19, 7] explore tree-based filters for pairwise relationship modeling. Table 5 compares our method with them. TreeFilter can capture the relationship with dis
Table 5: Comparisons with tree-based methods.
Method	AP	\( AP_{50} \)	\( AP_{75} \)
TreeFilter [19]	36.1	63.5	36.1
TreeFilter [19] + Local Pairwise	36.8	64.4	36.5
Global + Local Pairwise (Ours)	38.4	65.4	39.8
tant nodes to a certain extent (see Fig. 3). Directly using TreeFilter as the pairwise term leads to  $36.1\%$  AP. By combining TreeFilter with our local pairwise term, the model obtains  $36.8\%$  AP. In comparison, our proposed approach achieves  $38.4\%$  AP.
Iterated Local Affinity Modeling. We evaluate our local affinity propagation (LP) with different iterations, and compare it with the classical MeanField method [10, 42]. Table 6 reports the comparison results. Our APro with the LP process achieves  $36.0\%$  AP after 20 iterations. However, replacing our local affinity propagation with MeanFiled-based method [10] costs 100 iterations to obtain  $35.9\%$  AP. This indicates that our LP method possesses the attribute of fast convergence.
Soft Pseudo-label Generation. With the formulated GP and LP methods, we study how to integrate them to generate the soft pseudolabels in Table 7. We can cascade GP and LP sequentially to refine the pseudo labels. Putting GP before LP (denoted as GP-LP-C) achieves  $36.8\%$  AP, and putting LP before GP (denoted as LPGP-C) performs better with  $37.7\%$  AP. In addition, we can use
Table 6: Comparisons on local pairwise affinity modeling.
LP(Ours)	MeanField[10]
Iteration	AP	Iteration	AP
10	35.8	20	35.2
20	36.0	30	35.5
30	35.7	50	35.5
50	35.6	100	35.9
Table 7: Generation of soft pseudo labels.
Method	AP	\( AP_{50} \)	\( AP_{75} \)
GP-LP-C	36.8	63.7	37.8
LP-GP-C	37.7	65.1	39.1
GP-LP-P	38.4	65.4	39.8
GP and LP in parallel (denoted as GP-LP-P) to produce two pseudo labels, and employ both of them to optimize the segmentation network with  $L_{1}$  distance. Notably, GP-LP-P achieves the best performance with  $38.4\%$  mask AP. This indicates that our proposed affinity propagation in global and local views are complementary for optimizing the segmentation network.
Runtime Analysis. We report the average runtime of our method in Table 8. The experiment is conducted on a single GeForce RTX 3090 with batch size 1. Here we report the average runtime for one GP process duration of an epoch on the Pascal VOC dataset. When directly using Breadth First Search for each node with  $N$  times, the runtime is  $4.3 \times 10^{3}$  ms with  $\mathcal{O}(N^2)$  time complexity. While employing the
Table 8: Average runtime (ms) with and without the efficient implementation.
Effic. Imple.	Ave. Runtime
X	4.3×103
✓	0.8
The proposed efficient implementation, the runtime is only  $0.8\mathrm{ms}$  with  $\mathcal{O}(N\log N)$  time complexity. This demonstrates that the proposed efficient implementation reduces the computational complexity dramatically.
In this work, we proposed a novel universal component for weakly-supervised segmentation by formulating it as an affinity propagation process. A global and a local pairwise affinity term were introduced to generate the accurate soft pseudo labels. An efficient implementation with the light computational overhead was developed. The proposed approach, termed as APro, can be embedded into the existing segmentation networks for label-efficient segmentation. Experiments on three typical label-efficient segmentation tasks, i.e., box-supervised instance segmentation, point/scribble-supervised semantic segmentation and CLIP-guided annotation-free semantic segmentation, proved the effectiveness of proposed method.
This work is supported by National Natural Science Foundation of China under Grants (61831015). It is also supported by the Information Technology Center and State Key Lab of CAD&CG, Zhejiang University.
[1] Wei Shen, Zelin Peng, Xuehui Wang, Huayu Wang, Jiazhong Cen, Dongsheng Jiang, Lingxi Xie, Xiaokang Yang, and Q Tian. A survey on label-efficient deep image segmentation: Bridging the gap between weak supervision and dense prediction. TPAMI, 2023. 1, 2
[2] Amy Bearman, Olga Russakovsky, Vittorio Ferrari, and Li Fei-Fei. What's the point: Semantic segmentation with point supervision. In ECCV, pages 549–565, 2016. 1, 2, 7
[3] Junsong Fan, Zhaoxiang Zhang, and Tieniu Tan. Pointly-supervised panoptic segmentation. In ECCV, pages 319-336, 2022. 1, 3
[4] Wentong Li, Yuqian Yuan, Song Wang, Jianke Zhu, Jianshu Li, Jian Liu, and Lei Zhang. Point2mask: Point-supervised panoptic segmentation via optimal transport. In ICCV, pages 572-581, 2023. 1, 3
[5] Di Lin, Jifeng Dai, Jiaya Jia, Kaiming He, and Jian Sun. Scribblesup: Scribble-supervised convolutional networks for semantic segmentation. In CVPR, pages 3159-3167, 2016. 1, 2, 7
[6] Meng Tang, Federico Perazzi, Abdelaziz Djelouah, Ismail Ben Ayed, Christopher Schroers, and Yuri Boykov. On regularized losses for weakly-supervised cnn segmentation. In ECCV, pages 507-522, 2018. 2, 7
[7] Zhiyuan Liang, Tiancai Wang, Xiangyu Zhang, Jian Sun, and Jianbing Shen. Tree energy loss: Towards sparsely annotated semantic segmentation. In CVPR, pages 16907-16916, 2022. 1, 2, 3, 6, 7, 9
[8] Zhi Tian, Chunhua Shen, Xinlong Wang, and Hao Chen. Boxinst: High-performance instance segmentation with box annotations. In CVPR, pages 5443-5452, 2021. 1, 2, 3, 4, 5, 6, 7
[9] Wentong Li, Wenyu Liu, Jianke Zhu, Miaomiao Cui, Xian-Sheng Hua, and Lei Zhang. Box-supervised instance segmentation with level set evolution. In ECCV, pages 1-18, 2022. 2, 5, 6, 7
[10] Shiyi Lan, Xitong Yang, Zhiding Yu, Zuxuan Wu, Jose M Alvarez, and Anima Anandkumar. Vision transformers are good mask auto-labelers. In CVPR, 2023. 1, 2, 3, 4, 5, 6, 7, 9
[11] Wentong Li, Wenyu Liu, Jianke Zhu, Miaomiao Cui, Risheng Yu, Xiansheng Hua, and Lei Zhang. Box2mask: Box-supervised instance segmentation via level-set evolution. arXiv preprint arXiv:2212.01579, 2022. 1, 2, 6
[55] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, pages 10012-10022, 2021. 6
[56] Kai Chen, Jiaqi Wang, Jiangmiao Pang, Yuhang Cao, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, et al. Mmdetection: Open mmlab detection toolbox and benchmark. arXiv preprint arXiv:1906.07155, 2019. 6
[57] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet: large scale visual recognition challenge. IJCV, 115:211-252, 2015. 6, 7
[58] Golnaz Ghiasi, Yin Cui, Aravind Srinivas, Rui Qian, Tsung-Yi Lin, Ekin D Cubuk, Quoc V Le, and Barret Zoph. Simple copy-paste is a strong data augmentation method for instance segmentation. In CVPR, pages 2918-2928, 2021. 6
[59] Meng Tang, Abdelaziz Djelouah, Federico Perazzi, Yuri Boykov, and Christopher Schroers. Normalized cut loss for weakly-supervised cnn segmentation. In CVPR, pages 1818-1827, 2018. 7
[60] Roozbeh Mottaghi, Xianjie Chen, Xiaobai Liu, Nam-Gyu Cho, Seong-Whan Lee, Sanja Fidler, Raquel Urtasun, and Alan Yuille. The role of context for object detection and semantic segmentation in the wild. In CVPR, pages 891–898, 2014. 8
[61] Holger Caesar, Jasper Uijlings, and Vittorio Ferrari. Coco-stuff: Thing and stuff classes in context. In CVPR, 2018. 8
[62] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2020. 8