SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
L = \frac {1}{N _ {s}} \sum_ {\left(\mathbf {X} _ {i}, \mathbf {Y} _ {i}\right) \sim \mathcal {S}} L _ {p c e} \left(\mathbf {Y} _ {i} ^ {\prime}, \mathbf {Y} _ {i}\right) + \hat {\mathbf {Y}} _ {i} L _ {c e} \left(\mathbf {Y} _ {i} ^ {\prime}, \tilde {\mathbf {M}} _ {i}\right) + \hat {\mathbf {Y}} _ {i} L _ {I o U} \left(\mathbf {Y} _ {i} ^ {\prime}, \tilde {\mathbf {M}} _ {i}\right). \tag {11}
$$
Where the first term is the partial cross-entropy loss  $L_{pce}$  used to ensure consistency between the prediction maps and the sparse annotations  $\mathbf{Y}_i$  [34]. The second and third terms are the cross-entropy loss  $L_{ce}$  and the intersection-over-union loss  $L_{IoU}$ , both calculated using the pseudo label  $\hat{\mathbf{Y}}_i$  [35].
Implementation details. The image encoder uses ResNet50 as the backbone and is pre-trained on ImageNet [39]. The batch size is 36 and the learning rate is initialized as 0.0001, decreased by 0.1 every 80 epochs. For scribble supervision, we propose a nine-box strategy, namely constructing
Table 1: Results on COD with point supervision and scribble supervision.  $\mathrm{SCOD + }$  indicates integrating SCOD with our WS-SAM framework. The best two results are in red and blue fonts.
Methods	Pub.	CHAMELEON	CAMO	COD10K	NC4K
M↓	Fβ↑	Eφ↑	Sα↑	M↓	Fβ↑	Eφ↑	Sα↑	M↓	Fβ↑	Eφ↑	Sα↑	M↓	Fβ↑	Eφ↑	Sα↑
Scribble Supervision
SAM [19]	—	0.207	0.595	0.647	0.635	0.160	0.597	0.639	0.643	0.093	0.673	0.737	0.730	0.118	0.675	0.723	0.717
SAM-S [19]	—	0.076	0.729	0.820	0.650	0.105	0.682	0.774	0.731	0.046	0.695	0.828	0.772	0.071	0.747	0.832	0.763
WSSA [36]	CVPR20	0.067	0.692	0.860	0.782	0.118	0.615	0.786	0.696	0.071	0.536	0.770	0.684	0.091	0.657	0.779	0.761
SCWS [37]	AAAI21	0.053	0.758	0.881	0.792	0.102	0.658	0.795	0.713	0.055	0.602	0.805	0.710	0.073	0.723	0.814	0.784
TEL [38]	CVPR22	0.073	0.708	0.827	0.785	0.104	0.681	0.797	0.717	0.057	0.633	0.826	0.724	0.075	0.754	0.832	0.782
SCOD [30]	AAAI23	0.046	0.791	0.897	0.818	0.092	0.709	0.815	0.735	0.049	0.637	0.832	0.733	0.064	0.751	0.853	0.779
SCOD+	—	0.046	0.797	0.900	0.820	0.090	0.716	0.818	0.741	0.047	0.650	0.845	0.742	0.060	0.766	0.862	0.785
Ours	—	0.046	0.777	0.897	0.824	0.092	0.742	0.818	0.759	0.038	0.719	0.878	0.803	0.052	0.802	0.886	0.829
Point Supervision
SAM [19]	—	0.207	0.595	0.647	0.635	0.160	0.597	0.639	0.643	0.093	0.673	0.737	0.723	0.118	0.675	0.723	0.717
SAM-P [19]	—	0.101	0.696	0.745	0.697	0.123	0.649	0.693	0.677	0.069	0.694	0.796	0.765	0.082	0.728	0.786	0.776
WSSA [36]	CVPR20	0.105	0.660	0.712	0.711	0.148	0.607	0.652	0.649	0.087	0.509	0.733	0.642	0.104	0.688	0.756	0.743
SCWS [37]	AAAI21	0.097	0.684	0.739	0.714	0.142	0.624	0.672	0.687	0.082	0.593	0.777	0.738	0.098	0.695	0.767	0.754
TEL [38]	CVPR22	0.094	0.712	0.751	0.746	0.133	0.662	0.674	0.645	0.063	0.623	0.803	0.727	0.085	0.725	0.795	0.766
SCOD [30]	AAAI23	0.092	0.688	0.746	0.725	0.137	0.629	0.688	0.663	0.060	0.607	0.802	0.711	0.080	0.744	0.796	0.758
SCOD+	—	0.089	0.704	0.757	0.731	0.129	0.642	0.693	0.666	0.058	0.618	0.812	0.719	0.075	0.767	0.825	0.771
Ours	—	0.056	0.767	0.868	0.805	0.102	0.703	0.757	0.718	0.039	0.698	0.856	0.790	0.057	0.801	0.859	0.813
Table 2: Results for PIS and TOD with point supervision.
Methods	Polyp Image Segmentation (PIS)	Transparent Object Detection (TOD)
CVC-ColonDB	ETIS	Kvasir	GDD	GSD
M↓	Fβ↑	Eφ↑	Sα↑	M↓	Fβ↑	Eφ↑	Sα↑	M↓	Fβ↑	Eφ↑	Sα↑	M↓	Fβ↑	Eφ↑	Sα↑	M↓	Fβ↑	Eφ↑	Sα↑
SAM [19]	0.479	0.343	0.419	0.427	0.429	0.439	0.512	0.503	0.320	0.545	0.564	0.582	0.245	0.512	0.530	0.551	0.266	0.473	0.501	0.514
SAM-P [19]	0.194	0.587	0.664	0.671	0.144	0.625	0.719	0.715	0.108	0.793	0.811	0.802	0.164	0.668	0.715	0.625	0.177	0.687	0.730	0.668
WSSA [36]	0.127	0.645	0.732	0.713	0.123	0.647	0.733	0.762	0.082	0.822	0.852	0.828	0.173	0.652	0.710	0.616	0.185	0.661	0.712	0.650
SCWS [37]	0.082	0.674	0.758	0.787	0.085	0.646	0.768	0.731	0.078	0.837	0.860	0.831	0.170	0.631	0.702	0.613	0.172	0.706	0.738	0.673
TEL [38]	0.089	0.669	0.743	0.761	0.083	0.639	0.776	0.726	0.091	0.810	0.826	0.804	0.230	0.640	0.586	0.536	0.275	0.571	0.501	0.495
SCOD [30]	0.077	0.691	0.795	0.802	0.071	0.664	0.802	0.766	0.071	0.853	0.877	0.836	0.146	0.801	0.778	0.723	0.154	0.743	0.751	0.710
SCOD+	0.074	0.702	0.806	0.803	0.066	0.670	0.811	0.769	0.068	0.860	0.880	0.836	0.129	0.818	0.796	0.732	0.145	0.761	0.765	0.720
Ours	0.043	0.721	0.839	0.816	0.037	0.694	0.849	0.797	0.046	0.878	0.917	0.877	0.078	0.858	0.863	0.775	0.089	0.839	0.841	0.764
the minimum outer wrapping rectangle of the foreground/background scribble and dividing it into a nine-box grid, to sample one point in each box and send them to SAM for segmentation mask generation. Following [2], all images are resized as  $352 \times 352$  in both the training and testing phases. For SAM [19], we adopt the ViT-H SAM model to generate segmentation masks. We implement our method with PyTorch and run experiments on two RTX3090 GPUs.
Baselines. We explore SAM [19] for the WSCOS task by generating segmentation masks with sparse annotations as prompts and using the segmentation masks to train a COS segmenter. However, a more straightforward way to explore SAM for this task is to use the sparse annotation to fine-tune SAM and then directly apply SAM for the test. To verify the advantages of our method over this direct way, we construct two baseline methods, SAM-S and SAM-P, which fine-tune the mask decoder of SAM with scribble and point supervisions, respectively, by the partial cross-entropy loss. We will show the results of these two baselines in our comparative evaluations. For reference, we also report the results of the vanilla SAM. When applying SAM and its variants, SAM-S and SAM-P, on test images, we use the automatic prompt generation strategy and report the results with the highest IoU scores.
Metrics. Following existing methods [1, 2], we use four common metrics for evaluation, including mean absolute error  $(M)$ , adaptive F-measure  $(F_{\beta})$  [40], mean E-measure  $(E_{\phi})$  [41], and structure measure  $(S_{\alpha})$  [42]. Smaller  $M$ , or larger  $F_{\beta}$ ,  $E_{\phi}$ ,  $S_{\alpha}$  means better segmentation performance.
4.2 Comparative Evaluation
We perform evaluations on the following COS tasks, namely, Camouflaged Object Detection, Polyp Image Segmentation (PIS), and Transparent Object Detection (TOD). For all the tasks, we evaluate the performance with point annotations. We follow the previous weakly-supervised segmentation method [43] and randomly select two points (one from the foreground and one from the background) from the training masks as the point annotations. For COD, we additionally evaluate the performance using scribble annotations, using the scribble data provided in [43].
Camouflaged object detection. Four datasets are used for experiments, i.e., CHAMELEON [44], CAMO [45], COD10K [1], and NC4K [16]. Table 1 shows that our method reaches the best performance over all competing methods and baselines. Notably, while SAM has shown impressive
Table 3: Ablations for WS-SAM.
Baseline	MAF	PLW	ILS	M↓	\( {F}_{\beta } \uparrow \)	\( {E}_{\phi } \uparrow \)	\( {S}_{\alpha } \uparrow \)
✓	0.052	0.674	0.838	0.737
✓	✓	0.047	0.689	0.853	0.772
✓	✓	✓	0.044	0.697	0.866	0.793
✓	✓	✓	✓	0.038	0.719	0.878	0.803
Table 4: Ablations for MFG.
Metrics	w/o MFG	FG->SA	w/o multiscale	WGM->FC	w/ MFG
M ↓	0.044	0.038	0.040	0.039	0.038
Fβ ↑	0.684	0.708	0.702	0.710	0.719
Eφ ↑	0.857	0.868	0.858	0.871	0.878
Sα ↑	0.780	0.797	0.783	0.792	0.803
Table 5: Results of MFG with full supervision.
Metrics	Baseline	SegMaR [17]	PreyNet [8]	FGANet [15]	Ours
M↓	0.035	0.035	0.034	0.032	0.032
Fβ↑	0.688	0.699	0.715	0.708	0.706
Eφ↑	0.879	0.890	0.894	0.894	0.897
Sα↑	0.812	0.813	0.813	0.803	0.813
Table 6: Results on multi-object images.
Metrics	SCWS [37]	TEL [38]	SCOD [30]	Ours
M↓	0.094	0.101	0.084	0.070
Fβ↑	0.378	0.350	0.381	0.452
Eφ↑	0.740	0.726	0.718	0.772
Sα↑	0.625	0.617	0.643	0.687
performance for natural scene images, its performance on the challenging COD task is far from the existing methods particularly designed for this task. We do see performance gains after finetuning SAM with point (SAM-P) and scribble (SAM-S) supervision, but the results are still far below our method. This substantiates the superiority of our way of leveraging SAM to generate segmentation masks with sparse annotations and use the segmentation masks to train the segmenter. To verify our performance improvement over the existing WSCOS methods does not merely come from the usage of SAM, we integrate the most recent WSCOS method, SCOD [30], into our WS-SAM framework to also leverage the additional mask supervision. This results in the method, "SCOD+". We can see that our method still shows better performance, further verifying our advantages for this task.
Polyp image segmentation. Three widely-used Polyp datasets are selected, namely CVC-ColonDB [46], ETIS [47], and Kvasir [48]. Table 2 shows that our method significantly surpasses the second-best method, SCOD, with point supervision. SAM and SAM-P do not perform well on this task, further substantiating their weakness on this challenging segmentation task. While empowering SCOD with the proposed WS-SAM framework indeed improves the performance, the results are still lower than our method. This again verifies our benefit in handling challenging segmentation tasks.
Transparent object detection. Two datasets, GDD [6] and GSD [29], are used for evaluation. As shown in Table 2, our method surpasses all baseline methods and existing methods for this task as well. This shows strong robustness and generalizability of our proposed method.
Our method includes two main components, the SAM-based weakly-supervised mask generation framework, WS-SAM, and the multi-scale feature grouping (MFG) module. We conduct ablation studies about these two components on  $COD10K$  of the COD task with scribble supervision.
Ablation study for WS-SAM. We establish a baseline by using SAM to generate only one segmentation mask from one training image without augmentations for model training. On top of this baseline, we add the multi-augmentation fusion (MAF), pixel-level weighting (PLW), and image-level selection (ILS) techniques. Table 3 showing adding these components helps improve the performance, thus demonstrating their effectiveness.
Ablation study for MFG. We examine the effect of MFG by completely removing the MFG module, substituting the proposed feature grouping (FG) with slot attention (SA) [33], removing the multiscale strategy, and substituting the weighted gate mechanism (WGM) with fixed coefficient (FC). Table 4 shows that our designs reach better performance than the alternative ones.
MFG for the fully-supervised setting. The proposed MFG module is designed to evacuate discriminative cues from concealed images. We have demonstrated its effectiveness with sparse annotations for the weakly-supervised setting. However, it is expected to also work in the fully-supervised setting. To verify this, we conduct experiments for the standard fully-supervised COD task. Table 5 shows the results on the  $COD10K$  dataset. We can see that MFG indeed helps improve the performance of the baseline model, to the level comparative with state-of-the-art methods.
Performance on multi-object images. The proposed MFG module evacuates discriminative cues by performing feature grouping at different granularities, which facilitates discovering multiple objects
Table 7: Parameter analysis on  $K, {\tau }_{a},{\tau }_{r},T$  ,and  $\left( {{N}_{1},{N}_{2}}\right)$  .
Metrics	K	τa	τr	T	(N1,N2)
1	6	12	18	0.05	0.1	0.2	0.3	0.3	0.5	0.7	0.9	1	2	3	4	(2,4)	(2,8)	(4,8)	(2,4,8)
M↓	0.052	0.042	0.038	0.039	0.037	0.038	0.038	0.040	0.038	0.038	0.039	0.040	0.039	0.039	0.038	0.038	0.038	0.038	0.039	0.038
Fβ↑	0.674	0.697	0.719	0.718	0.706	0.719	0.716	0.704	0.723	0.719	0.715	0.700	0.706	0.715	0.719	0.720	0.719	0.714	0.711	0.721
Eφ↑	0.838	0.857	0.878	0.878	0.868	0.878	0.876	0.865	0.866	0.878	0.874	0.851	0.862	0.872	0.878	0.876	0.878	0.875	0.873	0.878
Sα↑	0.737	0.776	0.803	0.800	0.795	0.803	0.805	0.793	0.792	0.803	0.789	0.781	0.794	0.800	0.803	0.805	0.803	0.802	0.799	0.802
(c) Ours
Figure 4: Visualized results for COD tasks.
(d) SCOD
(e) TEL
(f) SCWS
(g) WSSA
(h) SAM-P
in images. To verify this, we evaluate the performance on the 186 images with more than one object from  $COD10K$ . Table 6 shows that MFG achieves the best performance, surpassing the second-best method (SCOD) by  $10.7\%$ . This gap is large than that with all test images, where the gap is  $5.8\%$ .
Randomness of point supervision. We follow the existing point-supervision segmentation methods and randomly select points from ground truth masks as the point annotation.
To study the variance of the random selection, we repeat the random selection 5 times and calculate the mean and standard deviation of the results. Fig. 5 shows that our method reaches the best results while having the smallest deviation.
Number of augmented views  $K$ . Table 7 shows that more augmented views help improve performance in the beginning, but the effect turns weaker when further increasing it.
Hyperparameters in image-level selection. Table 7 shows that it is best to set the absolute uncertainty threshold  $\tau_{a} = 0.1$  and the relative uncertainty threshold  $\tau_{r} = 0.5$ , and our method is not sensi
Figure 5: Five runs results with varied point annotations.
Hyperparameters in MFG. Table 7 shows that MFG achieves the best results when the iteration number  $T$  is set as 3, and the groups and scales setting is set as  $(N_{1}, N_{2}) = (2, 4)$ . Notice that when adopting  $(2, 4, 8)$ , RK2 is replaced with the third-order RK structure, resulting in extra computational burden with limited benefits. Hence, we select the RK2 structure with  $(N_{1}, N_{2}) = (2, 4)$ .
Result visualization. Fig. 4 shows the prediction maps with point supervision. We can see that our method produces more complete results than existing methods and localizes multiple objects more comprehensively. More visualization results can be found in the supplementary materials.
This paper proposes a new WSCOS method that includes two key components. The first one is the WS-SAM framework that generates segmentation masks with the recently proposed vision foundation model, SAM, and proposes multi-augmentation result fusion, pixel-level uncertainty weighting, and image-level uncertainty filtration to get reliable pseudo labels to train a segmentation model. The second is the MFG module that leverages the extracted clues for additional nuanced discrimination information. MFG improves feature coherence from a grouping aspect, allowing for alleviating incomplete segmentation and better multiple-object segmentation. Experiments on multiple WSCOS tasks confirm the superiority of our method over the baseline and existing methods.
Acknowledgements: This research is partly supported by the National Key R&D Program of China (Grants No. 2020AAA0108302 & 2020AAA0108303), and Shenzhen Science and Technology Project (Grant No. JCYJ20200109143041798) & Shenzhen Stable Supporting Program (WDZC20200820200655001) & Shenzhen Key Laboratory of next generation interactive media innovative technology (Grant No. ZDSYS20210623092001004). The authors express their appreciation to Dr. Fengyang Xiao for her insightful comments, improving the quality of this paper.
[41] Deng-Ping Fan, Ge-Peng Ji, Xuebin Qin, and Ming-Ming Cheng. Cognitive vision inspired object segmentation metric and loss function. Scientia Sinica Informationis, 6(6), 2021. 7
[42] Deng-Ping Fan, Ming-Ming Cheng, Yun Liu, Tao Li, and Ali Borji. Structure-measure: A new way to evaluate foreground maps. In ICCV, pages 4548-4557, 2017. 7
[43] Shuyong Gao, Wei Zhang, Yan Wang, Qianyu Guo, Chenglong Zhang, Yangji He, and Wenqiang Zhang. Weakly-supervised salient object detection using point supervision. In AAAI, volume 36, pages 670-678, 2022. 7
[44] Przemysław Skurowski, Hassan Abdulameer, J Błaszczyk, Tomasz Depta, Adam Kornacki, and P Koziel. Animal camouflage analysis: Chameleon database. Unpublished manuscript, 2(6):7, 2018. 7
[45] Trung-Nghia Le, Tam V Nguyen, Zhongliang Nie, Minh-Triet Tran, and Akihiro Sugimoto. Anabranch network for camouflaged object segmentation. Comput. Vis. Image Underst., 184:45-56, 2019. 7
[46] Nima Tajbakhsh, Suryakanth R Gurudu, and Jianming Liang. Automated polyp detection in colonoscopy videos using shape and context information. IEEE Trans. Med. Imaging, 35(2):630-644, 2015. 8
[47] Juan Silva, Aymeric Histace, Olivier Romain, Xavier Dray, and Bertrand Granado. Toward embedded detection of polyps in wce images for early diagnosis of colorectal cancer. Int. J. Comput. Assist. Radiol. Surg., 9:283-293, 2014. 8
[48] Debesh Jha, Pia H Smedsrud, Michael A Riegler, Pål Halvorsen, Thomas de Lange, Dag Johansen, and Håvard D Johansen. Kvasir-seg: A segmented polyp dataset. In MMM, pages 451-462. Springer, 2020. 8