SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Chunming He $^{1,\ast}$ , Kai Li $^{2,\ast}$ , Yachao Zhang $^{1}$ , Guoxia Xu $^{3}$ , Longxiang Tang $^{1}$ , Yulun Zhang $^{4}$ , Zhenhua Guo $^{5}$ , and Xiu Li $^{1,\dagger}$
<sup>1</sup>Shenzhen International Graduate School, Tsinghua University, <sup>2</sup>NEC Laboratories America, <sup>3</sup>Nanjing University of Posts and Telecommunications, <sup>4</sup>ETH Zürich, <sup>5</sup>Tianyi Traffic Technology
Weakly-Supervised Concealed Object Segmentation (WSCOS) aims to segment objects well blended with surrounding environments using sparsely-annotated data for model training. It remains a challenging task since (1) it is hard to distinguish concealed objects from the background due to the intrinsic similarity and (2) the sparsely-annotated training data only provide weak supervision for model learning. In this paper, we propose a new WSCOS method to address these two challenges. To tackle the intrinsic similarity challenge, we design a multi-scale feature grouping module that first groups features at different granularities and then aggregates these grouping results. By grouping similar features together, it encourages segmentation coherence, helping obtain complete segmentation results for both single and multiple-object images. For the weak supervision challenge, we utilize the recently-proposed vision foundation model, "Segment Anything Model (SAM)", and use the provided sparse annotations as prompts to generate segmentation masks, which are used to train the model. To alleviate the impact of low-quality segmentation masks, we further propose a series of strategies, including multi-augmentation result ensemble, entropy-based pixel-level weighting, and entropy-based image-level selection. These strategies help provide more reliable supervision to train the segmentation model. We verify the effectiveness of our method on various WSCOS tasks, and experiments demonstrate that our method achieves state-of-the-art performance on these tasks. The code will be available at https://github.com/ChunmingHe/WS-SAM.
Concealed object segmentation (COS) aims to segment objects visually blended with surrounding environments [1]. COS is a general term with different applications, e.g., camouflaged object detection [2, 3], polyp image segmentation [4, 5], transparent object detection [6, 7], etc. COS is a challenging task due to the intrinsic between foreground objects and background, which makes it extremely difficult to identify discriminative clues for accurate foreground-background separation. To address this challenge, existing methods have employed approaches that mimic human vision [8-10], introduce frequency information [11, 12], or adopt joint modeling in multiple tasks [13-18].
Weakly-Supervised COS (WSCOS) studies an even more challenging yet more practical problem, involving learning a COS model without relying on pixel-wise fully-annotated training data. WSCOS greatly reduces annotation costs by only requiring a few annotated points or scribbles in the foreground
*Equal Contribution, † Corresponding Author
or background. However, the sparsity of annotated training data diminishes the limited discrimination capacity of the segmenter during model learning, thus further restricting segmentation performance.
In this paper, we propose a new algorithm for the challenging WSCOS task. To tackle the intrinsic similarity of foreground and background, we introduce a Multi-scale Feature Grouping (MFG) module that first evacuates discriminative cues at different granularities and then aggregates these cues to handle various concealing scenarios. By performing feature grouping, MFG essentially promotes coherence among features and thus is able to alleviate incomplete segmentation by encouraging local correlation within individual objects, while also facilitating multiple-object segmentation by seeking global coherence across multiple objects.
To address the challenge of weak supervision, we propose to leverage the recently proposed vision foundation model, Segment Anything Model (SAM), to generate dense masks by using sparse annotations as prompts, and use the generated masks as pseudo labels to train a segmenter. However, due to the intrinsic similarity between foreground objects and the background, the pseudo labels generated by SAM may not always be reliable. We propose a series of strategies to address this problem. First, we propose to generate multiple augmentation views for each image and fuse the segmentation masks produced from all views. The fused mask can highlight reliable predictions resistant to image augmentations and tend to be more accurate and complete as an effect of ensemble. Second, we propose an entropy-based weighting mechanism that assigns higher weights to predictions of pixels of high certainty. Lastly, to deal with the extreme images that SAM fails to generate reasonably correct masks, we propose an entropy-based image-level selection technique to assess the quality of the generated mask and decide whether to use the masks as pseudo labels for model training. These strategies ensure that only high-quality pseudo labels are used for training the segmenter. For ease of description, we refer to our solution of using SAM to address this task as WS-SAM.
(1) We propose to leverage SAM for weakly-supervised segmentation by using the provided sparse annotations as prompts to generate dense segmentation masks and train a task segmentation model. To the best of our knowledge, this is the first attempt to leverage the vision foundation model to address the weakly-supervised segmentation task.
(2) We propose a series of strategies for dealing with potentially low segmentation mask quality, including the multi-augmentation result ensemble technique, entropy-based pixel-level weighting technique, and entropy-based image-level selection technique. These techniques help provide reliable guidance to train the model and lead to improved segmentation results.
(3) We introduce a Multi-scale Feature Grouping (MFG) technique to tackle the intrinsic similarity challenge in the WSCOS task. MFG evacuates discriminative cues by performing feature grouping at different granularities. It encourages segmentation coherence, facilitating to obtain complete segmentation results for both single and multiple object images.
(4) We evaluate our method on various WSCOS tasks, and the experiments demonstrate that our method achieves state-of-the-art performance.
Segment Anything Model. SAM [19] is a recently proposed vision foundation model trained on SA-1B of over 1 billion masks. Its primary objective is to segment any object in any given image without requiring any additional task-specific adaptation. Its outstanding quality in segmentation results and zero-shot generalization to new scenes make SAM a promising candidate for various computer vision tasks. However, recent studies have highlighted that SAM encounters difficulties when segmenting objects with poor visibility, such as camouflaged objects [20-22], medical polyps [23-27], and transparent glasses [28, 22]. These findings suggest that SAM still has limitations in COS tasks.
In this paper, we propose using SAM to generate dense segmentation masks from sparse annotations and introduce the first SAM-based weakly-supervised framework in COS, termed WS-SAM. To further increase the accuracy of the generated pseudo-labels, we propose a pseudo label refinement strategy. Such a strategy assigns higher weights to those reliable predictions that are resistant to various image augmentation. Therefore, WS-SAM can offer more precise and stable guidance for the learning process, ultimately boosting the segmentation performance of the segmenter.
Figure 1: Framework of WS-SAM with scribble supervision. Note that the corresponding masks of the augmented images are inversely transformed so as to be consistent with the original image.
Concealed Object Segmentation. With the rapid development of deep learning, learning-based segmenters have obtained great achievements in the fully-supervised COS tasks [4, 2, 29]. PraNet [4] proposed a parallel reverse attention network to segment polyps in colonoscopy images. Drawn inspiration from biology, SINet [2] designed a predator network to discover and locate camouflaged objects. To detect transparent objects, GSDNet [29] integrated an aggregation module and a reflection refinement module. However, there is limited research on the weakly-supervised COS task. SCOD [30] introduced the first weakly-supervised COD framework, but it is only supervised with sparse annotations, which greatly restricts its discrimination capacity and inevitably inhibits segmentation performance. To address this problem, we first propose using SAM to generate precise pseudo-labels. Besides, to tackle intrinsic similarity, we introduce the multi-scale feature grouping module to evacuate discriminative cues at different granularities and thus promote feature coherence.
Weakly-Supervised Concealed Object Segmentation (WSCOS) aims to learn a segmentation model from a sparsely-annotated training dataset  $S = \{\mathbf{X}_i, \mathbf{Y}_i\}_{i=1}^S$  and test the model on a test dataset  $\mathcal{T} = \{\mathbf{T}_i\}_{i=1}^T$ , where  $\mathbf{X}_i$  and  $\mathbf{T}_i$  denote the training and test images, respectively;  $\mathbf{Y}_i$  represents the sparse annotations, which could be a few points or scribbles annotated as foreground or background.
Learning the segmentation model could be a challenging task, as concealed objects usually blend well with their surrounding environment, making it hard to distinguish foreground from background. Besides, the sparse annotations  $\mathbf{Y}_i$  may not provide sufficient supervision to learn the model capable of making accurate dense predictions. To address these challenges, we first propose a strategy of leveraging the recently-proposed vision foundation model, Segment Anything Model (SAM), to generate high-quality dense masks from sparse annotations and use the dense masks as pseudo labels to train the segmentation model. In addition, we propose a Multi-scale Feature Grouping (MFG) module that groups features at different granularities, encouraging segmentation coherence and facilitating obtain complete segmentation results for various concealing scenarios.
3.1 Pseudo Labeling with SAM
SAM is a recently-released vision foundation model for generic object segmentation [19]. It is trained with more than one billion segmentation masks and has shown impressive capabilities of producing precise segmentation masks for a wide range of object categories (so-called "segment anything"). Unlike some enthusiasts who brag SAM has "killed" the segment task, we find that SAM is far from reaching that level, at least for the studied concealed object segmentation task. This is first because SAM requires "prompts" that provide clues about the objects of interest to produce segmentation results. While the prompts could be in many forms, e.g., points, masks, bounding boxes, etc., they are required to be provided by humans or other external sources (e.g., other algorithms) 2. This requirement of the additional prompt inputs makes SAM unable to be (directly) used for applications where only test images are provided. In addition, we find that while SAM exhibits
${}^{2}$  SAM includes an automatic prompt generation mechanism that first densely selects points across the whole image as prompts, then generates results based on the dense prompts, and last fuse the results. The problem is it segments everything possible in the image, with no distinguishing of objects of interest and others.
Figure 2: Masks of SAM with different augmented images. We inversely transform the masks to keep consistent with the original image. It is observed that fused masks contain more accurate and complete segmentation information.
impressive performance for general scene images, it still struggles for concealed object images, due to the intrinsic similarity between foreground objects and the background.
In this paper, we introduce SAM for the Weakly-Supervised Concealed Object Segmentation (WS-COS) task. As shown in Fig. 1, we use SAM to generate segmentation masks on training images by taking the sparse annotations as prompts, and take the segmentation masks as pseudo labels to train a COS model, which will be used for the test. It is expected that the SAM-generated pseudo labels are not reliable. We address this problem by proposing three techniques, namely, Multi-augmentation result fusion, and pixel-level weighting and image-level selection.
Multi-augmentation result fusion. Given a concealed image  $(\mathbf{X}_i, \mathbf{Y}_i) \in S$ , we generate  $K$  augmented images  $\{\mathbf{X}_i^k\}_{k=1}^K$  by applying stochastic augmentations randomly sampled from image flipping, rotation  $(0^\circ, 90^\circ, 180^\circ, 270^\circ)$ , and scaling  $(\times 0.5, \times 1.0, \times 2.0)$ . We send  $\{\mathbf{X}_i^k\}_{k=1}^K$  to SAM by using the sparse annotations  $\mathbf{Y}_i$  as prompts, and generate segmentation masks  $\{\mathbf{M}_i^k\}_{k=1}^K$ , where
$$
\mathbf {M} _ {i} ^ {k} = \operatorname {S A M} \left(\mathbf {X} _ {i} ^ {k}, \mathbf {Y} _ {i}\right). \tag {1}
$$
Note that  $\mathbf{M}_i^k$  has the same shape as input image  $\mathbf{X}_i^k$ , which may differ in shape from  $\mathbf{X}_i$ ; we perform inverse image transformation to ensure all masks have the same shape as the original image.
As different segmentation results can be obtained when feeding SAM with different prompts, we expect  $\{\mathbf{M}_i^k\}_{k=1}^K$  to vary since different augmented images are used for segmentation. Fig. 2 shows some examples. We can see that while these masks vary significantly in shape, they overlap in certain regions, which are reliably predicted by SAM regardless of image transformations and usually correspond to correctly predicted foreground regions. Besides, these masks complement each other, such that some foreground regions missed by one mask can be found in other masks. Based on these observations, we propose to fuse the segmentation masks for different augmented images, as
$$
\tilde {\mathbf {M}} _ {i} = \frac {1}{K} \sum_ {k = 1} ^ {K} \mathbf {M} _ {i} ^ {k}, \tag {2}
$$
where  $\tilde{\mathbf{M}}_i$  is the fused mask. We expect  $\tilde{\mathbf{M}}_i$  to be more reliable than the individual masks as it is an ensemble over various augmented images.
Pixel-level weighting. The prediction reliability of different pixels may vary. To highlight those more reliable ones, we propose to use entropy to weight the predictions. We calculate the entropy of each pixel and get an entropy map as
$$
\tilde {\mathbf {E}} _ {i} = - \tilde {\mathbf {M}} _ {i} \log \tilde {\mathbf {M}} _ {i} - (1 - \tilde {\mathbf {M}} _ {i}) \log (1 - \tilde {\mathbf {M}} _ {i}). \tag {3}
$$
As the entropy map is calculated from the fused mask, it measures the prediction uncertainty of each pixel across all augmentation as a pixel will have low entropy only if it is confidently and meanwhile consistently predicted from all augmented images. Therefore, we can use this entropy map to weigh the fused mask  $\mathbf{M}_i^k$  and assign higher weights to those reliable pixels.
Image-level selection. We have observed that for some highly challenging concealed images, SAM fails to produce even reasonably correct results with the sparse annotations as the prompts, with
Figure 3: Architecture of the proposed model.  $\Phi_P$  denotes feature grouping with  $P$  prototypes. We simplify the broadcast process in  $\Phi_P$  for space limitation.
whatever types of augmented images. This fundamentally invalidates the above pixel-wise weighting strategy. To deal with this case, we further propose an image-level selection mechanism to selectively choose images for training, further striving to provide reliable supervision to train the segmenter.
Similar to using entropy to define the pixel-level prediction uncertainty, we propose two entropy-based image-level uncertainty measurements, namely, absolute uncertainty  $U_{a}$  and relative uncertainty  $U_{r}$ . Absolute uncertainty  $U_{a}$  refers to the proportion of high-uncertainty pixels among all pixels, while relative uncertainty  $U_{r}$  indicates the proportion of high-uncertainty pixels and foreground pixels with low uncertainty, which is specifically designed to accommodate small-object scenarios. We regard a pixel as a high-uncertainty pixel when its entropy is above 0.9. We define the following indicator function to decide whether to keep an image for training or not:
$$
\tilde {\mathbf {M}} _ {i} = \mathbb {1} \left[ U _ {a} <   \tau_ {a} \right] \times \mathbb {1} \left[ U _ {r} <   \tau_ {r} \right], \tag {4}
$$
where  $\tau_{a}$  and  $\tau_{r}$  are the thresholds that are set as 0.1 and 0.5 in this paper, respectively.
Applying the entropy weights  $\tilde{\mathbf{E}}_i$  on the image selection indicator  $\hat{\mathbf{M}}_i$ , we reach our final mask that will be used for training the segmenter as,
$$
\hat {\mathbf {Y}} _ {i} = \left(1 - \tilde {\mathbf {E}} _ {i}\right) \times \hat {\mathbf {M}} _ {i}. \tag {5}
$$
Our technique leverages SAM to generate segmentation masks and further incorporates multi-augmentation result fusion, pixel-level uncertainty weighting, and image-level uncertainty filtering, thus being able to generate reliable pseudo labels to train the segmenter.
3.2 Multi-scale Feature Grouping
The intrinsic similarity in concealed objects may cause incomplete segmentation and partial object localization in multi-object segmentation. Such problems could be further aggravated in weakly supervised scenarios due to the limited discriminative capacity of the segmenter. To address this issue, we propose a Multi-scale Feature Grouping (MFG) module that evacuates discriminative cues at various granularities. MFG achieves this by exploring the coherence of foreground/background regions and performing feature grouping at different levels. By encouraging feature coherence, MFG can alleviate incomplete segmentation by enhancing local correlation within individual objects and further facilitate multiple-object segmentation by seeking global coherence across multiple objects. The architecture of the proposed MFG module is illustrated in Fig. 3.
Feature grouping. Suppose  $\mathbf{F} \in \mathbb{R}^{H \times W \times C}$  is the feature representation of an input image. We perform feature grouping by mapping  $\mathbf{F}$  to  $N$  learnable cluster prototypes  $\mathbf{P} \in \mathbb{R}^{N \times C}$ . These cluster prototypes  $\mathbf{P}$  are randomly initialized. We first append the learnable spatial positional embedding  $\mathbf{P}_e$  to the input feature  $\mathbf{F}$  and get  $\mathbf{F}_p$ . Then, we linearly transform the prototypes  $\mathbf{P}$  and the positioned feature  $\mathbf{F}_p$  into  $\mathbf{Q} \in \mathbb{R}^{N \times C}$ ,  $\mathbf{K} \in \mathbb{R}^{HW \times C}$ , and  $\mathbf{V} \in \mathbb{R}^{HW \times C}$ :
$$
\mathbf {Q} = \mathbf {W} _ {q} \mathbf {P}, \quad \mathbf {K} = \mathbf {W} _ {k} \mathbf {F} _ {p}, \quad \mathbf {V} = \mathbf {W} _ {v} \mathbf {F} _ {p}, \tag {6}
$$
where  $\mathbf{W}_q, \mathbf{W}_k, \mathbf{W}_v \in \mathbb{R}^{C \times C}$  are the learnable weights. To ensure the exclusive assignment of features to the cluster prototypes, we normalize the coefficients over all prototypes,
$$
\bar {\mathbf {A}} _ {i, j} = \frac {e ^ {\mathbf {A} _ {i , j}}}{\sum_ {l} e ^ {\mathbf {A} _ {i , l}}}, \quad \text {w h e r e} \quad \mathbf {A} = \frac {1}{\sqrt {C}} \mathbf {K} ^ {\top} \mathbf {Q}. \tag {7}
$$
We then calculate the integral value  $\mathbf{U}$  of the input values with respect to the prototypes as
$$
\mathbf {U} = \mathbf {D} ^ {\top} \mathbf {V}, \quad \text {w h e r e} \quad \mathbf {D} _ {i, j} = \frac {\mathbf {A} _ {i , j}}{\sum_ {l} \mathbf {A} _ {i , l}}, \tag {8}
$$
and update the prototypes  $\mathbf{P}$  by feeding  $\mathbf{P}$  and  $\mathbf{U}$  into a Gated Recurrent Units  $GRU(\cdot)$ :
$$
\mathbf {P} = G R U (i n p u t s = \mathbf {U}, s t a t e s = \mathbf {P}). \tag {9}
$$
By repeating Eqs. (6) - (9) for  $T$  iterations, the cluster prototypes are iteratively updated and gradually strengthen the association between similar features, where  $T = 3$  in this paper.
We broadcast each prototype onto a 2D grid augmented with the learnable spatial position embedding  $\mathbf{P}_e$  to obtain  $\{\mathbf{F}_i^{\prime}\}_{i = 1}^{N}\in \mathbb{R}^{H\times W\times C}$ , and use  $1\times 1$  convolution to downsample each prototype, obtaining  $\{\mathbf{F}_i^{\prime \prime}\}_{i = 1}^{N}\in \mathbb{R}^{H\times W\times C / N}$ . We concatenate those prototypes and obtain  $\mathbf{F}_c\in \mathbb{R}^{H\times W\times C}$ .
For ease of future use, we denote the feature grouping process with  $N$  prototypes as  $\mathbf{F}_c = \Phi_N(\mathbf{F})$ .
Multi-scale feature aggregation. The number of prototypes  $N$  in the above feature grouping technique controls the grouping granularity: a smaller value of  $N$  facilitates the extraction of global information, while a larger value of  $N$  can provide more valuable detailed information. To strike a balance, we propose to aggregate the multi-scale grouping features with different numbers of prototypes. Taking inspiration from the second-order Runge-Kutta (RK2) structure known for its superior numerical solutions compared to the traditional residual structure [31, 32], we employ RK2 to aggregate those features. Additionally, as shown in Fig. 3, we adopt a weighted gate mechanism  $\alpha_{g}$  to adaptively estimate the trade-off parameter rather than using a fixed coefficient. Given the feature  $\mathbf{F}$ , the adaptively aggregated feature  $\hat{\mathbf{F}}$  is formulated as follows:
$$
\hat {\mathbf {F}} = \mathbf {F} + \alpha_ {g} \Phi_ {N _ {1}} (\mathbf {F}) + (1 - \alpha_ {g}) \Phi_ {N _ {2}} (\mathbf {F} + \Phi_ {N _ {1}} (\mathbf {F})), \tag {10}
$$
where  $\alpha_{g} = S(\sigma \operatorname{cat}(\Phi_{N_{1}}(\mathbf{F}), \Phi_{N_{2}}(\mathbf{F} + \Phi_{N_{1}}(\mathbf{F}))) + \mu)$ .  $S$  is Sigmoid.  $\sigma$  and  $\mu$  are the learnable parameters in  $\alpha_{g}$ .  $N_{1}$  and  $N_{2}$  are the numbers of groups, which are empirically set as 4 and 2.
Our multi-scale feature grouping technique is inspired by the slot attention technique [33], but we differ from slot attention in the following aspects. Slot attention targets at instance-level grouping in a self-supervised manner, our MFG is proposed to adaptively excavate the feature-level coherence for complete segmentation and accurate multi-object localization. To relax the segmenter and ensure generalization, we remove the auxiliary decoder used in slot attention for image reconstruction, along with the reconstruction constraint. Additionally, we employ an RK2 structure to aggregate the multiscale grouping feature with different numbers of prototypes, which further facilitates the excavation of feature coherence and therefore helps improve segmentation performance.
3.3 Weakly-Supervised Concealed Object Segmentation
To use the proposed MFG technique for concealed object segmentation, we integrate MFG with the encoder and decoder architecture utilized in an existing camouflaged object detection model [12] to construct a novel segmenter. The model comprises a ResNet50-backed encoder  $E$  that maps an input image  $\mathbf{X}_i$  to a feature space as  $\mathbf{F}_i = E(\mathbf{X}_i)$ . Using the obtained  $\mathbf{F}_i$ , we apply MFG to perform multi-scale feature grouping, resulting in  $\hat{\mathbf{F}}_i = MFG(\mathbf{F}_i)$ . Subsequently, a decoder  $D$  maps  $\hat{\mathbf{F}}_i$  back to the image space, generating the predicted mask  $\mathbf{Y}_i' = D(\hat{\mathbf{F}}_i)$ . Fig. 3 provides a conceptual illustration of this model, and more architecture details can be found in the supplementary materials.
Following [30], we train the whole model jointly with the sparse annotations  $\mathbf{Y}_i$ , and the generated segmentation mask  $\hat{\mathbf{Y}}_i$  and  $\tilde{\mathbf{M}}_i$  by the SAM model, as