Weakly-Supervised Concealed Object Segmentation with SAM-based Pseudo Labeling and Multi-scale Feature Grouping
Authors: Chunming He
Kai Li
Yachao Zhang
Guoxia Xu
Longxiang Tang
Yulun Zhang
Zhenhua Guo
Xiu Li
Venue: NeurIPS 2023 poster (2023)
Keywords: Concealed Object Segmentation
Weakly-Supervised Learning
Segment Anything Model

Abstract: Weakly-Supervised Concealed Object Segmentation (WSCOS) aims to  segment objects well blended with surrounding environments using sparsely-annotated data for model training. It remains a challenging task since (1) it is hard to distinguish concealed objects from the background due to the intrinsic similarity and (2) the sparsely-annotated training data only provide weak supervision for model learning. In this paper, we propose a new WSCOS method to address these two challenges. To tackle the intrinsic similarity challenge, we design a multi-scale feature grouping module that first groups features at different granularities and then aggregates these grouping results. By grouping similar features together, it encourages segmentation coherence, helping obtain complete segmentation results for both single and multiple-object images. For the weak supervision challenge, we utilize the recently-proposed vision foundation model, ``Segment Anything Model (SAM)'', and use the provided sparse annotations as prompts to generate segmentation masks, which are used to train the model. To alleviate the impact of low-quality segmentation masks, we further propose a series of strategies, including multi-augmentation result ensemble, entropy-based pixel-level weighting, and entropy-based image-level selection. These strategies help provide more reliable supervision to train the segmentation model. We verify the effectiveness of our method on various WSCOS tasks, and experiments demonstrate that our method achieves state-of-the-art performance on these tasks.

Weakly-Supervised Concealed Object Segmentation with SAM-based Pseudo Labeling and Multi-scale Feature Grouping
Chunming He Kai Li:*, Yachao Zhang', Guoxia Xu² Longxiang Tang', Yulun Zhang* Zhenhua Guo and Xiu Li Shenzhen International Graduate School, Tsinghua University, 2EBC Laboratories America, 3 Nanjing University of Posts and Telecommunications 'ETH Zürich, Tianyi Traffic Technology
Abstract
Weakly-Superrvve Concealed Object Segmentation (WSCOS) aims to segment objects well blended with surrounding environments using sparsely-annotatee data for model training. It remains challenging task since (1) is hard distinguish concealed objects from the background due to the intrinsic similarity and (2) the sparsely-annotated training data only provide weak supervision for model learning. In this paper, we propose new WSCOS method to address these two challenges. To tackle the intrinsic similarity challenge, we design multi-scale feature grouping module that first groups features at different granularities and then aggregates these grouping results. By grouping similar features together, it encourages segmentation coherence, helping obtain complete segmentation results for both single and multiple-objer images. For the weak supervision challenge, we utilize the pecently-popposee vision foundation model, "Segment Anything Model (SAM)", and use the provided sparse annotations as prompts to generate segmentation masks, which are used to train the model. To alleviate the impact of low-quality segmentation masks, we further propose a series of strategies, including multi- culti-augmeenationn result ensemble, entropy-based pixel-level weighting, and entropy-baat based image-level selection. These strategies help provide more reliable supervision to train the segmentation model. We verify the effectiveness of our method on various WSCOS tasks, and experiments demonstrate that our method achieves state-of- th- performance on these tasks. The code will be available at https: //gi thub. com /aa//huumiii nggee -
1 Introduction
Concealed object segmentation (COS) aims to segment objects visually blended with surrounding environments [1]. COS is a general term with different applications, e.g., camouflaged object
detection [2, 3]. polyp image segmentation [4, 5], transparent object detection [6, 7J, etc. COS is challenging task due the intrinsic between foreground objects and background, which makes extremely difficult to identify discriminative clues for accurate foreground-backgroune separation. To address this challenge, existing methods have employed approaches that mimic human vision [8-10], introduce frequency information [11, 12], or adopt joint modeling in multiple tasks [13-18].
Weakly-Superrvise COS (WSCOS) studies even involving learning a COS model without relying on pixel-wise tnnottanottteed training data. WSCOS greatly reduces annotation costs by only requiring a few annotated points or scribbles in the foreground
challenging yet more practical problem,
Equal Contribution, f Corresponding Author
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

or background. However, the sparsity of annotated training data diminishes the limited discrimination capacity of the segmenter during model learning, thus further restricting segmentation performance.
In this paper, we propose a new algorithm for the challenging WSCOS task. To tackle the intrinsic similarity of foreground and background, introduce Multi-scale Feature Grouping (MPG) module that first evacuates discriminative cues at different granularities and then aggregates these cues to handle various concealing scenarios. By performing feature grouping, MFG essentially promotes coherence among features and thus is able to alleviate incomplete segmentation by encouraging local correlation within individual objects, while also facilitating multiple-object segmentation by seeking global coherence across multiple objects.
To address the challenge of weak supervision, we propose to leverage the recently proposed vision foundation model, Segment Anything Model (SAM), to generate dense masks by using sparse annota- tions as prompts, and use the generated masks as pseudo labels train segmenter. However, due to the intrinsic similarity between foreground objects and the background, the pseudo labels generated by SAM may not always be reliable. We propose series of strategies to address this problem. First, we propose to generate multiple augmentation views for each image and fuse the segmentation masks produced from all views. The fused mask can highlight reliable predictions resistant image augmentations and tend to be more accurate and complete as an effect of ensemble. Second, we propose an entropy-based weighting mechanism that assigns higher weights to predictions of pixels of high certainty. Lastly, to deal with the extreme images that SAM fails to generate reasonably correct masks, we propose an entropy based image-level selection technique assess the quality of the generated mask and decide whether to use the masks as pseudo labels for model training. These strategies ensure that only high-quality pseudo labels are used for training the segmenter. For ease of description, we refer to our solution of using SAM to address this task as WS-SAM.
Our contributions are summarized follows:
(I) We propose to leverage SAM for weakly-supervise segmentation by using the provided sparse annotations prompts generate dense segmentation masks and train task segmentation model. To the best of our knowledge, this is the first attempt to leverage the vision foundation model to address the supervis segmentation task.
(2) We propose series of strategies for dealing with potentially low segmentation mask quality, including the multi-augmentatioon result ensemble technique, entropy-based pixel-level weighting technique, and entropy-based image-level selection technique. These techniques help provide reliable guidance to train the model and lead to improved segmentation results.
(3) We introduce a Multi-scale Feature Grouping (MFG) technique to tackle the intrinsic similarity challenge in the WSCOS task. MFG evacuates discriminative cues by performing feature grouping at different granularities. It encourages segmentation coherence, facilitating to obtain complete segmentation results for both single and multiple object images.
(4) We evaluate our method on various WSCOS tasks, and the experiments demonstrate that method achieves state-of-the-art performance.
2 Related Works
Segment Anything Model. SAM [19] is a recently proposed vision foundation model trained on SA- IB of over billion masks. Its primary objective is to segment any object in any given image without requiring any additional task-specific adaptation. Its outstanding quality in segmentation results and zero-shot generalization to new scenes make SAM a promising candidate for various computer vision tasks. However, recent studies have highlighted that SAM encounters difficulties when segmenting objects with poor visibility, such as camouflaged objects [20-22], medical polyps [23-27], and transparent glasses [28, 22]. These findings suggest that SAM still has limitations in COS tasks.
In this paper, we propose using SAM to generate dense segmentation masks from sparse annotations and introduce the first SAM-based weakly weakly-supervised framework in COS, termed WS-SAM. To further increase the accuracy of the generated pseudo-labels, we propose a pseudo label refinement strategy. Such a strategy assigns higher weights to those reliable predictions that are resistant to various image augmentation. Therefore, WS-SAM can offer more precise and stable guidance for the learning process, ultímately boosting the segmentation performance of the segmenter.

---

promps
Retation 180
Sceling x0.5
SAM
Augmentations
Fused mase
Supentsion
Image
tom
weighing
Yos
No
Weighted
Disgard
Figure I: Framework of WS-SAM with scribble supervision. Note that the corresponding masks of the augmented images are inversely transformed so as to be consistent with the original image.
Concealed Object Segmentation. With the rapid development of deep learning, learning-baa based segmenters have obtained great achievements in the sully-supervised COS tasks [4, 2, 29]. PraNet [4] proposed a parallel reverse attention network to segment polyps in colonoscopy images. Drawn inspiration from biology, SINet [2] designed predator network discover and locate camou- flaged objects. To detect transparent objects, GSDNet [29] integrated an aggregation module and reflection refinement module. However, there is limited research the weakly-supervised COS task. SCOD [30] introduced the first weakly-supervised COD framework, but it is only supervised with sparse annotations, which greatly restricts its discrimination capacity and inevitably inhibits segmentation performance. To address this problem, we first propose using SAM to generate precise pseudo-labels. Besides, to tackle intrinsic similarity, we introduce the multi-scale feature grouping module to evacuate discriminative cues at different granularities and thus promote feature coherence.
3 Methodology
Weakly-Supervised Concealed Object Segmentation (WSCOS) aims to learn segmentation model from sparsely-annotated training dataset S (X;, = and test the model on a test dataset T= (Tr) where X, and T; denote the training and test images, respectively; Y, represents the sparse annotations, which could be few points scribbles annotated as foreground or background.
Learning the segmentation model could be challenging task, concealed objects usually blend well with their surrounding environment, making it hard to distinguish foreground from background. Besides, the sparse annotations Y; may not provide sufficient supervision to learn the model capable of making accurate dense predictions. To address these challenges, we first propose a strategy of leveraging the recently-proposed vision foundation model, Segment Anything Model (SAM), to generate high-qualio dense masks from sparse annotations and use the dense masks as pseudo labels to train the segmentation model. In addition. we propose Multi-scale Feature Grouping (MFG) module that groups features at different granularities, encouraging segmentation coherence and facilitating obtain complete segmentation results for various concealing scenarios.
3.1 Pseudo Labeling with SAM
SAM is recently-released vision foundation model for generic object segmentation [19]. It is trained with more than one billion segmentation masks and has shown impressive capabilities of producing precise segmentation masks for wide range of object categories (so-called "segment anything"). Unlike some enthusiasts who brag SAM has "killed" the segment task, we find that SAM far from reaching that level, at least for the studied concealed object segmentation task. This is first because SAM requires "prompts" that provide clues about the objects interest to produce segmentation results. While the prompts could be in many forms, e.g., points, masks, bounding boxes, etc., they are required to be provided by humans or other external sources (e.g., other algorithms) 2. This requirement of the additional prompt inputs makes SAM unable be (directly) used for applications where only test images are provided. In addition, we find that while SAM exhibits
"SAM includes an automatic prompt generation mechanism that first densely selects points across the whole image as prompts, then generates results based on the dense prompts, and last fuse the results. The problem is it segments everything possible in the image, with distinguishing of objects of interest and others.

---

Flip
Mask
Flip
Origin
Rotation 180"
Mask
Origin
Rotation 180"
Mask
Fused Scaling
Fused Scaling
Mask
Figure 2: Masks SAM with different augmented images. We inversely transform the masks to keep consistent with the original image. It is observed that fused masks contain more accurate and complete segmentation information.
impressive performance for general scene images, it still struggles for concealed object images, due to the intrinsic similarity between foreground objects and the background.
In this paper, we introduce SAM for the COS) task. As shown in Fig. we use SAM to generate segmentation masks on training images by taking the sparse annotations prompts, and take the segmentation masks as pscudo labels to train COS model, which will be used for the test. It is expected that the SAM-generated pseudo labels are not reliable. We address this problem by proposing three techniques, namely, Multi-augmentation result fusion, and pixel-level weighting and image-level selection.
Concealed Object Segmentation (WS-
Multi-auumeenattin result fusion. Given concealed image (X,, Y.) E S, we generate K augmented images { k==1 by applying stochastic augmentations randomly sampled from image flipping, rotation (0%, 90% 180%, 270"), and scaling (x0.5, x1.0, x2.0). We send {X, k ]k=1 to SAM
by using the sparse annotations Y; as prompts, and generate segmentation masks {M, Jk=1 where
M = SAM (X Y.).
(1)
Note that M has the same shape input image X which may differ shape from X,; we perform inverse image transformation to ensure all masks have the same shape the original image.
As different segmentation results can be obtained when feeding SAM with different prompts, expect [Mr ]==1 to vary since different augmented images are used for segmentation. Fig. 2 shows some examples. We
that while these masks vary significantly certain regions, which are reliably predicted by SAM regardless of image transformations and usually correspond to correctly predicted foreground regions. Besides, these masks complement each other, such that some foreground regions missed by one mask can be found in other masks. Based on these observations, we propose to fuse the segmentation masks for different augmented images, as
shape, they overlap in
1 K
M.
= Σ M
(2)
K
k=
where M, is the fused mask. We expect M, to be more reliable than the individual masks as it is an ensemble over various augmented images.
Pixel-level weighting. The prediction reliability of different pixels may vary. To highlight those more reliable ones, we propose use entropy to weight the predictions. We calculate the entropy of each pixel and get an entropy map as
È, -M, log M,
M,) log(1 M,).
As the entropy map is calculated from the fused mask, it measures the prediction uncertainty each pixel across all augmentation consistently predicted from all augmented images. Therefore, we can use this entropy map to weigh the fused mask M and assign higher weights to those reliable pixels.
pixel will have low entropy only if it is confidently and meanwhile
Image-level selection. We have observed that for some highly challenging concealed images, SAM fails to produce even reasonably correct results with the sparse annotations the prompts, with

---

ittrrationa
N
MFG
P.
Summation
Multioication
Weighted mechanism
Embedding
Broadcast
Architecture of the proposed model. Dp denotes feature grouping with P prototypes. We simplify the broadcast process Φ p for space limitation.
Figure 3:
whatever types of augmented images. This fundamentally invalidates the above pixel-wise weighting strategy. To deal with this case, we further propose an image-level selection mechanism to selectively choose images for training, further striving to provide reliable supervision to train the segmenter.
Similar to using entropy to define the pixel-level prediction uncertainty, we propose two entropy-based image-level uncertainty measurements, namely, absolute uncertainty Ua and relative uncertainty U,. Absolute uncertainty Ua refers to the proportion of high-uncertainty pixels among all pixels, while relative uncertainty U, indicates the proportion of high- uncertainty pixels and foreground pixels with low uncertainty, which is specifically designed to accommodate small- scenarios. We regard pixel as a high-uncertaing pixel when its entropy above 0.9. We define the following indicator function to decide whether to keep an image for training or not:
M, = [U, < Tal x 1U, < Tr].
where Ta and are the thresholds that are set as 0.1 and 0.5 in this paper, respectively.
Applying the entropy weights E, on the image selection indicator M,, we reach our final mask that will be used for training the segmenter as,
Y, = (1 E.) x M.
(5)
Our technique leverages SAM to generate segmentation masks and further incorporates multi- augmentation result fusion, pixel-level uncertainty weighting, and image-level uncertainty filtering, thus being able to generate reliable pseudo labels to train the segmenter.
3.2 Multi-scale Feature Grouping
The intrinsic similarity in concealed objects may cause incomplete segmentation and partial object localization in multi-object segmentation. Such problems could be further aggravated in weakly supervised scenarios due the limited discriminative capacity of the segmenter. To address this issue, we propose Multi-scale Feature Grouping (MFG) module that evacuates discriminative cues various granularities. MFG achieves this by exploring the coherence of foreground/baccgroondd regions and performing feature grouping at different levels. By encouraging feature coherence, MFG can alleviate incomplete segmentation by enhancing local correlation within individual objects and further facilitate multiple-object segmentation by seeking global coherence across multiple objects. The architecture the proposed MFG module is illustrated in Fig.
Feature grouping. Suppose E H XC is the feature representation of an input image. We perform feature grouping by mapping F to N learnable cluster prototypes P E N. These cluster prototypes P are randomly initialized. We first append the learnable spatial positional embedding P. the input feature F and get p. Then, we linearly transform the prototypes and the positioned feature F, into Q NCC K R HC and V E HCC.
Q W.P. K= = W.FF.
(6)
where W 9, WA. W, CCC are the learnable weights. To ensure the exclusive assignment of features to the cluster prototypes, we normalize the coefficients over prototypes,
A
A,
where A =
VC KTQ
EAA..
We then calculate the integral value U of the input values with respect to the prototypes as
Aij
U= DVD where Dig = Σ, Ari'
(8)
5

---

and update the prototypes P by feeding P and U into Gated Recurrent Units GRU(-))
P GRU (inputs U, states = P).
(9)
By repeating Eqs. (6)-(9) for T iterations, the cluster prototypes are iteratively updated and gradually strengthen the association between similar features, where = 3 in this paper.
We broadcast each prototype onto 2D grid augmented with the learnable spatial position embedding Pe to obtain F E "x"x and use x convolution to downsample each prototype, obtaining [ E R HXWXC/N We concatenate those prototypes and obtain F E IR Hx= xC
For ease of future use, we denote the feature grouping process with N prototypes as F. = 2 (F).
Multi-scale feature aggregation. The number of prototypes N in the above feature grouping technique controls the grouping granularity: a smaller value N facilitates the extraction of global information, while larger value of N provide more valuable detailed information. To strike a balance, we propose to aggregate the multi-scale grouping features with different numbers of prototypes. Taking inspiration from the second-order Runge- (RK2) structure known for its superior numerical solutions compared to the traditional residual structure [31, 32], we employ RK2 aggregate those features. Additionally, as shown in Fig. 3, we adopt a weighted gate mechanism ag to adaptively estimate the trade-off parameter rather than using fixed coefficient. Given the feature F, the adaptively aggregated feature F is formulated as follows:
(10) where 0g = S(o (at( (F), & N₂ (F + NN2 (F))) + p.). S is Sigmoid. and / are the learnable parameters in ag. N1 and N₂ are the numbers of groups, which are empirically set as 4 and 2.
F = F + DN (F) + (1 - Pg (F + ΦN, (F)),
Our multi-scale feature grouping technique is inspired by the slot attention techmique [33], but we differ from slot attention in the following aspects. Slot attention targets at instance-level grouping in self-supervised manner, our MFG is proposed to adaptively excavate the feature-level coherence for complete segmentation and accurate multi-object localization. To relax the segmenter and ensure generalization, we remove the auxiliary decoder used in slot attention for image reconstruction, along with the reconstruction constraint. Additionally, we employ an RK2 structure aggregate the multiscale grouping feature with different numbers of prototypes, which further facilitates the excavation feature coherence and therefore helps improve segmentation performance.
3.3 Weakly-Supervised Concealed Object Segmentation
To use the proposed MFG technique for concealed object segmentation, we integrate MFG with the encoder and decoder architecture utilized in existing camouflaged object detection model [12] to construct a novel segmenter. The model comprises a MaANNet50-backer encoder E that maps an input image X, to feature space as F, E. Using the obtained F,, we apply MFG to perform multi-scale feature grouping, resulting in F, M (F..... Subsequently, decoder D maps F, back to the image space, generating the predicted mask Y D(F.). Fig. provides a conceptual illustration of this model, and more architecture details can be found in the supplementary materials.
Following [30], we train the whole model jointly with the sparse annotations Y,, and the generated segmentation mask Y, and M, by the SAM model, as
1 No (X,
Σ
Lpee (Y,, Y.) + YY, Y. (Y', (Y,, M.).
(11)
Where the first term is the partial cross-entropy loss Liee used to ensure consistency between the prediction maps and the sparse annotations Y, [34]. The second and third terms are the cross cross-entropy loss Lee and the intersection-over-unio loss LIw,, both calculated using the pseudo label Y, [35].
4 Experiments
4.1 Experimental Setup
Implementation details. The image encoder uses ResNet50 the backbone and is pre-trained on ImageNet [39]. The batch size is 36 and the learning rate is initialized as 0.0001, decreased by 0.1 every 80 epochs. For scribble supervision, we propose nine-box strategy, namely constructing

---

Table I: Results on COD with point supervision and scribble supervision. SCOD+ indicates integrating SCOD with our WS-SAM framework. The best two results are in red and blue fonts.
CHAMELLON Fa E. S.
CAMO
CODIOK
NC4R
Methods
Pub.
1
E.
5.
4
Fg
E
S.
1
F6 Eg S.
Scribble Supervision
SAM [19]
0.207
0.395
0.647
0.635
0,160
0,597
0.639 0.643
|00093
0.673
0.737
0.730
0.118
0.675
0.723
0.717
SAM-S [19]
0.076
0.729
0.650
0.682
0.731
0.695
0.772
0.747
0.763
WSSA [36]
CVPR20
0.067
0.692
0.860
0.782
0,118
0.615
0.786
0.696
0.071
0,536
0,770
0,684
0.091
0.657
0.779
0.761
[37]
AAAI21
0.053
0.758 0.881
0.792
0.658
0.713
0.602
0.710
0.723 0.814
0.784
TEL [38]
CVPR22
0.073
0.708 0.827
0.785
0.104
0,681
0.797
0.717
0.057
0.633 0.826
0.724
0.075
0.754 0.832
0.782
SCOD [30]
AAA123
0.791
0.818
0.709
0.735
0.049
0.637
0.733
0.751
0.779
SCOD+
0.046
0.797
0.900
0.820
0.090
0.716 0.818
0.741
0.047
0.650
0,845
0.742
0.060
0.766
0.862
0.785
Ours
0.046
0.777
0.897
0.824
0.092
0.742 0.818 0.759
0.038
0.719
0.878
0.803
0.052
0.802
0.886
0.829
Point Supervision
SAM [19]
0.207
0.595
0.647
0.635
0.160
0.597 0.639 0.643
0.093
0.673
0.737
0.730
0.118
0.675
0.723
0.717
SAM-P [[1]
0.101
0.696
0.697
0.649
0.677
0.694
0.796
0.765
0.728
0.786
0.776
WSSA [6]]
CVPR20
0.105
0.660 0.712
0.711
0.148
0.607 0.652 0.649
0.087
0.509 0.733
0.642
0.104
0.688
0.756
0.743
SCWS [37]
AAAI2I
0.097 0.684 0.739
0.714
0,142
0.624 0.672 0.687
0.082
0.593 0.777
0.738
0.098
0.695 0.767
0.754
TEL [38]
CVPR22
0.094 0.712 0.751 0.746
0,135
0.662 0,674 0.645
0.063 0.623 0,803
0.727
0.085
0.725 0.795
0.766
SCOD [30]
AAA123
0.092 0.688 0.746
0.725
0.137
0.629 0.688 0.063
0.060
0.607 0.802
0.711
0.080
0.744
0.796 0.758
SCOD+
0.089 0.704 0.757
0.731
0.129
0.642 0.693 0.666 0.058 0.618 0.812
0.719
0.075
0.767
0,825
0.771
Ours
0.056 0.767 0.868 0.805
0.102
0,703 0.757 0.718 0.039 0.698 0.856 0.790 0.057 0.801 0.859 0.813
Table 2: Results for PIS and TOD with point supervision.
Polyp Image Seementation (PIS)
Transpurant Object Detection (TOD)
Methods
CVC ColonIa
E.78
Kyussr F
GDD
GSD
Fa
6. 1
5, 1
S,
Eg
S,
Fg
Eg.
S.
SAM [19]
00.499
0.343
0.419
0.427
0.429
0.439
0.512
0.503
0.320
0,545
0.564
0.582
0.245
0.512
0.530
0.551
0.266
0.473
0.501
0,514
SAM-P [1]
0.194
0.587 0.664 0.671
0.144
0.625
0.719
0.715 0,108
0.793
0.811
0.802
0.164
0.668
0.715
0.625
0.177
0.687
0.730 0.668
WSSA [36]
0.127
0.645 0.732
0.713
0.123
0.647 0.733
0.762
0.082
0.822 0.852
0.828
0.173
0.652
0.710
0.616
0.185
0.661
0.712 0.650
SCWS [37]
0.082
0.674 0.758
0.787
0.085
0.646 0.768
0.731
0.078
0.837 0.860
0.831
0.170
0.631
0.702
0.613
0.172
0.706
0.738 0.673
TEL [38]
0.089
0.669 0.743
0.761 0.083
0.639 0.776 0.726 0,091
0,810 0.866
0.804
0.230
0,640
0.586
0.536
0.275
0.571
0.501 0.495
SCOD [30]
0.077
0.691 0.795 0.802 0.071 0.664 0.802 0.766 0.071 0.853 0.877 0.836
0.146
0.801
0.778
0.723
0.154
0.743
0.751 0.710
SCOD+
0.074
0,702 0.806 0.803
0.066
0.670 0.811 0.769 0.068 0.860 0.880
0.836
0.129
0.818
0.796
0.732
0.145
0.761
0.765 0.720
Ours
0.043 0,721 0.839 0,816
0.037
0.694 0.849 0.797 0.046 0.878
0.917
0.877
0.078
0.858
0.863 0.775
0.089
0,839
0,841 0.764
the minimum outer wrapping rectangle of the foreground/bacckround scribble and dividing it into nine-box grid, to sample one point in each box and send them to SAM for segmentation mask
generation. Following [2], all images are resized as 352 x 352 in both the training and testing phases. For SAM [19], we adopt the VIT-H SAM model to generate segmentation masks. We implement our method with PyTorch and run experiments on two RTX3090 GPUs
Baselines. We explore SAM [19] for the WSCOS task by generating segmentation masks with sparse annotations as prompts and using the segmentation masks to train a COS segmenter. However, a more straightforward way to explore SAM for this task is to use the sparse annotation to fine-tune SAM and then directly apply SAM for the test. To verify the advantages of our method over this direct way, we construct two baseline methods, SAM-S and SAM-P, which fine-tune the mask decoder of SAM with scribble and point supervisions, respectively, by the partial cross-entrop loss. We will show the results of these two baselines in our comparative evaluations. For reference, we also report the results of the vanilla SAM. When applying SAM and its variants, SAM-S and SAM-P, on test images, we use the automatic prompt generation strategy and report the results with the highest ToU scores.
Metrics. Following existing methods [1, 21, we use four common metrics for evaluation, including mean absolute error (M), adaptive F-measure (Fş) [40], mean E-measure (Eş) [41], and structure measure (S.) [42]. Smaller M, or larger F3, Eg, Sa means better segmentation performance.
4.2 Comparative Evaluation
We perform evaluations on the following COS tasks, namely, Camouflaged Object Detection, Polyp Image Segmentation (PIS), and Transparent Object Detection (TOD). For all the tasks, we evaluate the performance with point annotations. We follow the previous weakly-supervised segmentation method [43] and randomly select two points (one from the foreground and one from the background) from the training masks the point annotations. For COD, we additionally evaluate the performance using scribble annotations, using the scribble data provided in [43].
Camouflaged object detection. Four datasets are used for experiments, i.e., CHAMELEON [44],
CAMO [45], CODIOK [1], and NC4K [16]. Table shows that our method reaches the best performance over all competing methods and basclines. Notably, while SAM has shown impressive
7

---

Table Ablations for WS-SAM.
Table 4: Ablations for MFG.
Baseline MAF PLW ILS M Fat Es t S.
Metrics
w/ MFG
FG-SAA
w/o multiscale WGM-SKC
MIG
10.052 0.674 0.838 0.737
0.044
0.038
0.040
0.039
0,038
0.047 0.689 0,853 0.772
Fa t
0.684
0.708
0.702
0.710
0.719
0.044 0.697 0.866 0.793
Es
0.857
0.868
0.858
0.871
0.878
0.038 0.719 0.878 0.803
Su t
0.780
0.797
0.783
0.792
0.803
Table 5: Results of MFG with full supervision.
Table 6: Results on multi-object images.
Metries| Baseline SegMaR (17)
PreyNet [8] FGANet [15]
Ours
Metrics
SSWS [37]
TEL [38] SCOD [30] Ours
M. 0.035
0.035
0.034
0.032
0.032
M
0.094
0.101
0.084
0.070
Fet f
0.688
0.699
0.715
0.708
0,706
Fe f
0.378
0.350
0.381
0.452
E. 0.879
0.890
0.894
0.894
0.897
E
0.740
0.726
0.718
0.772
8.1
t
0.812
0.813
0.813
0.803
0.813
S., 1
0.625
0.617
0.643
0.687
performance for natural scene images, its performance the challenging COD task is far from the existing methods particularly designed for this task. We do see performance gains after finetuning SAM with point (SAM-P) and scribble (SAM-S) supervision, but the results are still far below our method. This substantiates the superiority of our way of leveraging SAM to generate segmentation masks with sparse annotations and use the segmentation masks train the segmenter. To verify our performance improvement over the existing WSCOS methods does not merely come from the usage of SAM, we integrate the most recent WSCOS method, SCOD [30], into our WS-SAM framework to also leverage the additional mask supervision. This results in the method, "SCOD+". We can see that our method still shows better performance, further verifying our advantages for this task.
Polyp image segmentation. Three widely-used Polyp datasets are selected, namely CVC- ColonDB [46], ETTS [47], and Kvasir [48]. Table 2 shows that our method significantly surpasses the second-best method, SCOD, with point supervision. SAM and SAM-P do not perform well on this task, further substantiating their weakness on this challenging segmentation task. While empowering SCOD with the proposed WS-SAM framework indeed improves the performance, the results are still lower than our method. This again verifies our benefit in handling challenging segmentation tasks.
Transparent object detection. Two datasets, GDD [6] and GSD [29], are used for evaluation. As shown in Table 2, method surpasses all baseline methods and existing methods for this task as well. This shows strong robustness and generalizability of our proposed method.
4.3 Ablation Study
Our method includes two main components, the SAM-based weakly-supervised mask generation framework, WS-SAM, and the multi-scale feature grouping (MFG) module. We conduct ablation studies about these two components on CODIOK of the COD task with scribble supervision.
Ablation study for WS-SAM. We establish a baseline by using SAM to generate only one segmenta- tion mask from one training image without augmentations for model training. On top of this baseline, we add the multi-augmentation fusion (MAF), pixel-level weighting (PLW), and image-level selection (ILS) techniques. Table 3 showing adding these components helps improve the performance, thus demonstrating their effectiveness.
Ablation study for MFG. We examine the effect of MFG by completely removing the MFG module, substituting the proposed feature grouping (FG) with slot attention (SA) [33], removing the multi- scale strategy, and substituting the weighted gate mechanism (WGM) with fixed coefficient (FC). Table 4 shows that our designs reach better performance than the alternative ones.
4.4 Further Analysis
MFG for the fully-supervised setting. The proposed MFG module is designed to evacuate discrimi- native cues from concealed images. We have demonstrated its effectiveness with sparse annotations for the weakly-superviser setting. However, it is expected to also work in the fully-supervised setting. To verify this, conduct experiments for the standard supee-euee COD task. Table 5 shows the results on the CODIOK dataset. We can see that MFG indeed helps improve the performance of the baseline model, the level comparative with state-of-the-ant methods.
Performance on multi-object images. The proposed MFG module evacuates discriminative cues by performing feature grouping at different granularities, which facilitates discovering multiple objects
8

---

Table 7: Parameter analysis on K, Ta. Tr. T, and (N1, N2).
K
7
(N.
No)
Metrics
12
18
0.05
0.7
0.9
(2.4)
(2.3)
(4.8)
12.4.8)
M
0.052
0.042
0.038
0.039
0.037
0.038
0.038
0.040
0.038
0.038
0.039
0.040
0.039
0.039
0,038
0.038
0.038
0.038 0.039
0.038
0.674
0.697
0.719
0.218
0.706
0.719 0.216
0.704
0.723
0.719
0.715
0.200
0.706
0.715
0.719
0.720
0.719
0.714
0.721
0.838
0.857 0.878 0,878
0.868
0.878 0.876 0.865
0.866
0.878
0.874 0.851
0.862
0.872 0.878
0.876
0.878
0.875 0.873
0.878
0.737 0.776 0.803 0.800
0.795
0.803
0.805
0.793
0.799
0.803
0.789 0.781
0.794
0.800
0.803
0.805
0.803
0.802
0.799
0.802
(a) Origin
(b) GT
(C) Ours (d) SCOD
TEL (1) SCWS
(g) WSSA
(h) SAM-P
Figure 4: Visualized results for COD tasks.
images. To verify this, we evaluate the performance on the 186 images with more than one object from COD1OK. Table 6 shows that MFG achieves the best performance, surpassing the second-best method (SCOD) by 10.7%. This gap is large than that with all test images, where the gap is 5.8%.
Randomness of point supervision. We follow the existing point-supervision segmenta- tion methods and randomly select points from ground truth masks as the point annotation. To study the variance of the random selection, we repeat the random selection 5 times and calculate the mean and standard deviation of the results. Fig. 5 shows that our method reaches the best results while having the smallest deviation.
0,010E
06
0,0136
0,0125
Number of augmented views K. Table 7 shows that more aug- mented views help improve performance in the beginning, but the effect turns weaker when further increasing it.
0,5
00167
SAMP
73
W900
Omm
Hyperparameters in image-level selection. Table 7 shows that it
Figure 5: Five runs results
best to set the absolute uncertainty threshold Ta = 0.1 and the relative uncertainty threshold T. 0.5, and our method is not sensitive to these two parameters.
with varied point annotations.
Hyperparameters in MFG. Table 7 shows that MFG achieves the best results when the iteration number 7' is set as 3, and the groups and scales setting is set as (N1, N2) (2,4) 4). Notice that when adopting (2, 4, 8), RK2 is replaced with the third-order RK structure, resulting in extra computational burden with limited benefits. Hence, we select the RK2 structure with (N1, N₂) = (2,
Result visualization. Fig. 4 shows the prediction maps with point supervision. We can see that our method produces more complete results than existing methods and localizes multiple objects more comprehensively. More visualization results can be found in the supplementary materials.
Conclusions
This paper proposes a new WSCOS method that includes two key components. The first one is the WS-SAM framework that generates segmentation masks with the recently proposed vision foundation model, SAM, and proposes augti-augmmenttiinn result fusion, pixel-level uncertainty weighting, and image-level uncertainty filtration to get reliable pseudo labels to train a segmentation model. The second is the MFG module that leverages the extracted clues for additional nuanced discrimination information. MFG improves feature coherence from grouping aspect, allowing for alleviating incomplete segmentation and better multiple-object segmentation. Experiments on multiple WSCOS tasks confirm the superiority of our method over the baseline and existing methods.
Acknowledgements:: This research is partly supported by the National Key R&D Program of China (Grants No. 4020AAA0108300 & 2020AAA01083003 and Shenzhen Science and Tech- nology Project (Grant No. (0Y120000109143044
& Shenzhen Stable Supporting Program (WD7CC220M82220650000 & Shenzhen Key Laboratory of next generation interactive media inno- vative technology (Grant No. DDSYS20210623092001
The authors express their appreciation to Dr. Fengyang Xiao for her insightful comments, improving the quality of this paper.

---

References
[1] Deng-Ping Fan, Ge-Peng Ji. Ming-Ming Cheng, and Ling Shao. Concealed object detection. IEEE Trans. Puttern Anal. Mach. Intell., 2021. 7,7
[2] Deng-Ping, Fan, Ge-Peng Ji, Guolei Sun, Ming-Minn Cheng, Jianbing Shen, and Ling Shao. Camouflaged object detection. In CVPR, pages 2777 2787, 2020. 1,3,7
[3] Chumming He, Kai Li, Yachao Zhang. Yulun Zhang, Zhenhua Guo, Xiu Li, Martin Danelljan, and Fisher Yu. Strategic preys make acute predators: Enhancing camouflaged object detectors by generating camouflaged objects. arXiv preprint 2.38v:2.38..33 /666 2023. 1
[4] Deng- Ping Fan, Ge-Peng Ji, Tao Zhou, Geng Chen, Huazhu Fu, Jianbing Shen, and Ling Shao. Pranet: Parallel reverse attention network for polyp segmentation. In MICCAI, pages 263-273. Springer, 2020. 1,
[5] Ruifei Zhang, Guanbin Li, Zhen Li, Shuguang Cui, Dahong Qian, and Yizhou Yu. Adaptive context selection for polyp segmentation. In MICCAI, pages 253-262. Springer, 2020.
[6] Haiyang Mei, Xin Yang. Yang Wang, Yuanyuan Liu, Shengfeng He, Qiang Zhang, Xiaopeng Wei, and Rynson WH Lau. Don't 't hit me! glass detection in real-world scenes. In CVPR, pages 3687-3696, 2020. 1,
[7] Enze Xie, Wenjia Wang. Wenhai Wang, Mingyu Ding, Chunhua Shen, and Ping Luo. Segmenting transparent objects the wild. In ECCV. Springer, 2020.
[8] Miao Zhang, Shuang Xu, Yongri Piao, Dongxiang Shi. Shusen Lin, and Huchuan Lu. Preynet: Preying on camouflaged objects. In ACM MM, pages 5323 5332, 2022. 1,
[9] Haiyang Mei, Ge-Peng Ji, Ziqi Wei, Xin Yang, Xiaopeng Wei, and Deng-Ping Fan. Camouflaged object segmentation with distraction mining. In CVPR, pages 8772-8781, 2021.
[10] Youwei Pang, Xiaoqi Zhao, Tian Zhu Xiang, Lihe Zhang, and Huchuan Lu. Zoom in and out: A mixed- scale triplet network for camouflaged object detection. In CVPR, pages 2160-2170, 2022. 1
[11] Yijie Zhong, Bo Li, Lv Tang, Senyun Kuang, Shuang Wu, and Shouhong Ding. Detecting camouflaged object in frequency domain. In CVPR, pages 4504-4513, 2022.
[12] Chumming He, Kai Li, Yachao Zhang, Longxiang Tang, Yulun Zhang, Zhenhua Guo, and Xiu Li. Cam- ouflaged object detection with feature decomposition and edge reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages
2023. 1,6
[13] Qiang Zhai, Xin Li. Fan Yang. Chenglizhao Chen, Hong Cheng, and Deng-Ping Fan. Mutual graph learning for camouflaged object detection. In CVPR, pages 12997-13007, 2021. 1
[14] Yujia Sun, Shuo Wang, Chenglizhao Chen, and Tian Zhu Xiang. Boundary coundar--guided camouflaged object detection. arXiv preprint wwXiv2220.000744 2022.
[15] Wei Zhai, Yang Cao, Jing Zhang, and Zheng-Jun Jun Zha. Exploring ligure-ground assignment mechanism in perceptual organization. In NIPS, volume 35, 2022.
[16] Yunqiu Lv, Jing Zhang, Yuchao Dai, Aixuan Li, Bowen Liu, Nick Barnes, and Deng- Ping Fan. Simul- tancously localize, segment and rank the camouflaged objects. In CVPR, pages 11591-11601, 2021.
[17] Qi Jia, Shuilian Yao, Yu Liu, Xin Fan, Risheng Liu, and Zhongxuan Luo. Segment, magnify and reiterate: Detecting camouflaged objects the hard way. In CVPR, pages 4713-4722, 2022. 8
[18] Hongwei Zhu, Peng Li, Haoran Xie, Xuefeng Yan, Dong Liang, Dapeng Chen, Minggiang Wei, and Jing Qin. can find you! boundary-guided separated attention network for camouflaged object detection. In AAAL, volume 36, pages 3608 -616, 2022.
[19] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. arXiv preprint arXiv:2304.02645 2023. 3,7
[20] Ge-Peng Ji, Deng- Ping Fan, Peng Xu, Ming Ming Cheng, Bowen Zhou, and Lue Van Gool. Sam struggles in concealed scenes- cemereeal study on" segment anything". arXiv preprint 06X00 2023. 2
10

---

[21] Lv Tang, Haoke Xiao, and Bo Li. Can sam segment anything? when sam meets camouflaged object detection. arXiv preprint urXiv:2304.04709. 2023.
[22] Wei Ji, Jingjing Li. Qi Bi, Wenbo Li, and Li Cheng. Segment anything is not always perfect: An investigation of sam on different worl-world applications. arXiv preprint arXiv:2304.05770 055500 2023. 2
[23] Macicj Mazurowski, Haoyu Dong, Hanxue Gu, Jichen Yang, Nicholas Konz, and Yixin Zhang. Segment anything model for medical image analysis: an experimental study. arXiv preprint arXiv:2304.15111 2023. 2
[24] Dongjie Cheng, Ziyuan Qin. Zekun Jiang, Shaoting Zhang, Qicheng Lao, and Kang Li. Sam on medical images: A comprehensive study on three prompt modes. arXiv preprint arXiv:2305.00000 00035, 2023.
[25] Christian Mattjie, Luis Vinicius de Moura, Rafacla Cappelari Ravazio, Lucas Silveira Kupssinskü, Otávio Parraga, Marcelo Mussi Delucis, and Rodrigo Coelho Barros, Exploring the zero-shot capabilities of the segment anything model (sam) in 2d medical imaging: comprehensive evaluation and practical guideline. arXiv preprint arXiv:2305.00100 2023.
[26] Tao Zhou, Yizhe Zhang. Yi Zhou, Ye Wu, and Chen Gong. Can sam segment polyps? arXiv preprint orXiv:2304.07588 2023.
[27] Chunming He, Kai Li, Guoxia Xu, Jiangpeng Yan, Longxiang Tang, Yulun Zhang, Xiu Li, and Yaowei Wang. Hqg-net: Unpaired medical image enhancement with high-quality guidance. arXiv preprint arXiw:2337.077829 2023. 2
[28] Dongsheng Han, Chaoning Zhang, Yu Qiao, Maryam Qamar, Yuna Jung, SeungKyu Lee, Sung-Ho Bae. and Choong Seon Hong. Segment anything model (sam) meets glass: Mirror and transparent objects cannot be easily detected. arXiv preprint XXix:2305.00276 2023. 2
[29] Jiaying Lin, Zebang He, and Rynson WH Lau. Rich context aggregation with reflection prior for glass surface detection. CVPR, pages 13415-13424, 2021. 3,8
[30] Ruozhen He, Qihua Dong, Jiaying Lin, and Rynson WH Lau. Weakly-supervised camouflaged object detection with scribble annotations. AAAI, 2023. 3, 6, 7,8
[31] Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, pages 770-778, 2016. 6
[32] Chumming He, Kai Li, Guoxia Xu, Yulun Zhang, Runze Hu, Zhenhua Guo, and Xiu Li. Degradation- resistant unfolding network for heterogeneous image fusion. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 12611-12621, 2023.
[33] Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold. Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot attention. NIPS, 33: 11535-1138,, 2020. 6,8 8
[34] Longxiang Tang, Kai Li, Chunming He, Yulun Zhang, and Li. Source-free domain adaptive fundus image segmentation with class-balanced mean teacher. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 684-694. Springer, 2023. 6
[35] Longxiang Tang, Kai Li, Chunming He, Yulun Zhang, and Xiu Li. Consistency regularization for generalizable fouree-free domain adaptation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4323-4333, 2023.
[36] Jing Zhang, Xin Yu, Aixuan Li, Peipei Song, Bowen Liu, and Yuchao Dai. Weakly-supervised salient object detection via scribble annotations. CVPR, pages 12546-12555, 2020.
[37] Siyue Yu, Bingfeng Zhang. Jimin Xiao, and Eng Gee Lim. Structure-consiseent weakly supervised salient object detection with local saliency coherence. In AAAI, volume 35, pages 3234-3334300 3242, 2021. 7,8
[38] Zhiyuan Liang, Tiancai Wang, Xiangyu Zhang. Jian Sun, and Jianhing Shen. Tree energy loss: Towards sparsely annotated semantic segmentation. In CVPR, pages 16907-16916, 2022. 7,8 8
[39] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, pages 248-255. Ieee, 2009, 6
[40] Ran Margolin, Lihi Zelnik-Manor, and Ayellet Tal. How to evaluate foreground maps? In CVPR, pages 248-255. 225, 2014.7 7

---

[41] Deng-Ping Fan, Ge-Peng Ji. Xuebin Qin, and Ming-Ming Cheng. Cognitive vision inspired object segmentation metric and loss function. Scientia Sinica Informationis, 6(6). 2021.
[42] Deng-Ping Fan, Ming-Ming Cheng, Yun Liu, Tao Li, and Ali Borji. Structure-measure: A new way evaluate foreground maps. In ICCV, pages 4548 4557, 2017.
[43] Shuyong Gao, Wei Zhang. Yan Wang. Qianyu Guo, Chenglong Zhang, Yangji He, and Wenqiang Zhang. Weakly-supervised salient object delection using point supervison. In AAAI, volume 36, pages 670-678,
2022. 7
[44] Przemyslaw Skurowski, Hassan Abdulameer, Blaszczyk, Tomasz Depta, Adam Kornckii, and P Koziel. Animal camouflage analysis: Chameleon database. Unpublished munuscript. 2(6):7. 2018.
[45] Trung-Nghia Le, Tam V Nguyen, Zhongliang Nie, Minh-Triet Tran, and Akihiro Sugimoto. Anabranch network for camouilaged object segmentation. Comput. Vis. Image Underst., 184:45 56, 2019.
[46] Nima Tajbakhsh, Suryakanth R Gurudu, and Jiamming Liang, Automated polyp detection in colonoscopy videos using shape and context information. IEEE Trans. Med. Imaging, 35(2):630-644, 2015.
[47] Juan Silva, Aymeric Histace, Olivier Romain, Xavier Dray, and Bertrand Granado. Toward embedded detection polyps in wee images for early diagnosis of colorectal cancer. Int. Comput. Assist. Radiol. Surg. 9:283-293, 2014.
[48] Debesh Jha, Pia Smedsrud, Michael A Riegler, Halvorsen, Thomas Lange, Dag Johansen, and Hävard D Johansen. Kvasir-seg: A segmented polyp dataset. In MMM, pages 451-462 Springer, 2020.
12


Reviews:
[{"review_id": "uk0g5SIVEe", "reviewer": "Reviewer_76do", "summary": "The submission presents a method for Weakly-Supervised Concealed Object Segmentation (WSCOS) to address the challenges of segmenting objects that are well blended with their surroundings using sparsely-annotated data. \n+ The method introduces a multi-scale feature grouping module that groups similar features at different scales, promoting segmentation coherence and producing accurate results for single and multiple-object images. \n+ The authors leverage the Segment Anything Model (SAM) and utilize sparse annotations as prompts to generate segmentation masks for training the model. \n+ Additionally, the paper proposes strategies such as multi-augmentation result ensemble, entropy-based pixel-level weighting, and entropy-based image-level selection to mitigate the impact of low-quality segmentation masks and provide more reliable supervision during training. \n+ Experimental results demonstrate the effectiveness of the proposed WSCOS method.", "questions": "See above concerns.", "limitations": "No significant limitations.", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "This paper exploits the hot SAM for weakly-supervised COS, which is a good point. This paper is well-written and easy to read and understand. They verify the method on several interesting applications. Still, there have several concerns during my reviewing, please see the below sector.", "weaknesses": "The performance improvements may come from the strong capability and prediction of SAM, but not from the proposed weakly-supervised strategy. The observed performance of SCOD+ does not significantly surpass that of SCOD when integrated with the proposed WS-SAM framework. Thus, the proposed modules do not seem to offer a promising and significant solution to this challenging task.\n\nThe comparison might not be fair due to the introduction of an additional model (SAM) to generate weak labels. Could the authors elaborate on this, particularly in comparison to the other methods displayed in Table 1?\n\nSeveral of the latest and SOTA weakly-supervised semantic segmentation methods, such as [ref1] \"Weakly Supervised Semantic Segmentation via Adversarial Learning of Classifier and Reconstructor\", [ref2] \"Out-of-Candidate Rectification for Weakly Supervised Semantic Segmentation\", and [ref3] \"Weakly Supervised Semantic Segmentation via Adversarial Learning of Classifier and Reconstructor\" could be applied to this challenging task.\n\nOverall, this submission falls short of the standard for a top-tier conference due to its weak experimental validations. I am inclined to recommend borderline rejection in the first round.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the valuable comments. If not specifically stated, all experiments are conducted on the COD task with *COD10K* for space limitation.\n\n**Q1: Performance gains are from SAM, not from the weakly-supervised strategy.**\n\nWhile SAM has shown impressive performance for natural scene images, its performance on concealed images is far from satisfactory, even using points/scribbles as the prompts. The table presented below (**Table A1** in the attained PDF) shows that naively employing the SAM-generated masks (without using our weakly-supervised strategy) to train SCOD results in performance (SCOD+SAM) that is even lower than SCOD. This indicates that low-quality masks generated by SAM bring negative effects to train the model. On the other hand, training SCOD with our weakly-supervised strategy backed SAM, the result (SCOD+WS-SAM) is higher than SCOD. Similar observations can be found in applying SAM and WS-SAM on TEL and our segmenter. This indicates our weakly-supervised strategy successfully increases the quality of the generated masks, and enhance improve the performance. \n\nSCOD+  (SCOD+WS-SAM)  overall surpasses SCOD by **1.5\\%** (scribble) and **2.4\\%** (point). For some tasks, the improvement gains can reach up to **3.6\\%**. While it might be true that the numeric difference looks not significant, this is partially due to the used metrics tending to scale down the performance difference. Also COD is a very challenging task such that a ~1% gain is usually recognized as a valuable gain by the community. \n\n|Scribble|$M\\downarrow$|$F_\\beta\\uparrow$|$E_\\phi\\uparrow$|$S_\\alpha\\uparrow$|\n|-|-|-|-|-|\n|TEL[34]|0.057|0.633|0.826|0.724|\n|TEL+SAM|0.059|0.614|0.810|0.716|\n|TEL+WS-SAM|0.053|0.646|0.838|0.730|\n|SCOD[29]|0.049|0.637|0.832|0.733|\n|SCOD+SAM|0.050|0.623|0.813|0.725|\n|SCOD+WS-SAM|0.047|0.650|0.845|0.742|\n|Ours+SCOD|0.045|0.692|0.853|0.759|\n|Ours+SAM|0.052|0.674|0.838|0.737|\n|Ours+WS-SAM|0.038|0.719|0.878|0.803|\n\n**Q2: Comparison is not fair due to the use of SAM.**\n\nThe proposed WS-SAM is a general framework that can be integrated with various existing methods and improve performance. WS-SAM offers a solution orthogonal to existing methods by leveraging the power of SAM. To make fair comparisons, we not only compared with the existing WSCOS method, e.g., SCOD, and also compare with the variants that integrate SAM with SCOD (SCOD+SAM) and our WS-SAM with SCOD (SCOD+WS-SAM). The results in the table above show that our proposed WS-SAM framework can improve these methods. \n\n**Q3: Comparison with existing weakly-supervised semantic segmentation methods.**\n\nWe focused on concealed object detection and only compared existing methods in this field. As per your request, we train these Weakly-Supervised Semantic Segmentation (WSSS) methods (for which code is open-sourced) and get the table below (**Table A1** in the attained PDF).  We can see that these WSSS methods struggle on the challenging COD task, but their performance can be improved when empowered by our WS-SAM framework. This further substantiates the effectiveness of WS-SAM. \n\n|Scribble|$M\\downarrow$|$F_\\beta\\uparrow$|$E_\\phi\\uparrow$|$S_\\alpha\\uparrow$|\n|-|-|-|-|-|\n|ACR[46]|0.050|0.622|0.828|0.728|\n|ACR+WS-SAM|0.048|0.637|0.836|0.735|\n|OCR[47]|0.052|0.607|0.796|0.704|\n|OCR+WS-SAM|0.048|0.636|0.830|0.727|\n\n**Q4: Weak experimental validations.**\n\nWe have validated the superiority of our method on the COS task and the SOD task. The effectiveness of WS-SAM and MFG has also been demonstrated by abundant ablation studies.\n\nDuring the rebuttal period, we conducted comprehensive comparisons between our segmenter and several state-of-the-art (SOTA) segmenters, as highlighted in **Tables A1, A6, A7, A8**. These comparisons clearly underscore the superior performance and efficiency of our segmentor. Furthermore, we extended our assessment to verify the generalizability of our WS-SAM framework, as shown in **Tables A1, A5, A9**, effectively demonstrating its robustness. Notably, we showcased the versatility of our Module Fusion Gateway (MFG) as a plug-and-play component that significantly enhances the segmentation capabilities of existing segmenters, as indicated in **Table A3**. \n\nIn our pursuit of evaluating the efficacy of WS-SAM, we engaged in comparisons with existing strategies, unequivocally establishing our framework's superiority in addressing the COS task, as evidenced in **Table A2**. To comprehensively explore the potential of SAM, we introduced multiple modifications, including a learnable SAM baseline (**Tables A4, A5**), SAM-assisted refinement of pseudo labels (**Tables A9**), and leveraging SAM's \"segment-everything\" opinion for concealed object segmentation (**Tables A10**). Through these meticulous explorations, we uncovered that even with these modifications, segmenters still encounter challenges in the COS task, yet our WS-SAM framework consistently elevates their performance to a leading position.\n\nCollectively, these extensive investigations decisively illustrate that WS-SAM stands as a potent and versatile framework with remarkable generalizability for the concealed object segmentation task. To enhance clarity, we have included a diagram illustrating the nine-box strategy in **Fig. A2**, along with a visualization depicting the similarity among group prototypes and grid features in **Fig. A1**. We intend to incorporate all these experimental validations into our revised version for a more comprehensive representation of our work.\n\n[45]  LSR, CVPR, 2021.\n\n[46] ACR, CVPR, 2023.\n\n[47] OCR, CVPR, 2023."}, {"review_id": "l2OKxqieJ0", "reviewer": "Reviewer_yhjR", "summary": "This paper investigates Concealed Object Segmentation based on the recently proposed foundation model SAM, which provides dense annotation for learning. To alleviate the la-bel noise caused by imperfect segmentation results of SAM on concealed objects, they adopt pixel-level and image-level uncertainty-based strategies to weight predictions and select reasonable training samples. To tackle the intrinsic similarity problem in the task, they design a multi-scale grouping module to group features from different levels. Further, they conduct experiments on commonly used datasets, which demonstrates better performance.", "questions": "1. In the Supp., the authors describe that they adopt the encoder-decoder architecture utilized in a more recent work FEDER. But they do not compare the results with this model in their experiments.\n\n2. The ablation in Table 4 does not show the superiority of their sophisticated design choices, where the margins among experiment results are narrow.", "limitations": "Please refer to the weakness part.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "+ To alleviate the sparse-annotated challenge, this work utilizes the recently proposed foundation model SAM to augment the label, along with entropy-based acquisition strategies, to alleviate the impact of low-quality segmentation masks produced by SAM.\n\n+ To tackle the inherent similarity problem in COD/COS, they introduce multi-scale features grouping technique to group features with different granularities.\n\n+ To demonstrate the efficacy of the proposed method, they conduct experiments on several concealed datasets compared with recent methods and ablate their design components.\n\n+ The proposed model achieves high performance from the experimental results.", "weaknesses": "- ”Weakly-supervised” claim is not appropriate in this work. Since the segmentation mask generated by SAM is close to the full supervision setup but carries noise, instead of sparse as the weak-supervised task definition, such a claim here is inappropriate.\n\n- Lack of novelty. The proposed entropy-based acquisition strategies are classic methods used in active learning to sample informative samples and semi-supervised learning tasks to generate pseudo labels. Moreover, multi-augmentation fusion is also a popular technique in semi-supervised learning works. The aforementioned existing methods may refer to the following literature (but not limited to):\n\na. Settles, Burr. ”Active learning literature survey.” (2009).\nb. Ren, Pengzhen, et al. ”A survey of deep active learning.” ACM computing surveys (CSUR) 54.9 (2021): 1-40.\nc. Iscen, Ahmet, et al. ”Label propagation for deep semi-supervised learning.” Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2019.\nd. Berthelot, David, et al. ”Mixmatch: A holistic approach to semi-supervised learning.” Advances in neural information processing systems 32 (2019).\n\n- Literature review is not sufficient. There are more works related to the Concealed Object Detection/Segmentation task, which are not included and discussed the difference with the proposed method in the paper.\n\n- The mask generation method in Eq.5 is wrong. First, $\\tilde{E_i}$ is an entropy mask. Its value ranges from [0, ln2] rather than [0,1]. In Eq.5, the author uses $(1-\\tilde{E_i})$ to get the reverse mask is not the correct way. Second, the author regards $(1-\\tilde{E_i})$ as a certainty for pixels. They should use $(1-\\tilde{E_i})$ as weighting coefficients and multiply $(1-\\tilde{E_i})$ to cross-entropy loss to calculate the weighted loss. However, the author directly multiplies certainty $(1-\\tilde{E_i})$ to label $\\tilde{M_i}$, which causes the pseudo mask to no longer be 0 or 1, but a decimal point. For example, on pixel in $\\tilde{M_i}$ equals 1, but it will become 0.5 when $(1-\\tilde{E_i})=0.5$. Using the label=0.5 to supervise the segmentation model does not make sense. Third, the author proposes the image selection value \\hat{\\alpha} to remove images with high uncertainty. However, they multiply the $\\hat{\\alpha}$ to the pseudo mask in Eq.5, which confuses the reviewer. Because this operation will generate an all-zero mask as the pseudo mask. Obviously, using the all-zero mask to supervise modeling training brings the wrong guidance. The authors need to repolish the theoretical part of the paper.\n\n- The experiment is not convincing. In line 241, the authors set two baselines SAM-S and SAM-P but only fine-tune the decoder part and use the automatic prompt generation strategy. This way can not show the generalization ability of SAM. The authors should adopt the SAM encoder as the backbone network, finetune the whole model and abandon the automatic prompt generation strategy. Only in this way can we understand the generalization of SAM to the COD task. Furthermore, by comparing the way of pseudo label generation and the way of encoder finetuning, the reviewer can determine whether we need better pseudo labels for COD or just a SAM backbone.\n\n- The writing is not clear enough, especially the use of symbols is confusing. For example, $\\phi_p$ in the caption of Figure 3 is not displayed in the figure. \"S\" in line 204 is given without any explanation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the valuable comments. Unless otherwise specified, all experiments are conducted on the *COD10K* dataset for the COD task to save space.\n\n**Q1: ''Weakly-supervised'' claim is not appropriate.**\n\nWe follow SCOD [29] and use the term ''weakly-supervised'' to indicate that training data only provide weak supervision in the forms of sparse points or scribble. While it is true that we use dense masks to train our segmenter, the dense masks are generated from the sparse annotations and they are not reliable. In **Table A1** in the attached file, we show that training with these unreliable dense masks brings negative effects and only refined with our proposed techniques, the dense masks can bring performance gains.  \n\n**Q2: Novelties.**\n\nWS-SAM offers the **first solution** to address the WSCOS task that is **orthogonal to existing methods** by leveraging the power of SAM. We are inspired by the impressive capability of SAM on natural scene images and extend it to conceal object images by designing multiple effective techniques to address the limitations. While these techniques may be rooted in some existing techniques used in various tasks, we are first to explore and adapt them to address our target problems. More concrete differences are as follows. \n\n- *Entropy-based acquisition strategy*: Our difference compared with semi-supervised learning methods is that we use the entropy-based metric to perform both pixel-level weighting and image-level selection, previous methods usually include either of them. \n\n- *Multi-augmentation fusion strategy*:  Different from Mixmatch [50] using self-ensemble strategies to minimize the variance among pseudo labels, our purpose is to fully explore the potential of SAM to balance the accuracy and completeness of pseudo labeling (line 138). To achieve this, we employ flipping, rotation, and scaling for image augmentations rather than the elastic deformation and salt-and-pepper noise used in Mixmatch. To verify this, we replaced our augmentation strategies with that used in Mixmatch and found a decrease in performance (**Table A2** in the attached file).\n\n**Q3: Discussion with more related works.**\n\nWe have cited (~20) most recent COS methods. We will search and add more relevant ones in the revised version. \n\n**Q4: Some formulas.**\n\nWe double-checked our implementation and confirmed there are some subtle differences between the implementation and the writing. \n\n- *$\\tilde{\\mathbf{E}}_i$ in Eq. (3)*. We calculate our entropy mask using the logarithm of base 2 instead of e. Therefore, the value ranges of $\\tilde{\\mathbf{E}}_i$ is [0,1]. We will highlight this in the revised version.\n\n- *Eqs. (5) and (11)*. The idea is to use a mask to assign higher weights to reliable pixels and discard low-quality pseudo labels when calculating the cross-entropy loss. We implement this as the weighted cross-entropy loss. However, in our writing, we thought that this practice was equivalent to multiplying the mask with the binarized pseudo labels. However, after carefully checking the math, we realize that there is a subtle difference between these two formulations. The weighted cross-entropy loss applies the mask outside of the logarithmic operator, but multiplying the mask with the binarized pseudo labels applies the mask inside the logarithmic operator, which causes problems as you described. So, the formulations for Eq (5) and Eq. (11) that match our implementation should be\n\n$$\\hat{\\mathbf{Y}}_i=(1-\\tilde{\\mathbf{E}}_i)\\times\\hat{\\mathbf{M}}_i.$$\n\n$$L=\\frac{1}{N_s}\\sum_{(\\mathbf{X}_i,\\mathbf{Y}_i)\\sim\\mathcal{S}}L\\_{pce}(\\mathbf{Y}_i',\\mathbf{Y}_i)+{\\hat{\\mathbf{Y}}_i}L\\_{ce}(\\mathbf{Y}'_i, \\tilde{\\mathbf{M}}_i)+{\\hat{\\mathbf{Y}}_i}L\\_{IoU}(\\mathbf{Y}'_i, \\tilde{\\mathbf{M}}_i).$$\n\nWe will correct the two equations in the revised version and release our code.\n\n**Q5: Another learnable SAM baseline.**\n\nWe construct the baseline, SAM-L, as described and train SAM-L for the COD, PIS, and TOD tasks. We report the results from *CVC-ColonDB* in PIS and from *GDD* in TOD. **Table A4** (in the attached file) shows that SAM-L still struggles in the COS task. This is mainly because of the sparse annotation and the limited training data, which leads to overfitting when training SAM. Such discovery indicates that directly utilizing SAM to address those extreme tasks, such as COS, maybe a sub-optimal solution.\n\nSimilar to Table 5 in the Supp, we explore the quality of pseudo labels generated by SAM-L, report the results in **Table A5**, and achieve a similar conclusion to that in lines 115-122 in the Supp. Such findings verify that our WS-SAM framework is crucial to SAM in addressing the weakly-supervised COS task.\n\n**Q6: Writing clarification.**\n\n$\\Phi_P$ is a generic term that stands for $\\Phi_{N_1}$ and $\\Phi_{N_2}$. $S$ in line 204 denotes Sigmoid. We will modify them accordingly.\n\n**Q7: Comparison with FEDER.**\n\nWe compare our segmenter with FEDER [51] in weak supervision (with WS-SAM) and full supervision. **Table A6** in the attached PDF shows that our method outperforms FEDER in weak supervision and has comparable performance to FEDER in full supervision. Besides, our method is 28.3\\% lighter than FEDER overall in the full supervision (see **Table A7**).\n\n**Q8: Narrow gain of MFG.**\n\nTable 4 shows that MFG overall brings an improvement of **6.0\\%**, which is significant in this field.  The absolute value differences look small because the used metrics tend to scale down the performance difference.  Furthermore, employing feature grouping, multiscale strategy, and weighted gate mechanism can correspondingly improve the performance by **0.9\\%**, **3.1\\%**, and **1.5\\%**, which are recognized as significant for the used metrics in the field. In **Table A3**, we further integrate our MFG with other segmenters and get performance gains.\n\n[48] A survey of deep active learning, CSUR, 2021.\n\n[49] LPSSL, CVPR, 2019.\n\n[50] Mixmatch, NIPS, 2019.\n\n[51] FEDER, CVPR, 2023."}, {"review_id": "OxnuDCkHyF", "reviewer": "Reviewer_ka5E", "summary": "This paper tackles the problem of Weakly-Supervised Concealed Object Segmentation (WSCOS). The first strategy is using SAM, which is a vision foundation model for generic segmentation. To handle the miss-segmentation of SAM, the authors propose techniques using multi-augmentation and entropy-based scoring. In addition, to deal with the similarity challenge of FG and BG, Multi-scale Feature Grouping (MFG) is proposed. The overall experiments show that the proposed method achieves state-of-the-art performance.", "questions": "Authors mention that directly using SAM sometimes miss-segment the confusing regions. Have the authors tried the segment-everything option of SAM? I'm not sure about it, but in such confusing cases, this option usually provides better results than using point (or scribble) prompts.", "limitations": "NA", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Using SAM is one of the promising directions in the field of segmentation. It is good to see a SAM-based weakly-supervised segmentation approach.\n2. Entropy-based weighting and selection methods are technically sounding.", "weaknesses": "1. It seems that the baseline of the proposed method is already quite higher than the existing methods. Could the authors provide the backbone architecture of the proposed method and existing ones, and match them for fair comparison?\n\n2. SAM also can be used to refine the pseudo label. One of the most simple approaches is using the conventional WSCOS method and then refining the obtained pseudo label with SAM. Since the proposed method is integrated with SAM in a learning phase, it should achieve better performance than the refining strategy. Please provide the comparison in detail. \n\n3. I want to check the performance of using SAM (PLW+ILS) alone, without MFG. I'm not sure but the gain of MFG can be overlapped with that of using SAM.\n\n4. Inference using SAM is quite heavy in terms of both time and memory. This drawback is particularly pronounced when inferring images with multiple augmentations. This computational burden should be addressed in this paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the valuable comments. Unless otherwise specifically stated, all experiments are conducted on the *COD10K* dataset for the  COD task to save space.\n\nQ1: Fair comparison with the same backbone.\n\nQ1-A: In Table 3, ''Baseline'' denotes using SAM to generate only one segmentation mask from one training image without augmentations for model training (line 281) rather than the backbone architecture of our segmenter. We further provide the results of TEL and SCOD with our backbone architecture, which are denoted as TEL' and SCOD'. In the table below, our segmenter still outperforms the compared methods with the same backbone.\n|Scribble|$M\\downarrow$|$F_\\beta\\uparrow$|$E_\\phi\\uparrow$|$S_\\alpha\\uparrow$|\n|-|-|-|-|-|\n|TEL'|0.056|0.635|0.837|0.716|\n|SCOD'|0.047|0.642|0.833|0.736|\n|Ours|0.038|0.719|0.878|0.803|\n\nQ2: Refine pseudo labels with SAM.\n\nQ2-A: We conduct detailed experiments to explore the capacity of SAM in the refinement of pseudo labels.\n\n- We employ a pretrained segmenter, e.g., SCOD [29] or our segmenter, to generate coarse pseudo labels and treat those pseudo labels as dense prompts to SAM for further refinement. We then use the refined pseudo labels to directly supervise our segmenter without the usage of our refinement strategy in the training phase. Note that the modifications only lie in that we add a pretrained segmenter to provide dense prompts for SAM and remove the proposed refinement strategy for the generated pseudo labels.\n\n- In the implementation, we provide two compared methods, termed Ours-1 and Ours-2. Ours-1 and Ours-2 mean using pretrained SCOD and our segmenter to generate the initial pseudo labels, respectively. As shown in **Table A9** in the attached file, we report the results in both scribble and point supervision and find that our segmenter achieves the best results when being trained with our WS-SAM framework.\n\n- Having analyzed the results, we discover that the results of Ours-1 and Ours-2 both surpass those of the ''Baseline'' reported in Table 3, which indicates that refining pseudo labels with SAM can really promote the quality of pseudo labels. However, in **Table A9**, when further reinforced by our WS-SAM framework, the leaderships of Ours-1 and Ours-2 are mitigated and have a comparative performance with our segmenter. This indicates the effect of our WS-SAM in generating high-quality pseudo labels to supervise the segmenter.\n\n|Scribble|$M\\downarrow$|$F_\\beta\\uparrow$|$E_\\phi\\uparrow$|$S_\\alpha\\uparrow$|\n|-|-|-|-|-|\n|Ours-1|0.047|0.688|0.851|0.765|\n|Ours-2|0.046|0.693|0.855|0.772|\n|Ours-1+WS-SAM|0.038|0.716|0.877|0.805|\n|Ours-2+WS-SAM|0.038|0.720|0.873|0.802|\n|Ours+WS-SAM|0.038|0.719|0.878|0.803|\n\n|Point|$M\\downarrow$|$F_\\beta\\uparrow$|$E_\\phi\\uparrow$|$S_\\alpha\\uparrow$|\n|-|-|-|-|-|\n|Ours-1|0.053|0.630|0.811|0.732|\n|Ours-2|0.050|0.641|0.820|0.747|\n|Ours-1+WS-SAM|0.039|0.700|0.853|0.782|\n|Ours-2+WS-SAM|0.039|0.695|0.855|0.790|\n|Ours+WS-SAM|0.039|0.698|0.856|0.790|\n\nQ3: Effect of MFG.\n\nQ3-A: MAF in Table 3 denotes multi-augmentation fusion rather than MFG. Table 4 indicates that the use of MFG brings a **6.0\\%** improvement when using the proposed MS-SAM framework. In the following table (**Table A3**), we further add our MFG to other segmenters and find performance gains.\n\n|Scribble|$M\\downarrow$|$F_\\beta\\uparrow$|$E_\\phi\\uparrow$|$S_\\alpha\\uparrow$|\n|-|-|-|-|-|\n|TEL[34]|0.057|0.633|0.826|0.724|\n|TEL+MFG|0.054|0.642|0.833|0.731|\n|SCOD [29]|0.049|0.637|0.832|0.733|\n|SCOD+MFG|0.046|0.672|0.840|0.744|\n\nQ4: Computational burden from SAM.\n\nQ4-A: SAM and the multi-augmentation strategy are only employed in the training phase. Additionally, we propose a scheme that generates all augmented images offline using SAM and then fuses the final supervised results with randomly selected images from these sets during online training. This is the same as using multiple augmentations for online inference with SAM when we are training the segmenter. Therefore, such behavior brings no performance decrease but greatly reduces the computational burden required. The computational burden of the proposed pseudo-label refinement strategy, including pixel-level weighting and image-level selection, is almost negligible, thus ensuring our efficiency. In **Table A8**, we further provide our parameters and FLOPs in such a scheme and find that we achieve the best performance among all compared methods in the setting of weak supervision.\n\n|Weak|WSSA[32]|SCWS[33]|TEL[34]|SCOD[29]|Ours|\n|-|-|-|-|-|-|\n|Parameters|53.31|58.83|48.62|49.93|31.12|\n|FLOPs|65.87|77.18|69.12|58.63|40.06|\n\nQ5: Can the segment-everything option of SAM work in COS?\n\nQ5-A: When using the segment-everything (SE) option, SAM outputs a series of binarized segmentation results and sorts them in order of IoU scores. We choose the segmentation mask with the highest IoU score and refine the mask with our WS-SAM framework. From the table (**Table A10**), our segmenter achieves better results in both scribble and point supervision. This is because point or scribble actually provides important localization information. Whereas, the output of SE with the highest IoU score may not have accurate localization, leading to the generation of low-quality pseudo-labels.\n\n|Scribble|$M\\downarrow$|$F_\\beta\\uparrow$|$E_\\phi\\uparrow$|$S_\\alpha\\uparrow$|\n|-|-|-|-|-|\n|SE|0.044|0.662|0.849|0.762|\n|Ours|0.038|0.719|0.878|0.803|\n\n|Point|$M\\downarrow$|$F_\\beta\\uparrow$|$E_\\phi\\uparrow$|$S_\\alpha\\uparrow$|\n|-|-|-|-|-|\n|SE|0.046|0.646|0.820|0.749|\n|Ours|0.039|0.698|0.856|0.790|\n"}, {"review_id": "5MVXubrH3n", "reviewer": "Reviewer_DmES", "summary": "This paper proposes to tackle Concealed Object Segmentation tasks by generating high-quality pseudo segmentation masks using state-of-the-art vision foundation models--SAM.  SAM takes prompts as provided sparse annotations (e.g. points or scribbles) to produce figure-ground pixel labels, which are used to train the segmenter.  Yet, the task remains challenging as SAM is not robust to out-of-distribution images where objects visually blended with the surrounding environment.  This paper proposes to infer and select high-quality pseudo labels using 1) multi-augmentation result fusion and 2) entropy-based label filtering.  Additionally, this paper proposes a multi-scale feature grouping machinery to enhance the performance of the segmenter.  The proposed method is demonstrated on multiple benchmark datasets and achieves good results.\n\n", "questions": "1.  How do you use scribbles as prompts to the SAM?  It would also be nice to include a brief formulation about how SAM generate masks for self-completeness.\n\n2. I think the formulation in Sec 3.2 might be wrong (as far as I understand).  In my understanding, just like slot attention, the cluster prototypes $P$ aggregates pixel features with weighted-mean pixel-to-prototype attention. (See Eqn 1 and 2 in Slot Attention paper)\n    * In Eqn 7, $A$ should be $A = \\frac{1}{\\sqrt{C}} K Q^\\top$ instead\n    * In Eqn 8, $D$ should be $D_{i, j} =\\frac{\\bar{A_{i, j}}}{\\sum_l \\bar{A_{l, j}}}$\n    * How to broadcast cluster prototype to 2D grid?\n    * What is $S$ in Eqn 10?\n\n3. I think it would be interesting to see what are learned in the cluster prototype at different granularity.  It would be nice if we can see the visualization of feature similarity (or attention) among cluster prototype and grid features for $N_1$ and $N_2$.", "limitations": "The authors do not describe the limitations of the proposed method.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The motivation is elaborated well.  It's easy to understand what the tasks and the challenges are.\n2. The performance is good among most of the benchmark datasets.", "weaknesses": "1. Although the motivation is illustrated clearly, this paper does not provide good review of the literature, given the fact that the vision community has made great success for weakly-supervised segmentation tasks.  For example, I list some reference by searching through google scholar.\n    * Weakly-supervised semantic segmentation:\n       -  Token contrast for weakly-supervised semantic segmentation. Ru et al. CVPR 2023\n       -  Ts-cam: Token semantic coupled attention map for weakly supervised object localization. Gao et al. ICCV 2021.\n       -  Multi-class token transformer for weakly supervised semantic segmentation. Xu et al. CVPR 2022.\n       -  Learning affinity from attention: End-to-end weakly-supervised semantic segmentation with transformers. Ru et al. CVPR 2022.\n       -  Universal Weakly Supervised Segmentation by Pixel-to-Segment Contrastive Learning. ICLR 2021.\n       - ScribbleSup: Scribble-Supervised Convolutional Networks for Semantic Segmentation. Lin et al. CVPR 2016.\n    * Weakly-supervised instance segmentation\n       -  Weakly supervised instance segmentation using class peak response. Zhou et al. CVPR 2018.\n       -  Simple does it: Weakly supervised instance and semantic segmentation. Khoreva et al. CVPR 2017.\n       -  Boxinst: High-performance instance segmentation with box annotations. Tian et al. CVPR 2021.\n\n2. Generally speaking, the proposed machineries--multi-augmentation fusion and entropy-based filtering, are not novel.  These techniques are commonly used in weakly-supervised segmentation literature, e.g. AffinityNet[1].  The contribution of this paper is more like to employ SAM to tackle weakly-supervised concealed segmentation tasks.\n\n3. I think the major contribution of this paper is not highlighted.  If the authors believe multi-feature grouping module is the major contribution, then demonstrating on the concealed object segmentation is not enough.  It would be essential to show its efficacy on other segmentation tasks.  If the authors believe the employment of SAM is the major contribution, then I think the novelty of this paper is not impressive.  The multi-augmentation fusion and entropy-based filtering tricks are well known in the literature.  It would be nice to see what are the distinctive perspectives or challenges when deploying SAM on the tasks.  Overall, the proposed method is reasonable but not novel enough.\n\n4. The authors do not describe the limitations of the proposed method.\n\n[1]: Learning Pixel-level Semantic Affinity with Image-level Supervision for Weakly Supervised Semantic Segmentation. Ahn et al. CVPR 2018.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the valuable comments. Unless otherwise specified, all experiments are conducted on the *COD10K* dataset for the COD task to save space.\n\n**Q1: Literature review.**\n\nWe focused on concealed object segmentation and mainly discussed and compared existing methods in this field. However, we agree that it is better to have the literature review in a broader context and includes methods for weakly-supervised segmentation. While we have cited and discussed 8 existing methods related to weakly-supervised segmentation, we will add a subsection to fully review the weakly-supervised segmentation task in the related work of the revised version.\n\n**Q2: Novelties.**\n\nWS-SAM offers the first solution to address the WSCOS task that is orthogonal to existing methods by leveraging the power of SAM. We are inspired by the impressive capability of SAM on natural scene images and extend it to conceal object images by designing multiple effective techniques to address the limitations. Extensive experiments demonstrate that our WS-SAM is a general and robust framework that can be applicable to any segmenter and promote their performance in the COS task. More concrete differences are as follows. \n\n- Although our entropy-based strategy is similar to that in AffinityNet [52], we further propose the multi-augmentation fusion and image-level selection to improve the quality of the pseudo label, whose advancements are proved in Table 3 by correspondingly bringing **4.6\\%** and **4.9\\%** improvements.\n\n- Unlike AffinityNet [52] using augmentation for self-ensemble, our multi-augmentation fusion aims to fully explore the potential of SAM to balance the accuracy and completeness of pseudo labeling (line 138). To achieve this, we further fuse the segmentation masks for different augmented images and acquire a more reliable pseudo label, which promotes the performance of our method. \n\n- We replace our refinement strategies counterpart with those in AffinityNet [52] and validate the effect of our WS-SAM.\n\n**Q3: Contributions.**\n\nBoth WS-SAM and MFG are designed out of the characteristics of weak supervision and the COS task. They can jointly solve the weakly-supervised COS task. \n\n- WS-SAM: Given that SAM struggles in the COS task, directly using pseudo labels generated by SAM to supervise the segmenter is a sub-optimal solution. As shown in **Table A1** in the attained PDF, the segmenter trained in this manner performs poorer than that trained with the strategy of SCOD. To address this, we propose a series of refinement strategies, including multi-augmentation fusion, pixel-level weighting, and image-level selection, to fully explore the potential of SAM and thus provide reliable guidance for segmenter training, leading to improved performance. The effect of those strategies is proved in Table 3. Besides, our WS-SAM is a general framework applicable to various segmenters, whose superiority has been widely proven in **Tables A1, A5, A9**.\n\n- MFG: Given the intrinsic similarity problem in COS, we propose to address the COS task with feature grouping and introduce MFG to evacuate critical cues at different granularities, promoting feature coherence. The effect of our MFG has been proven in Table 4. We further demonstrate that our MFG is a plug-and-play module and can enhance the capability of existing segmenters in **Table A3**. Besides, as shown in **Tables A7, A8**, our segmenter equipped with the MFG is a lightweight network but achieves comparative performance with existing SOTAs in full supervision (**Tables 5, A6**).\n\n**Q4: Limitations.**\n\nWe have discussed our limitations, including our WS-SAM framework and MFG, in Sec. C of the Supp. We will include this discussion in the main text in the revised version.\n\n**Q5: The use of scribble and the formulation of SAM.**\n\n- As shown in **Fig. A2**, we use a nine-box strategy to sample critical points in the scribble, which can maximize the conveyance of the scribble information to SAM, resulting in more accurate pseudo-labels (line 231).\n\n- For point supervision, the formula about how SAM generates masks is defined in Eq. 1. For scribble supervision, the corresponding formulation is defined as:\n$$\\mathbf{M}_i^k=\\mathrm{SAM}(\\mathbf{X}^k_i,\\\\{\\mathbf{Y}\\_i^{f,c}\\\\}\\_{c=1}^C,\\\\{\\mathbf{Y}\\_i^{b,c}\\\\}\\_{c=1}^C),$$\nwhere $\\\\{\\mathbf{Y}\\_i^{f,c}\\\\}\\_{c=1}^C$, $\\\\{\\mathbf{Y}_i^{b,c}\\\\}\\_{c=1}^C$ denote prompts from foreground and background that are sampled from the scribble annotation $\\mathbf{Y}_i$. We will add this in the revised version.\n\n**Q6: Formulation in Sec 3.2 and implementation details.**\n\n- We double-checked the code and confirm that you are right. The transpose operator in Eq. 7 should be applied on $\\\\mathbf{Q}$, and the $\\\\mathbf{D}$ in Eq. 8 should be calculated from $\\\\bar{\\\\mathbf{A}}$. We will correct this in the revised version. \n\n- Following [31], We use the broadcast mechanism to broadcast the cluster prototype to a 2D grid, i.e., the prototype is copied horizontally and vertically several times to form a 2D grid. We will release our code.\n\n- $S$ in Eq. 10 means Sigmoid. We will clarify this in the revised version.\n\n**Q7: Visualizations.**\n\nWe explore how MFG facilitates complete segmentation and multiple-object segmentation. We visualize the similarity among group prototypes and grid features in **Fig. A1**. For complete segmentation, we observe that each single group can excavate local correlation within the individual object, helping complete segmentation through the synergy of multiple groups. Besides, we observe that every group can seek global coherence across multiple objects by aggregating those similar features, thus improving the accuracy in multiple-object segmentation. **Fig. A1** shows that clustering prototypes can learn to aggregate similar features and achieve more fine-grained aggregation at a higher level of granularity. Thus, MFG can extract critical cues by fusing the multi-scale grouping features.\n\n[52] AffinityNet, CVPR, 2018."}]

Author rebuttal:
We extend our sincere gratitude to all the reviewers (**R1**-76do, **R2**-yhjR, **R3**-ka5E, and **R4**-DmES) for their insightful and considerate reviews, which help us to emphasize the contributions of our approach. We are encouraged to hear that the reviewers found the work is well-motivated with good presentation (**R1**, **R3**, **R4**), as well as the comprehensive experimental evaluation and commendable performance (**R1**, **R2**, **R3**, **R4**).

We are delighted to see reviewers confirm our contributions to the field of concealed object segmentation. These encompass our novel weakly-supervised framework, WS-SAM, and the ingenious plug-and-play feature grouping module, MFG.

In direct response to your thoughtful comments, we have methodically addressed each point in our individual responses, and we provide a summary here:

- We compared our method with more SOTAs to underscore our superiority in performance and efficiency.

- We added experiments to verify the generalizability of our WS-SAM framework and the advancement of our plug-and-play MFG module.

- We introduced modifications to SAM to explore its potential and validate the efficacy of WS-SAM.

- We added diagrams and revised certain formulations to enhance clarity.

Thanks again for all of your valuable suggestions. We will update the paper accordingly and release our code for community study. We appreciate the reviewers' time to check our response and **hope to further discuss with the reviewers whether the concerns have been addressed or not**. If the reviewers still have any unclear parts about our work, please let us know.

Meta-review: The paper addresses Weakly-Supervised Concealed Object Segmentation by utilizing SAM mask predictions over augmentations and multi-scale feature grouping.  Both technical aspects have been thoroughly validated to be helpful in diverse settings upon reviewers' questioning and authors' extensive rebuttals.  All reviewers reach the consensus of accept.
Decision: Accept (poster)