WeakSAM: Segment Anything Meets Weakly-supervised Instance-level Recognition

Authors: gen  weaksam
reg  head
roi  drop
#  regularization
msk  head
cam  fine
cam  head
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-03 15:24:12

Abstract: Weakly-supervised visual recognition using inexact supervision is a critical yet challenging learning problem. It significantly reduces human labeling costs and traditionally relies on multi-instance learning and pseudo-labeling. This paper introduces WeakSAM and solves the weakly-supervised object detection (WSOD) and segmentation by utilizing the pre-learned world knowledge contained in a vision foundation model, i.e., the Segment Anything Model (SAM). WeakSAM addresses two critical limitations in traditional WSOD retraining, i.e., pseudo ground truth (PGT) incompleteness and noisy PGT instances, through adaptive PGT generation and Region of Interest (RoI) drop regularization. It also addresses the SAM's shortcomings of requiring human prompts and category unawareness in object detection and segmentation. Our results indicate that WeakSAM significantly surpasses previous state-of-the-art methods in WSOD and WSIS benchmarks with large margins, i.e. average improvements of 7.4% and 8.5%, respectively.

Main paper: 
 related work 2.1 segment anything model: 
 The recent Segment Anything Model (SAM) [34] draws great attention from researchers. The SAM is trained on SA-1B with over 1 billion masks, following the model-in-the-loop manner. Besides, SAM performs superior zero-shot transfer capabilities and is applied in many visual tasks, e.g., FGVP [78] incorporates SAM to achieve zero-shot fine-grained visual prompting, MedSAM [48] adapts SAM into a large scale medical dataset to build a medical foundation model, and some methods [7, 30, 62] utilize SAM to deal with the weakly-supervised semantic segmentation problem. However, SAM is an interactive segmentation method, which heavily relies on human prompts.In our approach, we innovatively propose to automatically prompt SAM using classification clues for extracting region proposals. This method results in high-recall proposals that surpass traditional methods like Selective Search in terms of both efficiency and effectiveness. This advancement represents a significant improvement in the domain of proposal generation within the WSOD framework.Weakly-supervised object detection (WSOD) with image-level labels [2, 3, 12, 17, 29, 35, 40, 45, 61, 66, 70, 71, 86] is important for reducing the human annotation burden. The previous works, i.e., WSDDN [4] and OICR [65] , proposed the Multiple Instance Learning and online refinement paradigms. The later works aimed to improve the WSOD performance from different perspectives. Such as WSOD 2 [81] introduced bottom-up object evidence, PCL [64] proposed to cluster proposals, MIST [53] utilized a self-training algorithm, etc. Besides, some methods [26, 31, 38, 60, 64, 88] also retrained a fully-supervised object detection network with generated pseudo ground truth (PGT). However, most of them used the proposals generated from low-level methods, i.e., Selective Search [69] , EdgeBox [95] , and MCG [50] , which contain a great number of redundant proposals and bring an optimization challenge.Different from previous methods, our WeakSAM proposals have fewer numbers and higher recall, which reduces the difficulty of finding the correct proposals for WSOD methods. For the key problem of PGT incompleteness and noisy PGT instances, we propose adaptive PGT generation and Region of Interest (RoI) drop regularization to address them, respectively.Weakly-supervised instance segmentation (WSIS) aims to achieve instance segmentation through weak supervision, such as box-level supervision [11, 24, 32, 37, 39, 42, 67, 72, 83, 93] , and image-level supervision [19, 23, 25, 28, 36, 46, 49, 84, 94] . The WSIS with imagelevel supervision is challenging because it lacks accurate instance locations. Some image-level WSIS methods use class activation map (CAM) [89] to extract coarse object locations, such as PRM [90] , IAM [94] , IRNet [1] , BESTIE [33] , etc. Some other image-level WSIS methods try to incorporate instance clues from extra priors, such as Fan et al. [15] , LIID [46] , CIM [41] , etc. However, they always need complicated networks and lack high-quality instance segments.Different from previous WSIS methods, the proposed WSIS extension using WeakSAM PGT and SAM's prediction is concise and effective. The generated pseudo instance labels can further be applied to any fully-supervised instance segmentation method. 
methods: 
 We present the WeakSAM framework as shown in Fig. 2 . At first, WeakSAM collects classification activations from a classification ViT. Subsequently, WeakSAM automatically generates prompts from classification activations and spatial samples. Next, WeakSAM sends the prompts to SAM for WeakSAM proposals. Then, we launch the weakly-supervised object detection (WSOD) pipeline, which is enhanced by WeakSAM proposals, adaptive pseudo ground truth (PGT) generation, and RoI drop regularization. Last, we use the SAM-enhanced pseudo instance labels to launch the weaklysupervised instance segmentation extension.Previous WSOD methods face an optimization problem caused by the redundant proposals, e.g., Selective Search [69] and Edge-Box [95] , because these proposals are only based on low-level features. To address this problem, we propose to transfer knowledge in the foundation model, i.e., SAM, for proposal generation. We use classification clues to prompt SAM automatically, which also solves the shortcoming of SAM requiring interactive prompts Classification Activation Generation. As shown in Fig. 2 , we extract classification clues from a classification ViT. Specifically, we choose the pre-trained weakly-supervised semantic segmentation network, WeakTr [92] , to provide classification clues because of its superior localization ability. At first, we extract cross-attention maps CA ∈ R 𝐾 ×𝐻 ×𝑁 ×𝑁 ×𝐶 from the self-attention maps, where 𝐾 is the number of transformer encoding layers, 𝐻 is the number of attention heads in each layer, 𝑁 × 𝑁 is the spatial size of the visual tokens, and 𝐶 represents the total number of classification categories. Then, we obtain coarse class activation map (CAM) [89] , CAM 𝑐𝑜𝑎𝑟𝑠𝑒 ∈ R 𝑁 ×𝑁 ×𝐶 , from the convolutional CAM head, which takes visual tokens at the final transformer layer as input and produces coarse CAM. Last, we use coarse CAM and weighted selfattention maps to produce fine CAM, CAM 𝑓 𝑖𝑛𝑒 ∈ R 𝑁 ×𝑁 ×𝐶 .WeakSAM Prompts Generation. As shown in Fig. 2 , we extract prompts from dense sampling points and activations, which include cross-attention maps, coarse CAM, and fine CAM. At first, the dense sampling requires splitting the image into 𝑆 × 𝑆 patches and taking the center points as prompts. Notably, the dense sampling points provide spatial-aware prompts but lack explicit reference to objects and semantics. Then, we get peak points from the crossattention maps as prompts. We observe that these maps do not solely concentrate on objects from their corresponding categories but also give attention to objects from different categories. So, we mark these prompts as instance-aware ones. Last, we extract peak points from coarse CAM and fine CAM as semantic-aware prompts, which are more precise and focus on areas of foreground objects. Specifically, we extract peak points from cross-attention maps and CAMs, as shown in Algorithm 1. Given cross-attention maps or CAMs as input, we first initialize the peak points list 𝑃, peak values list 𝑉 , deleted lists 𝑃 delete , 𝑉 delete , and max pooling operation. Next, we reshape the input maps and ensure the last two dimensions correspond to the original image size and the others as the first dimension. Then, we apply max pooling on the input maps 𝑀, and sort 𝑉 and 𝑃 in descending order based on 𝑉 . Last, we remove points with low activation values or close to high-score points.WeakSAM Proposals Generation. At the WeakSAM proposal generation stage, we use the three kinds of prompts to prompt SAM automatically. We directly add semantic-aware prompts and spatialware prompts to the prompt list, because they usually have clear localization to foreground objects and spatial positions, respectively. For the instance-aware prompts that have some redundancy, we cluster them to filter the duplicated ones and then add them to the prompt list. Finally, the prompt list is used to prompt SAM for WeakSAM proposals. Weakly-supervised Detector Training. A primary challenge in traditional WSOD methods is the low training efficiency, largely attributed to the redundancy of proposals. Traditional approaches often involve the Region of Interest pooling layer processing thousands of proposals per image, which impairs both effectiveness and efficiency. To address this issue, our WeakSAM proposals adopt transferred knowledge from SAM and classification clues. The proposed method focuses on generating a smaller quantity of proposals while maintaining high recall, thereby enhancing the overall efficiency and efficacy of the detection process in a WSOD context. We apply the proposed WeakSAM on previous WSOD methods, including OICR [65] and MIST [53] , which receive significant improvements. As shown in Table 1 , quantitative results show that WeakSAM-enhanced WSOD can annotate bounding boxes for objects more precisely.Adaptive PGT Generation. Generating high-quality pseudo ground truth (PGT) is the key to the WSOD paradigm. Traditional WSOD methods often encounter the issue of PGT incompleteness. This occurs because these methods typically select top-scoring proposals as PGT or apply a uniform threshold to filter proposals across all categories. Such approaches can lead to the omission of objects or entire categories, especially when proposals in certain categories score low. To address these problems, we propose an adaptive PGT generation method to normalize the score distribution of proposals, ensuring they fall within a similar range, as shown in Algorithm. 2.For box list 𝐵 ∈ R 𝑁 ×5 and corresponding score list 𝑆 ∈ R 𝑁 ×1 , we first select them with a specific classification label and then normalize the scores. The 𝑁 is the number of predicted boxes, and the second dimension of 𝐵 is the combination of a category label and four coordinate values. Next, we keep boxes with scores higher than the threshold 𝜏 𝑠 . Please note that the normalization enables the threshold to work for all categories adaptively, so we would not lose a ground truth category even if all boxes in this category have low scores. Then, we select the boxes whose main parts are not contained in some bigger boxes. Because the boxes that have more 𝑜𝑣𝑒𝑟𝑙𝑎𝑝 are often local components of some objects. Last, we return the box list 𝐵 ′ as the final PGT. end for 19: end for 20: return 𝐵 ′ RoI Drop Regularization. A recognized issue in the retraining phase of WSOD is noisy PGT instances. These noisy instances result in PGT acting as the inaccurate supervision. Alleviating this problem is critical for enhancing the performance of WSOD retraining. To analyze this problem in depth, we first divide the RoIs into different loss intervals. Then, we mark the RoIs whose corresponding PGTs do not have at least 70% IoU with the ground truth boxes as error ones. Last, we present the statistics as shown in Fig. 3 , which demonstrates that the RoIs with larger losses are in a small amount and have a high error rate.Intuitively, we propose a method, named RoI drop regularization, to adaptively drop the RoIs with larger losses. Notably, the proposed method is easy to implement and can further help the query-based detectors to alleviate the noisy PGT problem by its variant, query drop regularization. For anchor-based FSOD methods, e.g., Faster-RCNN [52] , we first determine the thresholds 𝜏 𝑐𝑙𝑠 and 𝜏 𝑟𝑒𝑔 for classification loss and regression loss, respectively. Then, we compute the drop signal 𝑑 𝑖 for 𝑖-th RoI. represent the classification loss and regression loss for each RoI, respectively. When the two losses of a RoI are all below their thresholds, we set its drop signal 𝑑 𝑖 as 1. Finally, we integrate the 𝑑 𝑖 into the computation of final loss L.𝑑 𝑖 = 1,EQUATIONwhere 𝑝 * 𝑖 is 1 if the box is positive, and 0 if the box is negative. The 𝜆 is a balancing weight.For query-based FSOD methods, e.g., DINO [82] , since queries can be regarded as dynamic RoIs, we apply query drop regularization on them. Because only a few matched queries need to calculate box loss 𝑙 𝑏𝑜𝑥 and IoU loss 𝑙 𝑖𝑜𝑢 , we only set a percentile threshold based on classification loss 𝑙 𝑐𝑙𝑠 . Only when the 𝑖-th query's loss 𝑙 𝑐𝑙𝑠 𝑖 is less than the loss at 𝜏% percentile, i.e., 𝑙 𝑐𝑙𝑠 𝜏 , will its corresponding 𝑑 𝑖 be set to 1.EQUATIONEQUATIONThanks to the high-quality WeakSAM PGT, we can directly use them to prompt SAM for precise segments as pseudo instance labels. Following the practices in the WeakSAM WSOD pipeline, we evaluate the quality of WeakSAM PGT using R-CNN-based and query-based instance segmentation methods, respectively. Notably, we do not introduce more techniques in the WeakSAM WSIS, because the WeakSAM pseudo instance labels are accurate enough. 
experiment 4.1 experimental setup: 
 Datasets and Metrics. We evaluate the proposed WeakSAM on both weakly-supervised object detection (WSOD) and weakly-supervised instance segmentation (WSIS) benchmarks. Notably, the same datasets for different tasks may have different settings.For WSOD, we use three datasets, i.e., PASCAL VOC 2007 [14] , PASCAL VOC 2012 [14] , and COCO 2014 [44] Implementation Details. For WeakSAM proposals generation, we adopt the WeakTr [92] with DeiT-S [68] model for generating classification clues, the SAM [34] with ViT-H [13] model to generate proposals. For WeakSAM WSOD pipeline, we use the WSOD networks, i.e., OICR [65] , and MIST [53] , with the VGG-16 [20] backbone to generate pseudo ground truth (PGT), and FSOD networks, i.e., Faster R-CNN [52] and DINO [82] , with the ResNet-50 [22] backbone to retrain. As for the WeakSAM WSIS, we use SAM-ViT-H to generate pseudo instance labels and train the R-CNN-based and query-based methods, i.e., Mask R-CNN [21] and Mask2former [10] , respectively. All hyper-parameters in Alg. 1 and Alg. 2 are following the default manners as Zhu et al. [92] and Sui et al. [60] .Weakly-supervised object detection. We present the quantitative WSOD results in Table . 1. Compared with our WSOD baseline methods, i.e., OICR and MIST, the proposed WeakSAM achieves over 10% improvements on all metrics. The results of WeakSAM (MIST) surpass all WSOD methods on all metrics, which demonstrate the effectiveness of WeakSAM proposals. Compared with WSOD methods retrained by pseudo ground truth (PGT), the WeakSAM (MIST) with Faster R-CNN retraining still outperforms the SoS-WSOD [60] and W2N [26] on all metrics, and the WeakSAM (MIST) with DINO retraining even has comparable performance with fully-supervised Faster R-CNN. The retraining results demonstrate the effectiveness of the proposed WSOD pipeline, which includes the adaptive PGT generation and RoI drop retraining. Compared with concurrent work, WSOVOD [43] , which also incorporates SAM, our WeakSAM (MIST) also achieves better performance.Weakly-supervised instance segmentation. We first present the quantitative WSIS results of the PASCAL VOC 2012 𝑣𝑎𝑙 set in Table 2 . The proposed WeakSAM with Mask R-CNN retraining achieves the best performance, which demonstrates the WeakSAM can benefit Table 1 : Comparisons of the WSOD performance in terms of AP metrics on three benchmarks: PASCAL VOC 2007, PASCAL VOC 2012, and COCO 2014. The 𝑆𝑢𝑝. column denotes the type of supervision used for training including full supervision (F ), point-level labels (P), image-level labels (I). "*" means the results rely on MCG [50] proposals. " ‡" means this method use the a heavy RN50-WS-MRRP [58] backbone (1.76 × parameters than VGG16 and 10.10 × parameters than RN50). We mark the best WSOD results in bold. WSIS effectively. Furthermore, the pseudo instance labels generated by WeakSAM can also be used by the modern query-based methods, e.g., Mask2Former [10] , which achieves the best results.We then show the quantitative WSIS results on COCO 2017 𝑣𝑎𝑙 and 𝑡𝑒𝑠𝑡 sets. On these more challenging benchmarks, WeakSAM with Mask R-CNN retraining achieves better results than CIM [41] . In this section, we present the ablation studies to evaluate the improvements brought by the proposed methods, i.e., WeakSAM prompts, adaptive PGT generation, and RoI drop retraining.Due to the limitation of pages, we leave more ablation studies in the supplementary material, including additional efficiency analysis, sensitivity analysis, qualitative analysis, discussions, etc. Improvements of WeakSAM Prompts. To further analyze the improvements brought by the proposed WeakSAM prompts, we conduct ablation experiments for different prompts as shown in Table 4 . Here, we use the Selective Search [69] as the baseline method and list the proposals' number, recall, and corresponding WSOD performance. When only using the densely sampled points as SAM prompts, the generated proposals can achieve 5.1% higher Recall (IoU=0.90), and 9.7% lower AP 50 for MIST. After adding peak CAM points and peak cross attention points as prompts, we can achieve 5 . Here, we follow the common practice to set a baseline that uses the predicted boxes with the top-1 score as PGT and plain Faster R-CNN as the retraining network. The results show that both adaptive PGT generation and RoI drop regularization can help improve the AP 50 of the detector. Furthermore, both the RoI-based detector, Faster R-CNN [52] and query-based detecotr, DINO [82] , can benefit from the proposed WSOD techniques.MIST WeakSAM Ground TruthTo further analyze the efficiency improvement brought by our WeakSAM, we present the efficiency comparison between Selective Search [69] and our WeakSAM on a machine with 4 GPU cards, as shown in Table 6 . Our WeakSAM reduces the number of proposals by 89.4%, the proposal generation time by 65.5%, the WSOD network training time by 43.8%, and the GPU memory cost by 68.2%. The results demonstrate the significant efficiency improvement brought by the proposed WeakSAM.Fig. 4 presents the object detection results using WeakSAM (MIST), showing its capability to accurately capture entire objects without generating excessive noisy bounding boxes. In Fig. 5 , the instance segmentation results of WeakSAM Mask2Former retraining are showcased. The results indicate effective segmentation of entire instances with a notable reduction in overlapping segments. 
conclusion: 
 In this paper, we introduce WeakSAM, a novel framework utilizing the Segment Anything Model (SAM) for weakly-supervised instance-level recognition, demonstrating leading performance in WSOD and WSIS benchmarks. Different from the original SAM, which requires interaction and can not be aware of categories, WeakSAM represents an innovative fusion of SAM with weaklysupervised learning (WSL), overcoming the redundancy problem of WSOD proposals. To further address WSOD issues such as pseudo ground truth (PGT) incompleteness and noisy PGT instances, our approach includes adaptive PGT generation and a Region of Interest (RoI) drop regularization. The adaptability of WeakSAM is further showcased through its extension to weakly-supervised instance segmentation (WSIS). Our work aims to inspire further research with SAM and WSL, contributing significantly to the development of a universal framework for weakly-supervised recognition.  
: 
 lack of external knowledge, which restricts the performance of WSL and hinders it from matching fully-supervised learning (FSL). Nowadays, foundation models are gaining increasing attention because of their transferable pre-learned world knowledge, which can be regarded as powerful external knowledge for WSL. As a vision foundation model, SAM [34] achieves outstanding performance in interactive, class-agnostic segmentation. SAM owes its success to promptable training on a large-scale dataset. However, there are two main drawbacks to SAM: First, SAM requires interactive operations as input, which means it cannot work automatically without human prompts. Second, SAM produces class-agnostic segments and cannot assign class labels. These drawbacks severely restrict the application of SAM as a direct and generic visual framework. As a strong complement, WSL is good at mining classification clues through inexact supervision, which can provide automatic prompts for SAM. Subsequently, WSL with SAM's knowledge can further bring class-aware perception.This motivates us to assimilate SAM within the WSL paradigm. The WeakSAM framework is designed to harness transferable knowledge from SAM, thereby enriching the WSL process. Simultaneously, it offers the capability to deliver automatic classification clues to SAM. This bidirectional enhancement constructs a promising foundation-model-based weakly-supervised visual perception framework. Specifically, in a weakly-supervised object detection (WSOD) setting, WeakSAM uses classification clues as SAM prompts to produce proposals automatically. These proposals are then used in WSOD training for class-aware perception.Within the scope of the WeakSAM framework, our analysis identifies two prevailing limitations in the iterative WSOD retraining approach: the issue of pseudo ground truth (PGT) incompleteness and the presence of noisy PGT instances. The former, PGT incompleteness, refers to the tendency of WSOD-generated PGT to omit some objects or categories, leading to insufficient training for these categories. The latter, noisy PGT instances, pertain to the prevalent presence of noise within the PGT, which adversely impacts the retraining process. To effectively mitigate these challenges, we introduce two key strategies: adaptive PGT generation to address the PGT incompleteness problem, and Region of Interest (RoI) drop regularization to counteract the noise in PGT instances. Moreover, WeakSAM's capability enables the extension in the realm of weaklysupervised instance segmentation (WSIS). In this context, SAM is employed to further refine WeakSAM-PGT, enabling the generation of pseudo instance segmentation labels. This approach exemplifies WeakSAM is promising to build a unified weakly-supervised instance-level recognition framework.The main contributions of this paper can be summarized as follows:• We propose a weakly-supervised instance-level recognition framework (WeakSAM), which automatically prompts SAM by classification clues for proposals. The WeakSAM proposals reduce the generation time by 65.5% and improve the recall (IoU=0.9) by 22.9%, compared to Selective Search [69] .• We analyze the weaknesses in traditional WSOD retraining, and propose adaptive PGT generation and RoI drop regularization to address them, respectively. After the WeakSAM-WSOD is complete, the proposed WeakSAM can be easily applied to WSIS further. • The proposed WeakSAM achieves state-of-the-art (SOTA) results on the WSOD and WSIS benchmarks, significantly surpassing previous SOTA methods as shown in Fig. 1 . 
 
 Appendix:  


Decision: Initially, the paper received one Weak Reject, one Borderline Reject, one borderline Accept, and one Weak Accept. The reviewers praised the proposed approach's effectiveness but raised significant concerns regarding the claim of weak supervision and the fairness of comparisons.

The rebuttal addressed these points by providing additional experimental results and clarifications. After the rebuttal, Reviewer DBpV upgraded the score from Weak Reject to Borderline Accept, confirming the method's novelty of combining vision foundation models for WSOD. Reviewer YbAk maintained Weak Accept for the good performance and interesting motivation. Reviewer Urx4 decreased the score and expressed concerns about the fairness and reasonability of the WSOD setting.

The AC has read the paper, reviews, and rebuttal. The AC agrees with Reviewer Urx4 that exploiting large-scale pretrained models like SAM can be unfair in comparison with traditional methods. However, the AC believes that this limitation may not overturn the overall rating of this paper, considering its good performance and novel pipeline for addressing WSOD with foundation models.

The authors should carefully improve the final paper by following reviewer recommendations, particularly as suggested by Reviewer Urx4.

Reviews: review: paper_summary: paper_summary: This paper introduces WeakSAM and leverages the pre-learned world knowledge contained in a vision foundation model SAM to solve the weakly supervised object detection (WSOD) and segmentation problems.
strength_weakness: strength_weakness: The WeakSAM framework proposed in this paper integrates the Segment Anything Model (SAM) into weakly supervised learning (WSL), improving the performance of weakly supervised object detection (WSOD) and instance segmentation (WSIS).

The experimental results show that WeakSAM performs significantly better than previous state-of-the-art methods in multiple benchmarks, demonstrating its effectiveness.
limitations: limitations: (1) Regarding "weak supervision": Let's briefly forget about SAM. If a class-agnostic proposal generator (e.g., RPN) is trained using a large amount of fully labeled data and then applied to the WSOD pipeline, can this still be considered weak supervision?

(2) It seems necessary to separate detection into classification and localization. The classification task can be viewed as weakly supervised, while localization benefits significantly from SAM, which is fully supervised. Although the paper title indicates weakly supervised recognition, this seems blurred in the experimental section. In other words, the paper should clearly separate classification and localization and demonstrate how much gain the fully supervised localization capability brings.

(3) The paper conducts many comparisons with SS, which may not be very necessary. As mentioned in (2), the gains are primarily due to SAM. Therefore, the paper should focus on how to effectively utilize SAM, for example, by exploring various ways to trigger SAM to generate higher-quality proposals.
paper_summary: paper_summary: This paper introduces the SAM model into weakly supervised object detection and designs two modules: adaptive PGT generation and RoI drop regularization. I think the major novelty of this paper appears to be a unified innovation, i.e., putting together many existing techniques into a unified framework. The novelty of the two newly proposed modules is limited.
strength_weakness: strength_weakness: Strong results.
limitations: limitations: 1. The proposed framework seems to be a unified innovation. The novelty of adaptive PGT generation and RoI drop regularization modules is limited.
2. In adaptive PGT generation, the authors normalize candidate proposals of each category to enhance system performance. I wonder whether this strategy is also effective in previous WSOD frameworks, i.e., without using the proposals generated by SAM. I am not entirely convinced of the effectiveness of this module, and I consider that the major efficacy is attributed to the object localization capability of SAM.
3. More visualization results should be provided without using the two proposed modules, which can demonstrate the effectiveness of the designing modules.
paper_summary: paper_summary: This paper proposes WeakSAM and solves the weakly-supervised object detection (WSOD) and segmentation by utilizing the pre-learned world knowledge contained in a vision foundation model, i.e., the Segment Anything Model. The proposed method can address the issues of pseudo ground truth (PGT) incompleteness and the presence of noisy PGT instances by introducing adaptive PGT generation and region of Interest (RoI) drop regularization, respectively.
strength_weakness: strength_weakness: 1. The paper is well-written and is easy to follow.
2. The motivation of this paper is that assimilating SAM within the WSL paradigm is interesting.
3. The SOTA experimental results show the effectiveness of the proposed WeakSAM.
limitations: limitations: It is recommended to try the WSOD model in recent years as a baseline.
paper_summary: paper_summary: This paper proposed WeakSAM, it solves the weakly-supervised object detection (WSOD) and segmentation by utilizing the pre-learned world knowledge contained in a vision foundation model. It addresses the SAM's shortcomings of requiring human prompts and category unawareness in object detection and segmentation. Quantitive experiments demonstrate the effectiveness of the model.
strength_weakness: strength_weakness: 1. The proposed method can exceed previous methods by a large margin.
2. The paper is well-written and is easy to read.
limitations: limitations: SAM is well-trained using a lot of labeled/unlabeled data.  I am wondering if it is fair to compare the previous method, using such a powerful auxiliary model to generate pseudo-labels. What about the performance of the other models if using the SAM model. If not using the powerful SAM model, can the model perform well?