CBNet: Cooperation-Based Weakly Supervised Polyp Detection

Authors: xiuquan  du
jiajia  chen
xuejun  zhang
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-04 15:49:59

Abstract: Missed polyps are the major risk factor for colorectal cancer. To minimize misdiagnosis, many methods have been developed. However,
they either rely on laborious instance-level annotations, require labeling of prompt points, or lack the ability to filter noise proposals
and detect polyps integrally, resulting in severe challenges in this area. In this paper, we propose a novel Cooperation-Based network
(CBNet), a two-stage polyp detection framework supervised by image labels that removes wrong proposals through classification
in collaboration with segmentation and obtains a more accurate detector by aggregating adaptive multi-level regional features. Specifically, we conduct a Cooperation-Based Region Proposal Network (CBRPN) to reduce the negative impact of noises by deleting proposals without polyps, enabling our network to capture polyp features. Moreover, to enhance location integrity and classification precision of polyps, we aggregate multi-level region of interest (ROI) features under the guidance of the backbone classification layer, namely Adaptive ROI Fusion Module (ARFM). Extensive experiments on the public and private datasets show that our method achieves stateof-the-art performance for weakly supervised methods and even outperforms full supervision in some terms. All code is available at https://github.com/dxqllp/CBNet.

Main paper: 
 introduction: 
 Colorectal cancer (CRC) ranks as the third most frequent cancer, with more than 80% originating from polyps [31] . However, untreated polyps might become malignant and life-threatening cancer [22] . In clinical practice, the colonoscopy procedure is regarded as the golden standard for detecting and removing polyps. Unfortunately, the method highly depends on the experience and proficiency of the medical practitioner that suffers a high miss rate (as much as 30%) [37] . Fortunately, computer-aided detection (CAD) has been proven to help doctors detect polyps and reduce the incidence of missed diagnoses [20] . Although CAD has demonstrated Figure 1 : The schematic diagram of the previous works with separable mechanism and our proposed collaboration approach. And four extremely challenging scenarios, i.e., missed flat polyp, non-entire polyp, low-quality proposals with unfilterable noise and wrong polyp mask with point annotations existed in polyp detection.impressive capabilities in polyp detection [4, 15, 19, 24, 26, 33, 36] , they need to be combined with instance-level bounding box annotations, which are very laborious and challenging owing to the complexity of colonic images and the diversity of polyps. To reduce the labeling burden, researchers hope to make detectors in a weakly supervised (WS) fashion. For example, several promising works combining multiple instance learning (MIL) with deep learning [21, 25, 39] have greatly pushed the boundaries of natural images to successfully apply WS to the medical field. Unfortunately, accurate and reliable polyp detection can be easily fooled by flat polyps because they lack clearly visible borders and exhibit a similar aspect to the surrounding colorectum, which makes the network struggle to accurately distinguish polyps from backgrounds. In addition, most of these studies directly use coarse proposals generated by standard methods for training, yet most of them are negative cases (noise) that do not contain the target, which affects the learning of the target features while lowering the performance of the detection.Worse still, they severely rely on region-level classifiers that overly focus on the most discriminatory local regions (over-fitting), which not only reduces classification accuracy but also results in over-fitting, as shown in Figure 1 (a) left. More recently, some highprofile works hope to solve this task through transfer learning, such as weakly supervised polyp segmentation (WSPS) with the help of the Segment Anything Model (SAM). But it is essentially a promotable method, which needs point annotations and will lose the ability to accurately identify polyps if there are no additional point annotations as shown in Figure 1 (a) right. Additional points not only increase the burden of annotation but also deviate from the requirements of only image labels. Therefore, a further mechanism is necessary for a more reasonable inference.Through deeper study, we find that the MIL-based and SAMaided approaches have complementary advantages and disadvantages. For instance, the classifiers among MIL tend to focus on parts of objects that match the need for prompts in SAM, because it can automatically generate point prompts based on the focused region, and eliminate the burden for additional point annotations. Meanwhile, the high-quality polyp masks produced from point prompts can not only ensure the integrity of the target to avoid over-fitting but also filter the noise in the proposal so that the network can better learn polyp features and gain inspiring performance. In a word, instead of working separately, they should cooperate with each other to overcome their internal weaknesses.As a result, in this work, we innovatively come up with a new solution, namely CBNet, as shown in Figure 1 (b), a weakly supervised polyp detection model is designed to solve the above challenges: (1) over-fitting polyp detection. (2) unfilterable noise. (3) failed polyp mask without point annotation. (4) missed flat polyps. Our approach mainly includes two innovative modules: Cooperation-Based Region Proposal Network (CBRPN), Adaptive ROI Fusion Module (ARFM), and a common module: Multiple Instance Detection Network (MIDN). Specifically, inspired by the complementary strengths and weaknesses, we design CBRPN mainly for challenges (2) -(3) while initially addressing challenge (1) , which includes a proposal generator and SAM. The former generates a series of rough proposals by graph-based segmentation and merging strategies. The latter generates pseudo masks based on point prompts (derived from the gradient matrix of the backbone). After that masks and proposals are computed intersection over union (IOU) one by one to leave proposals that contain relatively full polyps and remove the noise without polyps. Furthermore, we employ ARFM to solve challenge (4) and further address challenge (1), which consists of two parts: classification layer and ROI fusion. During the training of the module, the parameters of the classification layer are frozen to ensure that the backbone network can focus on polyps from the whole image for further completeness, and judge the morphology of them from the global features to improve classification accuracy. In the fusion part, the differences between polyps and background will be refined by combining deep-shallow region features to improve the detection of flat polyps. This module adaptively adjusts features learned in the second stage, resulting in a more comprehensive global-local feature. Finally, we introduce MIDN for setting the target existence and category possibility scores of the proposal region to achieve weakly supervised detection under the supervision of image-level labels.In summary, our main contributions are listed four-fold:• We give a new solution under the weakly supervised, CBNet, a framework that employs the collaborative mechanism to detect polyps with only image-level annotation.  
weakly supervised cooperation detection: 
 Due to the absence of instance-level annotations, weakly supervised detection methods are easy to over-fit on object parts. To address this issue, many cooperation-based works such as C-MIDN [12] , P-MIDN [38] , WS-JDS [30] and NDI-WSOD [35] , have been proposed. Despite achieving promising results, over-fitting is still a generate image feature maps: 𝐺𝐴𝑃 (𝐶𝑜𝑛𝑣_𝑃 (𝐹 5 )).generate scores:𝑆𝑜 𝑓 𝑡𝑚𝑎𝑥 (𝑋 𝑖𝑚𝑔 ) → 𝑆 𝑖𝑚𝑔 , 𝑆𝑜 𝑓 𝑡𝑚𝑎𝑥 𝐶 (𝑋 𝑖𝑚𝑔 ) → 𝑆 𝑀𝐼 𝐶 , 𝑆𝑜 𝑓 𝑡𝑚𝑎𝑥 𝑅 (𝑋 𝑖𝑚𝑔 ) → 𝑆 𝑀𝐼 𝑅 . 6:combine region score to image scores according to Eq. 10.7:compute and backward 𝐿 𝑖𝑚𝑔 + 𝐿 𝑀𝐼 in Eq. 12 for CBNet.continue until convergence Output: 2: the optimized CBNet for polyp detection. challenging task because segmentation networks in collaboration are explicitly tailored for the specific domain and their performance can degrade significantly when applied to different types of imaging data. Recently, SAM attracted a lot of attention and was introduced into the medical area [17, 19, 41] , owing to the excellent ability of generality. However, it is still unexplored to employ SAM in weakly supervised polyp segmentation only with image annotations. Towards the same goal, we propose the novel CBRPN reduces the possibility of over-fitting and successfully employs SAM for imagesupervised WSPS. Meanwhile, ARFM is also designed to further avoid over-fitting while better detecting flat polyps. Both of them have less complexity and particularity while performing better. 
method 3.1 overview: 
 With the aim of high-quality proposals under image annotations and more accurate detection, we designed CBNet. The overall architecture of our network is shown in Figure 2 . The proposed CBNet consists of two stages during training and testing, where the first stage is the region proposal network (CBRPN, see Section 3.3) for proposal generation and the second stage is the WS network (ARFM, see Section 3.4 & MIDN, see Section 3.5) of polyp detection. For more clarity, the training and the testing of our CBNet are summarized in Algorithm 1.First Stage: Given input image I ∈ R 𝐶 ×𝐻 ×𝑊 (C: channels, H: height, W: width), it is simultaneously fed into SSW (Selective Search Windows), pre-trained backbone (see Section 3.1) and SAM for generating a coarse set of window boxes 𝐵 𝑆𝑆𝑊 , obtaining the most significant point in the polyp, and producing the other box set 𝐵 𝑆𝐴𝑀 (according to the pseudo-mask) for filtering, respectively. After that, 𝐵 𝑆𝑆𝑊 and 𝐵 𝑆𝐴𝑀 will be sent to iou threshold filter (ITF) to remove noise, and the retained proposals form 𝐵 𝐶𝐵𝑅𝑃 𝑁 .Second Stage: Features 𝐹 1 -𝐹 5 from the backbone and refined proposals 𝐵 𝐶𝐵𝑅𝑃 𝑁 from the first stage are processed by ARFM, which contains three cross-layer aggregation blocks stacked sequentially and the following fusion operation to get 𝑋 𝑓 𝑢𝑠𝑖𝑜𝑛 . In addition, a convolution layer with fixed parameters 𝐶𝑜𝑛𝑣_𝑃 and the global average pooling are added in parallel for 𝑋 𝑖𝑚𝑔 to enable the accuracy of classification. Finally, the features 𝑋 𝑓 𝑢𝑠𝑖𝑜𝑛 and 𝑋 𝑖𝑚𝑔 further go through the MIDN and a common softmax to set proposal scores and obtain the detection results through non-maximum suppression.As CNN can gradually extract features at different levels in the image (e.g. from edges and textures to semantic features of objects) by stacking convolutional and pooling layers, and pre-trained weights can speed up the learning of the network. Therefore, we built our method on the pre-trained CNN that has been trained on the ImageNet [23] , and fine-tuned it on polyp data with only imagelevel supervision (i.e. no bounding box annotations) to get features 𝐹 = {𝐹 1 , 𝐹 2 , ..., 𝐹 𝑁 } (where N is the number of feature blocks and N=5 in this study) and 𝐶𝑜𝑛𝑣_𝑃. We will give details of usage (𝐹 & 𝐶𝑜𝑛𝑣_𝑃) in other Sections.Since the purpose consistency between weakly supervised polyp detection and multi-instance learning, it is usually treated as MIL that requires previous proposals for the training data. Superpixel-based SSW is commonly used to generate the initial candidate boxes due to their ability to adapt to targets with blurred edges and does not require any annotation. However, they are rough because many of them have only the background or contain only a small percentage of polyps. Fortunately, SAM can generate accurate pseudo-masks for polyps to optimize the proposal but requires additional point prompts. In the classification task, pixels will respond according to their relevance with the target class, so there must be a pixel with the highest response in the object, and the coordinate of this point can exactly be used as a prompt for SAM. Therefore, we design the CBRPN to take advantage of both SSW and SAM to filter background noise as well as reduce incomplete polyp proposals before training to reduce the over-fitting risk.In detail, the CBRPN takes the image I and feature 𝐹 1 -𝐹 5 as input. 𝐼 is used to generate an initial set proposals 𝐵 𝑆𝑆𝑊 ∈ R 𝐵 1 ×4 (𝐵 1 : the number of proposals from SSW, 4: coordinates of each proposal for top, left, right and bottom) through SSW, and 𝐹 1 -𝐹 5 are used to produce the coordinate (𝑥 𝑝𝑜𝑖𝑛𝑡 , 𝑦 𝑝𝑜𝑖𝑛𝑡 ) for point according to the predicted category, formulated as below:EQUATIONwhere 𝑦_𝑝𝑜𝑖𝑛𝑡&𝑥_𝑝𝑜𝑖𝑛𝑡 ∈ R, 𝐹 𝑘 & 𝜔 𝑐 𝑘 denotes the 𝑘 𝑡ℎ feature maps and their weights corresponding to class c, 𝑎𝑟𝑔𝑚𝑎𝑥 (•) is the operation to find the max pixel, 𝑓 𝑢_𝑖 𝑖 is the unrval_index function to find the coordinate of the max pixel. 𝜔 𝑐 𝑘 is defined as:EQUATIONwhere Z represents the number of pixels in the feature map, 𝜕 is the derivation operation. 𝑦 𝑐 and 𝐹 𝑘 𝑖 𝑗 are the gradient of the 𝑐 𝑡ℎ score and the pixel value of the 𝐾 𝑡ℎ feature map at coordinates (i, j), respectively.Since (x_point, y_point) can be regarded as the input prompt of the SAM, thereby another set of candidate boxes for filtering can be defined as:EQUATIONwhere 𝐵 𝑆𝐴𝑀 ∈ R 𝐵 2 ×4 (𝐵 2 :number of proposals from SAM, 4:coordinates of each proposal for top, left, right and bottom). 𝑓 𝑆𝐴𝑀 is the segment anything model with pre-training weights loaded. Thus, the significant noise can be filtered by ITF. The processes and rules can be formulated as:EQUATIONwhere 𝐵 𝐶𝐵𝑅𝑃 𝑁 ∈ R 𝐵×4 (B: number of proposals after filtering, 4: coordinates of each proposal for top, left, right, and bottom), 𝑚 𝑖 & 𝑤 𝑗 are the proposals from 𝐵 𝑆𝐴𝑀 and 𝐵 𝑆𝑆𝑊 , respectively. ∩ is the intersection of two sets, |𝑚 𝑖 ∩ 𝑤 𝑗 |, |𝑚 𝑖 | and |𝑤 𝑗 | represent the number of pixels in 𝑚 𝑖 ∩ 𝑤 𝑗 , 𝑚 𝑖 and 𝑤 𝑗 , respectively. 𝜏 is a threshold for the judgement of trade-offs, and 𝜅 denotes the stack of proposals. Since the classification network may learn data bias [11] during training, the prompt point may appear in the background, which causes 𝐵 𝐶𝐵𝑅𝑃 𝑁 to be empty. When this occurs, Eq. 4 can be re-written as 𝐵 𝐶𝐵𝑅𝑃 𝑁 = 𝐵 𝑆𝑆𝑊 .By calculating the IOU between 𝐵 𝑆𝑆𝑊 and 𝐵 𝑆𝐴𝑀 and then removing proposals with low overlap through the ITF, the 𝐵 𝐶𝐵𝑅𝑃 𝑁 can suppress noise without polyps while strengthening the target learning by reducing false positives proposals.The ROI feature in the existing WS network typically is directly generated by the last convolution layer and then sent to region softmax to calculate the probability score. However, the features of this layer are limited and inaccurate, shown in Figure 3 (a), 2 𝑡ℎ row Conv_5, which is inappropriate for medical images, e.g., the polyp images with a typical resolution of 224 × 224 in the morphological classification task. In addition, the flat polyp is usually small, and constant down-sampling will drown it in the background, and it is often ignored due to the unclear boundary. To address the above problem, we develop an adaptive ROI fusion module (ARFM) inspired by the path aggregation mechanism (Refine-FPN) [18] . Specifically, ARFM has an agg and a fusion structure with detailed configuration inside as shown in Figure 3 (b), (c ). We use {𝐹 2 , 𝐹 3 , 𝐹 4 , 𝐹 5 } to denote feature levels generated by the backbone. The aggregated mechanism starts from the lowest level 𝐹 2 and gradually approaches 𝐹 5 . From 𝐹 2 to 𝐹 5 , the spatial size is gradually being halved. We use { F2 , F3 , F4 , F5 } to denote newly produced feature maps corresponding to {𝐹 2 , 𝐹 3 , 𝐹 4 , 𝐹 5 }. Note that F2 = 𝐹 2 , without any processing. The calculation process is as follows:EQUATIONwhere F𝑖 ∈ R Further, we pool features from all levels for proposals to the same size 7 × 7 as:EQUATIONwhere R𝑖 ∈ R 𝑏 ×128×7×7 ( b: batch_size ), 𝑅 𝑖 is the 𝑖 𝑡ℎ region feature cropped according to the proposal from 𝐵 𝐶𝐵𝑅𝑃 𝑁 , roi_align is an adaptive pooling operation to adjust 𝑅 𝑖 to a uniform size 7 × 7. Then, they are fused for the following prediction. Considering the influence of position information on subsequent modules, we design different fusion strategies: cnn-based and vector-based, detailed formulas are described as:EQUATIONwhere 𝑋, 𝑋∈ R 𝐵×𝐶 (B: the number of proposals from 𝐵 𝐶𝐵𝑅𝑃 𝑁 , C: the number of categories), GAP is the global average pooling (GAP), FC is the fully connected layer, 𝑐𝑜𝑛𝑣 𝑘 𝑠 follow the same definition as Eq. 6.Since ROI features only contain the local region of the image, this reduces classification accuracy and leads to over-fitting. To overcome this problem, we add the specific classification layer from the backbone to this module in parallel for global image feature 𝑋 𝑖𝑚𝑔 , which is calculated according to:EQUATIONwhere 𝑋 𝑖𝑚𝑔 ∈ R 𝐶 (C is the number of morphological categories), 𝐶𝑜𝑛𝑣_𝑃 denotes the convolution that loads and freezes the pretrained weights.Current WS object detection methods usually choose WSDDN as the criterion, which solves the problem that maps proposal scores at the instance level to image labels at the image level. Following the same double branch, we build a multiple instance detection network (MIDN) to set scores for proposals. Specifically, MIDN consists of a location branch and a classification branch, the former selects which proposal region is more likely to contain the entire polyp fragment while the latter predicts which class to associate with the proposal region. Hence, the different scores of proposals can be represented:𝑆 𝑀𝐼 𝐶 = 𝑆𝑜 𝑓 𝑡𝑚𝑎𝑥 𝐶 (𝑋, 𝑑𝑖𝑚 = 1)EQUATIONwhere 𝑆 𝑀𝐼 𝐶 , 𝑆 𝑀𝐼 𝑅 ∈ [0, 1] 𝐵×𝐶 , 𝑆𝑜 𝑓 𝑡𝑚𝑎𝑥 𝐶 and 𝑆𝑜 𝑓 𝑡𝑚𝑎𝑥 𝑅 are both softmax operation that is responsible for mapping the feature matrix to the category dimension (dim=1) and the proposal dimension (dim=0), respectively. Finally, these score vectors are element-wise producted and added to obtain the image-level classification scores. The rule can be formulated as:EQUATION)where𝑆 𝑀𝐼 𝐶•𝑅 ∈ [0, 1] 𝐵×𝐶 , 𝑆 𝑀𝐼 ∈ [0, 1] 𝐶 .Additionally, the feature 𝑋 𝑖𝑚𝑔 of the whole image also be fed to a softmax to get the imagelevel 𝑆 𝑖𝑚𝑔 ∈ [0, 1] 𝐶 :EQUATION3.6 Loss Function BCE Loss: Binary Cross Entropy Loss is a measure used to evaluate the distance between the prediction and image label. We employ it as the loss function to train our network:EQUATIONwhere N indicates the number of predicted object groups, 𝑦 𝑖 is the one-hot label of 𝑖 𝑡ℎ category, 𝑝 (𝑆 𝑖𝑚𝑔 𝑖 ) and 𝑝 (𝑆 𝑀𝐼 𝑖 ) are the probabilities belong to 𝑖 𝑡ℎ class predicted by model. 
experiments 4.1 datasets: 
 CVC-ClinicDB1 [3] : The dataset comprises 612 images sourced from 29 colonoscopy video sequences, each with a resolution of 288 × 384. It was developed in partnership with the Hospital Clinic of Barcelona, Spain, and consists of 322 flat, 282 pedicle, and 66 edge polyps.Kvasir 2 [9]: This dataset was assembled by Vestre Viken Health Trust in Norway which includes 1,000 polyp images along with their ground truth annotations from colonoscopy videos. The bounding boxes for ground truth were initially marked by medical doctors and later confirmed by experienced gastroenterologists. The images span a range of resolutions from 332 × 487 to 1920 × 1072 pixels. The number of three morphologies is 255, 851, and 53, respectively.Private: Our internal dataset extracted 290 static images from OlympiusEurope colonoscopy videos at a hospital, which consists of 177 patients, annotated and validated by experienced endoscopists. We rejected all the images with extremely high patient preparation or bad visualization quality due to image blurring and each image in the dataset is paired with a morphological image label (87 flat, 158 pedicle, 45 edge) to guide the training. The images vary in resolution spanning from 564×480 to 600×530 pixels. To assess the location of polyps, experts also provided ground truth (bounding boxes) for all images. In addition, to keep the dataset up-todate we are still continuously collecting updated data.Training Details. The slightly modified classifier layer of VGG16 serves as our backbone. Specifically, we replace the classifier of three FC operations with a GAP layer and reshape after the original last feature layer. In the first stage, we only train the backbone for a total of 25 epochs with a learning rate of 1e-4 and a batch size of 32. And the trained weight sam_vit_b_01ec64 3 on nature images is used as the parameter of SAM. In the second stage, we train the CBNet for a weight decay of 5e-4, a batch size of 1, and 15 epochs with a learning rate 1e-5 following 5 epochs with 1e-6. As discussed in Eq 4, we empirically set 𝜏 as 0.5. We divided data into two splits on each dataset: training, and test. The training split comprises about 80% of data; the test about 20% each.Evaluation Metric. To evaluate detection performance, we employ three performance measures. The first one follows the standard PASCAL VOC protocol, calculating average precision (AP) and mean AP (mAP) at IOU thresholds of 10%, 30%, 50% between the detected boxes and the ground truth ones. Additionally, we report CorLoc for location, a commonly used weakly supervised detection measure [10] , which means the percentage of images where at least one instance of the target object class is correctly localized with the most confident detected bounding box overlapping at least 50%. Detailed explanations of all evaluation indicators can be found in the cited references.Compared Methods: Some well-performed detection methods are selected for comparison. They are categorized into fully and weakly supervision (Fully sup. and Weakly sup.). methods. In the fully sup. methods, we choose Faster 3 https://dl.fbaipublicfiles.com/segment_anything/ Rcnn [28] , Yolo [27] and Diffusion [6] with dynamic boxes 50 & 500. In the weak sup. methods, we choose WSDDN [13] , OICR [32] , WSOD2 [40] , Grad-CAM [29] , Grad-CAM++ [2] , IDC [34] and LPCAM [7] . Unfortunately, the last two methods don't work very well, so we only list the results of other methods. The results of Faster RCNN, Yolo, Diffusion-Det are given from mmdetection 4 [5], WSOD2 are referred from codes 5 and the rest are implemented by ourselves with source codes.Main Results: For a clearer comparison, we represent the best indicator of CBNet as red and the best indicator of weak supervision as blue. As shown in Table 1 , traditional weakly supervised methods have a limited ability to deal with challenging polyps, thereby having much worse scores than fully supervised methods and ours. For example, the best mAP of weakly sup. is only 0.3 (IOU@10-50) on CVC-ClinicDB but full supervision can reach 0.6. Even better, our method reaches 0.64, which not only exceeds weak supervision or even due to full supervision. In comparison with CVC-ClinicDB and private dataset, the best mAP (IOU@10) on Kvasir is only 0.22, which is 55% lower than CVC-ClinicDB, and 30% lower than private. Besides, we also find that although the full-supervision methods have the high CorLoc, mAP does not have the same advantages, e.g. 1.0 CorLoc v.s. 0.39 mAP (IOU@30) on the private of DiffusionDet, which indicates that they are inefficient in learning discriminative category representation, leading to false detection or missed detection. Compared with them, our proposed CBNet offers higher mAP despite having slightly low CorLoc due to the lack of instance-level annotations, which shows that CB-Net can learn a better target representation to deal with flat polyps and background noise issues. More detail for each category can be seen in supplementary material.Precision-Recall Curves: We also plot the Precision-Recall (P-R) curves of different methods on the CVC-ClinicDB dataset and private dataset for flat polyps in Figure 4 . As can be seen, the performance of our CBNet is significantly better than all other methods, where the area under the P-R In Figure 5 , we present some visual detection results of different methods. As we can see, the 9 𝑡ℎ column test image contains a flat polyp, all the methods including both fully supervised and weakly methods falsely detect or miss the polyp as a target except our CBNet, which shows that our CBNet has more excellent representation ability in complex flat polyps. We attribute this success to the unique structure of our ARFM, i.e., it aggregates different level features such that it has the ability to provide more information for following modules and recognize polyp as target rather than background. Moreover, the last two rows show that overall our method has better performance, and different proposal filtering strategies will make a negative or positive impact on the results.To gain more in-depth in-sight into what backbone has learned, we also visualize the gradient-weighted class activation mapping (Grad-CAM), as in Figure 6 . It can be observed that baseline has the ability to successfully locate polyps but only focus on a small part of the target e.g., 1 𝑡ℎ row & 2 𝑡ℎ column, and may occasionally fail e.g., 1 𝑡ℎ row & 1 𝑡ℎ column, 2 𝑡ℎ row & 3 𝑡ℎ row.Considering that the detection of backbone is CAM-based and has a nature performance gap (e.g. the performance of WSDDN v.s. Grad-CAM in Table 1 ) with the MIL-based method, we selected WSDDN as the reference in order to ensure the fairness of comparison.Impact of CBRPN: To investigate the impact of the proposed CBRPN, we conduct ablation studies by using SAM's proposal as a result, directly using proposals from SAM and filtering them based on the area before using. The results are summarized in Table 2 . As can be seen, compared to other proposal generation methods, CBRPN achieves higher mAP. And the lower flat AP is due to insufficient learning ability in the subsequent modules, as demonstrated in ablation experiments of the ARFM module. ' -' represents 0.0, and the possible reasons for this result are as follows: 1) more negative samples (compared with those related to SSW) which affect the learning of positive samples; 2) backbone network lacks the ability to learn small differences among classes.In addition, we also compared the average number of proposals used for training and the average overlap with ground truth boxes. The detailed results are in supplementary material.Impact of the design of ARFM: We investigate the effect of ARFM on the original reference network as well as different proposal filtering strategies. The results on CVC-ClinicDB and Kvasir are shown in Table 3 . Comparing strategy 1 v.s. 2, we can find that the addition of ARFM contributes to a gain of about 28.85% mAP for CVC-ClinicDB and 2.47% mAP for Kvasir, as well as increasing the AP of each morphology in different degree. Again, from the 3 𝑡ℎ strategy to 4 𝑡ℎ strategy ARFM outperforms 1 𝑡ℎ & 2 𝑡ℎ strategy in all metrics. Finally, comparing strategies 3 and 4 we find that SAM self-filtering based on the area can further improve performance on Kvasir, but is negative on CVC-ClinicDB. The probable reason for this is that the latter has many small polyps and the inappropriate area threshold filters them out. In a word, the proposed ARFM can help the network achieve better performance in all metrics. Impact of the components of CBRPN & ARFM: We further performed ablation experiments on the components within each module to verify their role for the module. The results of CBRPN {𝑆𝑆𝑊 , 𝑆𝐴𝑀} and ARFM {𝐶𝐺: 𝐶𝑜𝑛𝑣_𝑃 + 𝐺𝐴𝑃, 𝐴𝐹 : 𝐴𝑔𝑔 + 𝐹𝑢𝑠𝑖𝑜𝑛} are reported in Table 4 . ' / 'indicates that it cannot be evaluated because the input of subsequent modules is the output of AF. We can find that all components are beneficial to our framework because the performance decreases (4.83 ˜7.14 for CBRPN, 1.14 ˜1.99 for ARFM) while removing each component. 
conclusion: 
 In this paper, we propose a novel CBNet for image-level weakly supervised polyp detection. CBNet adopts CBRPN to automatically generate point prompts for SAM taking advantage of the local properties in the classification and adds iou threshold filter to improve the quality of proposals. We also propose the ARFM to enhance the region feature, which further helps to detect flat polyps and avoid over-fitting. Experiments on public datasets and internal demonstrate the superiority of our CBNet. 
 
 Appendix: https://polyp.grand-challenge.org/CVCClinicDB/https://datasets.simula.no/downloads/kvasir-seg.ziphttps://github.com/open-mmlab/mmdetectionhttps://github.com/researchmm/WSOD2 


Decision: Overall, all reviewers are satisfied with the response given by the authors, and are glad to see that the quality of the paper has been improved substantially.

Reviews: review: paper_summary: paper_summary: The authors introduce a novel Cooperation-Based Network (CBNet), a two-stage framework for polyp detection that is supervised by image labels. In the proposal generation stage, they present a Cooperation-Based Region Proposal Network (CBRPN) that leverages the collaboration of classification and segmentation to filter noise and remove incomplete polyp proposals, thereby mitigating the risk of overfitting. Additionally, they design an Adaptive ROI Fusion Module (ARFM) to enhance the detection of flat polyps by fusing multi-level region features. Extensive experiments demonstrate the effectiveness of the proposed method, achieving state-of-the-art performance on both public and private datasets.
strength_weakness: strength_weakness: 1. The authors effectively identify and address key challenges in current weakly supervised polyp detection approaches, including over-fitting, unfilterable noise, failed polyp masks without point annotation, and missed flat polyps. Their analysis and observations provide a solid foundation for the proposed model.
2. The adoption of the Segment Anything Model (SAM) to obtain relatively complete polyp proposals and filter out noise is both novel and effective.
3. The proposed method demonstrates a significant improvement over compared approaches. Extensive experiments show that the performance of CBNet not only surpasses other weakly supervised methods but also, in some respects, exceeds fully supervised methods, showcasing its effectiveness and potential impact in the field.
limitations: limitations: 1.The comparison approaches in the experimental section are somewhat outdated. Incorporating more recent state-of-the-art approaches would provide a stronger benchmark and better highlight the improvements and relevance of the proposed method.
paper_summary: paper_summary: This paper introduces a novel deep learning framework, termed CBNet, for Weakly Supervised Polyp Detection. The proposed method contains two innovative modules: Cooperation-Based Region Proposal Network (CBRPN) and Adaptive ROI Fusion Module (ARFM). The proposed method is interesting enough. However, the writing quality of this paper is not good. The authors should revise it for the camera-ready version.
strength_weakness: strength_weakness: 1) The code is available.
2) A comprehensive set of experiments in three datasets have been conducted. 
3) The proposed method is theoretically sound, and aligned with the problem formulation and the motivation.
limitations: limitations: 1) The writing quality of this paper is not particularly good. Some sentences need to be clarify: 

In lines 111-113, the authors point out that the MIL-based and SAM-aided approaches have complementary advantages and disadvantages. However, the authors do not mention about the disadvantages.

In line 135-138, the authors mention about "the former" and "the latter". "The former" and "the latter" mean what? 

What is SSW. In the related work section, authors only mention about the Selective Search (SS).

In line 370-374, this sentence could be rewritten for better clarity. the CBRPN takes the image I and feature F_1 - F_5 as inputs. The image I is used to generate the B_SSW and the feature F_1 - F_5 are used to produce the coordinates.

The authors should mention the WSDDN in the related work section. Since the MIDN is built upon the WSDDN.

The caption of Table 1 need to be rewrite. Since the best indicator of CBNet is red font and the best indicator of weak supervision is blue font. Besides, What is CBIA-WSPD in Table 1?

2) In Table 2, why SAM and SAM(filter) do not have results?
paper_summary: paper_summary: This paper proposes a novel cooperation-based weakly supervised polyp detection method (CBNet). Based on the four challenges facing the weakly supervised polyp detection model, two innovative modules are proposed correspondingly. In which the cooperation-based regional proposal network calculates IOU through proposals obtained from the proposal generator and masks obtained from SAM, leaving relatively complete information. And, the adaptive ROI fusion module combines deep-shallow region features to improve the model's ability to detect flat polyps. Extensive experiments on three polyp datasets demonstrate the effectiveness and superiority of the proposed method.
strength_weakness: strength_weakness: (1) This paper is well-written and well-organized. The motivation of the paper is quite clear.
(2) The proposed methodology is innovative. The cooperation-based regional proposal network is a promising idea. This method can filter background noise and reduce the risk of overfitting could be recognized.
(3) The experiment is quite thorough. Extensive experiments in the paper demonstrate that the proposed method achieves state-of-the-art performance. The proposed method shows higher accuracy compared to other weakly supervised methods and even higher than fully supervised methods.
limitations: limitations: (1) Missing statement of hyperparameters. I would like to know how the hyperparameters (e.g. $\tau$) are determined. Are there any papers referenced or relevant ablation experiments performed?
(2) Lack of comparison with the latest methods. This paper does show extensive experiments. However, the comparison with the latest weakly supervised detection method (published in 2022 or 2023) is missing, and thus the superiority of the proposed method cannot be fully proved.
(3) Inadequate explanation of some experiments. The paper lacks the explanation of why the mAP and CorLoc of the CBIA-WSPD (samAndssw) and CBIA-WSPD (samSelfFilterAndssw) methods differ more on the Kvasir dataset relative to the other datasets.
(4) MINOR: Some typos need to be corrected, i.e. 'Kvasir' is written as 'Kavsir' in Table 1.
paper_summary: paper_summary: This paper introduces a Cooperation-Based network (CBNet) for polyp detection, supervised by image labels, which operates in two stages. The model combines classification and segmentation to eliminate wrong proposals and achieve more accurate detection results by aggregating multi-level roi features. The proposed model aims to address several issues present in existing research: (1) over-fitting polyp detection; (2)unfilterable noise; (3)failed polyp mask without point annotation; (4)missed flat polyps. Moreover, experimental validation on three datasets demonstrates the excellent performance of the proposed model in the weakly supervised polyp detection task. Further, through ablation and visualization experiments, the effectiveness of each component of the model is further validated.
strength_weakness: strength_weakness: 1. The authors proposed a comprehensive model to address the challenges faced by existing methods. As for the CBRPN module, it effectively reduces the negative impact of noise and ensure the quality of generated proposals, thus improving the network's ability to capture relevant features. 
2. The ARFM module’s ability to aggregate multi-level ROI features enhances the accuracy and integrity of polyp detection. This addresses the challenge of detecting small and flat polyps that are often missed.
3. The paper includes extensive experiments on both public and private datasets, demonstrating the robustness and generalizability of the proposed method. And it achieves state-of-the-art performance for weakly supervised methods and, in some metrics, outperforms fully supervised approaches, showcasing its practical utility in clinical settings.
limitations: limitations: 1. There is still inconsistency between the text and the figures in the article. For instance, the description of the convolution operation in Equation 5 on page four is inconsistent with Fig. 3.
2. The ablation experiments only focused on the removal of each module. It would be more persuasive if further evaluation could be conducted to assess the roles played by individual components within each module.
3. The proposed method involves multiple stages and components, which may increase the computational complexity. Therefore, an analysis of its training time and model complexity is expected.