###### Abstract  
Ensuring precise segmentation of colorectal polyps holds critical importance in the early diagnosis and treatment of colorectal cancer. Nevertheless, existing deep learning-based segmentation methods are fully supervised, requiring extensive, precise, manual pixel-level annotation data, which leads to high annotation costs. Additionally, it remains challenging to train large-scale segmentation models when confronted with limited colonoscopy data. To address these issues, we introduce the general segmentation foundation model--the Segment Anything Model (SAM)--into the field of medical image segmentation. Fine-tuning the foundation model is an effective approach to tackle sample scarcity. However, current SAM fine-tuning techniques still rely on precise annotations. To overcome this limitation, we propose WSPolyp-SAM, a novel weakly supervised approach for colonoscopy polyp segmentation. WSPolyp-SAM utilizes weak annotations to guide SAM in generating segmentation masks, which are then treated as pseudo-labels to guide the fine-tuning of SAM, thereby reducing the dependence on precise annotation data. To improve the reliability and accuracy of pseudo-labels, we have designed a series of enhancement strategies to improve the quality of pseudo-labels and mitigate the negative impact of low-quality pseudo-labels. Experimental results on five medical image datasets demonstrate that WSPolyp-SAM outperforms current fully supervised mainstream polyp segmentation networks on the Ksavai-SEG, ColonDB, CVC-300, and EITS datasets. Furthermore, by using different amounts of training data in weakly supervised and fully supervised experiments, it is found that weakly supervised fine-tuning can save 70% to 73% of annotation time costs compared to fully supervised fine-tuning. This study provides a new perspective on the combination of weakly supervised learning and SAM models, significantly reducing annotation time and offering insights for further development in the field of colonoscopy polyp segmentation. w 29 April 2024 29 April 2024  
In recent years, substantial advancements have been achieved in both semantic segmentation [3; 4] and medical image segmentation [5; 6] through the utilization of deep learning methodologies. Among these advancements, methods for colonoscopy polyp segmentation have also been widely explored [7; 8]. Nevertheless, existing segmentation models often focus on specific imaging modalities and segmentation targets [9]. Constructing large-scale segmentation models necessitates a considerable volume of medical data. However, acquiring and annotating such data poses significant challenges and expenses, as they demand skilled physicians to conduct meticulous pixel-level annotations. To mitigate this challenge, fine-tuning of foundational models has emerged as an effective approach, particularly in scenarios with limited data availability [10]. In recent times, the Segment Anything Model (SAM) [11] has appeared as a noteworthy segmentation model. The SAM trained on a large-scale visual segmentation dataset exhibits robust generalization capabilities and adaptability to previously unseen data. Composed of an image encoder, a prompt encoder, and a mask decoder, SAM is adept at creating precise object masks derived from a variety of input cues, including points, bounding boxes, and existing masks. Considering its outstanding zero-shot segmentation performance, numerous researchers have begun fine-tuning SAM to tailor it for specific medical image segmentation domains. For instance, [PERSON] et al. [12] directly fine-tuned SAM (excluding the image encoder) for skin cancer segmentation tasks, while [PERSON] et al. [13] presented the Medical SAM Adapter (MSA), which embeds medical domain expertise by interspersing new components within the SAM's encoder-decoder framework. While these approaches have achieved notable advancements, their reliance primarily on fully supervised learning entails a considerable investment of time and manpower for precise pixel-level annotations.