SAM-MIL: A Spatial Contextual Aware Multiple Instance Learning Approach for Whole Slide Image Classification

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-08 13:58:49

Abstract: Multiple Instance Learning (MIL) represents the predominant framework in Whole Slide Image (WSI) classification, covering aspects such as sub-typing, diagnosis, and beyond. Current MIL models predominantly rely on instance-level features derived from pretrained models such as ResNet. These models segment each WSI into independent patches and extract features from these local patches, leading to a significant loss of global spatial context and restricting the model's focus to merely local features. To address this issue, we propose a novel MIL framework, named SAM-MIL, that emphasizes spatial contextual awareness and explicitly incorporates spatial context by extracting comprehensive, image-level information. The Segment Anything Model (SAM) represents a pioneering visual segmentation foundational model that can capture segmentation features without the need for additional fine-tuning, rendering it an outstanding tool for extracting spatial context directly from raw WSIs. Our approach includes the design of group feature extraction based on spatial context and a SAM-Guided Group Masking strategy to mitigate class imbalance issues. We implement a dynamic mask ratio for different segmentation categories and supplement these with representative group features of categories. Moreover, SAM-MIL divides instances to generate additional pseudo-bags, thereby augmenting the training set, and introduces consistency of spatial context across pseudo-bags to further enhance the model's performance. Experimental results on the CAMELYON-16 and TCGA lung cancer datasets demonstrate that our proposed SAM-MIL model outperforms existing mainstream methods in WSIs classification.

Main paper: 
 introduction: 
 The analysis of histopathological images is pivotal in modern medicine, particularly in cancer treatment, where it is considered the gold standard for diagnosis [21, 25, 31, 49] . The digitization of pathological images into Whole Slide Images (WSIs) via digital slide scanners has opened new avenues for computer-assisted analysis [8, 34] . Given the significant dimensions of WSIs and the absence of detailed pixel annotations, the analysis of histopathological images is commonly approached via Multiple Instance Learning (MIL) [10, 29, 36] , where MIL serves as a form of weakly supervised learning [26, 41] . In the MIL framework, each WSI or slide is considered as a bag, where each slide is divided into thousands of separate patches through a patching operation. A pre-trained feature extractor [17, 27, 34, 46] is used to extract features from each patch. The extracted instance features are stored within the bag as unmarked instances (patches) to serve as input data. A bag is classified as positive if it contains at least one instance indicative of disease; otherwise, it is classified as negative.MILWhile the MIL approach simplifies the problem by segmenting images into small patches for independent feature extraction [16, 17, 27, 34, 46] , this method fundamentally overlooks the global spatial context among patches. Since each patch is treated independently with features extracted in isolation, the MIL model tends to focus solely on local features, as illustrated in Figure 1(b) . The absence of this spatial context, especially in the ultra-high resolution context of WSIs, is detrimental to understanding tissue architecture, identifying pathological changes, and classifying diseases. For instance, certain pathological features may span multiple patches and can only be correctly identified and interpreted when considering the spatial relationship between these patches. Pathologists, when analyzing WSIs, consider the spatial layout of tissues and the relationships between adjacent patches to identify tumor markers and understand pathological processes, as depicted in Figure 1 (a). Previous studies have explored the modeling of spatial context through implicit methods. These approaches primarily involve constructing models capable of capturing long-range dependencies within images [1, 34] , as well as utilizing the spatial coordinates to form graph networks [6] . Nevertheless, these methods often fail to take full advantage of the richness of spatial contextual information. This implicit use of spatial context may underestimate the critical role of spatial relationships in tissue architecture analysis and tumor diagnosis.However, there has been insufficient focus on explicitly recovering and utilizing the spatial context lost during the patching process.To effectively integrate spatial contextual information into MIL, it is essential to leverage raw image-level features. As a pre-trained foundation model, the Segment Anything Model (SAM) [19] can be directly applied to WSI analysis, providing additional, semanticindependent visual segmentation prior features without the need for further training or fine-tuning. The SAM provides high-quality segmentation features from slides, thereby making it an outstanding tool for extracting spatial context from raw WSIs.In this paper, we propose a novel WSI classification method named SAM-MIL, which aims to mine and exploit spatial context information from slides to enhance the MIL model. As a foundational segmentation model, SAM acquires and encodes prior visual context knowledge in spatial and appearance aspects. By deducing these knowledge in SAM, the original slide is segmented, thereby explicitly modeling the relationships of instances based on neighboring relations and visual similarities, termed 'spatial context'. Utilizing this contextual knowledge, we elaborate a SAM-Guided Group Masking (SG 2 M) scheme that filters out the redundant instances by segments according to their areas to alleviate the extreme distribution imbalance of instances. To reduce the typical feature loss risk caused by the sharp instance masking ratio, we aggregate the instances by segment categories and complement them into the preserved instances for introducing a completed representation. Furthermore, SAM-MIL divides instances to create additional pseudo-bags, enhancing the training set, while ensuring spatial context consistency across pseudo-bags to further improve the model. Extensive experiments on several popular benchmarks demonstrate the superiority of SAM-MIL over baselines, and also confirms the importance of spatial contextual information in WSI classification. The contribution of this paper is summarized as follows:• We propose a meticulously MIL framework based on spatial contextual awareness named SAM-MIL. We pioneer the use of SAM to extract spatial contexts in  
related work 2.1 multiple instance learning in wsi analysis: 
 Multiple Instance Learning (MIL) [10] represents the most widely applied paradigm in WSI analysis [3, 17, 27] . Given the ultra-high resolution of WSIs, instance features are typically extracted using pre-trained models [4, 16, 20, 27, 33, 50] . Feature extraction from local instances eliminates the spatial context in the original WSIs, and the extracted features only contain localized information in the delineated patches, which is used for subsequent bag label prediction. Previous algorithms can be broadly categorized into two types: instance-level [3, 11, 14, 44] and embedding-level [8, 35, 42, 43, 46] . The former obtains instance labels and aggregates them to obtain bag labels, while the latter aggregates all instance features into a high-level bag embedding for bag prediction. However, the MIL model described previously did not focus on how to recover and utilize the spatial context lost during the patching process. Instead, it used the patch features directly for training. Some studies are attempting to reconstruct spatial context. For example, Chen et al. [6] conceptualizes whole slide images as 2D point clouds, employing Patch-based Graph Convolutional Networks to foster context-aware survival predictions by leveraging the spatial relationships between patches to learn context-aware embeddings implicitly. Bai et al. [1] introduces a novel framework utilizing transformers for object localization, implicitly capturing spatial context through activation diffusion techniques. But the reconstruction of spatial context in current MIL models is primarily done implicitly. In this paper, we aim to provide explicit spatial context to guide the MIL model training by introducing visual features at the image level to improve the performance of WSIs classification.Foundation models have profoundly revolutionized traditional artificial intelligence across various domains, including medicine [5, 24] , due to their exceptional zero-shot and few-shot generalization capabilities in downstream tasks [22, 40] . Among these, the Segment Anything Model (SAM) [19] , as a pioneering image segmentation foundation model, has garnered widespread attention for its ability to generate accurate target masks in a fully automatic or interactive manner. This marks the entry of the prompt-driven paradigm into the realm of image segmentation, achieving favorable outcomes in numerous tasks [7, 12, 28, 39, 45] . However, due to the uniqueness of medical images (with traditional foundation models predominantly trained on non-medical datasets), the effectiveness of SAM in medical imaging still requires continuous exploration. Nevertheless, extensive work has already introduced SAM into the field of medical images, mainly focusing on lower-resolution image segmentation tasks [15, 18, 30, 32, 38, 47] . This work can be roughly categorized by medical imaging modalities: 1) CT images [15, 18, 32] ,2) MRI images [30, 47] , 3) Endoscopic images [38] , etc. However, much of this work remains a step behind the current mainstream methods, requiring fine-tuning of the SAM model or the provision of suitable prompts. Hence, the direct application of SAM to medical imaging tasks is limited in generalizability, with significant differences across various datasets and tasks [48] . Due to the ultra-high resolution of WSI images and the extremely small proportion of diseased areas, there have been few attempts to introduce SAM. Deng et al. [9] evaluated SAM for tumor segmentation, non-tumor tissue segmentation, and cell nucleus segmentation in WSIs data, demonstrating the feasibility of SAM application in WSIs. Beyond this, there is virtually no work involving SAM in the WSIs domain.Previous work has demonstrated the generalizability of SAM and its usefulness in medical images, but due to the specificity of medical images, there are still substantial challenges for SAM to be used directly as a medical image segmentation model. In this paper, we explore the adaptation of the original SAM to WSIs classification tasks. Through ingenious model design, the spatial context mined by SAM is utilized to guide the training of MIL models, offering a novel approach for the application of SAM in medical images. 
proposed method 3.1 preliminary: 
 Within the paradigm of MIL, we encounter the scenario where a data collection is comprised of bags, denoted as B = {B 1 , B 2 , ..., B N }, each consisting of a number of instances B i = {𝑥 𝑖 1 , 𝑥 𝑖 2 , ..., 𝑥 𝑖 𝑛 } in a 𝐷dimensional feature space. In classification tasks, a bag is associated with a predetermined label 𝑌 , while each instance within the bag carries an unspecified label 𝑦 𝑖 . The bag is classified as positive if it contains at least one positive instance; if not, it is deemed negative. The learning objective of a MIL model M is to infer the bag label considering the information embedded within its instances, formalized as:EQUATIONwhere F denotes the feature extraction function, transforming each instance into a more expressive feature representation; A is the feature aggregation function that synthesizes the extracted features into a bag representation; and C is the classification function, predicting the bag label from its aggregated feature representation. However, traditional MIL paradigms primarily concentrate on the local features of patches, resulting in a substantial overlook of spatial context. In this paper, we will explicitly introduce spatial context for guiding model training via SAM in the feature extraction phase and the feature aggregation phase: where 𝑆𝐴𝑀 denotes the method that is guided by the spatial context within the Segment Anything Model.EQUATIONAs illustrated in Figure 2 , we introduce a novel approach to the MIL model that enhances classification performance by incorporating spatial context. Previous works predominantly exploited local features from WSIs, overlooking the potential of global visual content.To address this, we integrate spatial context from the SAM into our MIL framework. Specifically, we have designed a composite feature extractor that not only includes the traditional feature extraction phase but also leverages the segmentation results from SAM to extract a representative group feature from each segmented region. Let 𝑆 = {𝑠 1 , 𝑠 2 , . . . , 𝑠 𝑙 } denote the output of the SAM, representing 𝑙 segmented areas. For each segmented area 𝑠 𝑟 , the composite feature 𝑔 𝑟 is extracted as follows:EQUATIONwhere G is an average aggregation function used to extract group features from each segmented area. These features are then incorporated as special tokens in the training process of the MIL model,EQUATIONThis allows our MIL model to capture the intricate spatial context of the image, thereby improving classification performance. By leveraging the patch features and SAM segmentations extracted during the instance feature extraction phase, we can incorporate SAM's spatial context at two levels during the instance feature aggregation stage. At the instance level, the abundance of similar and redundant patches in WSIs can lead the MIL model to excessively concentrate on trivial details. This focus on non-essential information can adversely affect the model's ability to generalize and its overall efficiency. Therefore, we employ a SAM-Guided Group Masking (SG 2 M) strategy based on spatial context to mask the instance features and mitigate class imbalance in WSIs. This process is represented by the following formula:EQUATIONwhere 𝑀 𝑝 is the patch mask determined by SAM information, ⊙ represents the Hadamard product, 𝑒 𝑝 is the masked patch feature embedding, and 𝐼 represents the instance features.At the bag level, SAM-MIL augments the training set by dividing it into additional pseudo-bags and ensuring spatial context consistency across these bags, thereby enhancing its efficacy. Given the paucity of slide labels, we introduce an approach to create a synthetic diversity of training data. Instance features are randomly allocated into 𝑚 pseudo bags, as formalized by the following equation:EQUATIONwhere 𝑃 𝑘 1 , 𝑃 𝑘 2 , . . . , 𝑃 𝑘 𝑚 are the feature sets corresponding to each pseudo bag.We utilize the corresponding SAM segmentation information at both levels to enhance model optimization, and we provide a detailed description of the implementation for each component of this approach below.In the Instance-Level, we introduce a large-scale instance masking strategy, named SAM-Guided Group Masking (SG 2 M), which is designed to filter out instances that significantly impact the final classification task. Given the presence of numerous similar and redundant instances within WSIs, this repetitive information may cause the MIL model to excessively focus on unimportant details, thereby compromising the model's generalization ability and efficiency. Consequently, the goal of the masking strategy is to diminish the influence of these unimportant instances, ensuring that the model focuses on critical areas. To eliminate the excess of redundant information in the data through masking, we designed an instance-level masking strategy as depicted in Figure 3 .Random Masking Strategy: As demonstrated in Figure 3 (a) and (b), we introduce two parallel types of random masking methods. The first is the "full-feature random mask, " which randomly masks the entire feature set to diminish the influence of certain instances, thus reducing redundant information. The second strategy is the "non-group feature random mask," which selectively masks only the features of ordinary instances not included in the group features. This approach strives to preserve the important structural information identified by SAM.SAM-Guided Group Masking Strategy: Since the redundancy degree varies among different types of patches in a slide, employing direct fair masking without consideration can exacerbate the scarcity of already scarce patches. Consequently, we group all patches based on the spatial context provided by SAM, assigning each group a dynamic mask ratio determined by the area information as assessed by SAM. Figure 3 (c) showcases a more refined masking method. Firstly, instances are classified into segment groups {𝐺 1 , 𝐺 2 , . . . , 𝐺 𝑧 } based on the segmented areas determined by SAM. Each instance 𝑝 𝑖 is assigned to a segment group 𝐺 𝑘 according to the category of segmentation it falls under, with the assignment 𝑝 𝑖 → 𝐺 𝑘 indicating that 𝑘 is the segment category index of 𝑝 𝑖 .For each segment group 𝐺 𝑘 , we define 𝐴 𝐺 𝑘 as the area of the group category 𝑘. Instead of using a direct ratio, we process 𝐴 𝐺 𝑘 through an adjusted sigmoid function to calculate a normalized ratio 𝑅 𝐺 𝑘 for each group,EQUATIONwhere 𝜎 adj is the adjusted sigmoid function with slope 𝑎 and centerpoint shift 𝑏, parameters that control the steepness and the horizontal shift of the sigmoid curve, respectively. The normalized ratio 𝑅 𝐺 𝑘 is then multiplied by the target mask ratio provided as input (𝑀𝑅 target ), to calculate the final mask ratio for each segment groupEQUATIONThis ratio defines the extent to which features in each segment group will be masked. Each segment group's mask is calculated and then aggregated to create a composite mask 𝑀 comp for all instances,EQUATIONFinally, the composite mask 𝑀 comp is applied across all instances to produce the masked feature set,EQUATIONwhere 𝑀𝑎𝑠𝑘 (•, •) is performing mask operations on instances.By dividing the instance features from the WSIs into multiple pseudo bags, we effectively amplify our training dataset. Each pseudo-bag inherits the label of its originating slide, thus preserving the ground truth in the expanded dataset. This division into pseudo-bags maximizes the use of available data, reduces model overfitting, and enhances the learning potential of the model. These pseudo-bags are then individually passed through the MIL model for optimization. As illustrated in Figure 4 , the mean training loss EQUATIONwhere L MIL represents the loss function of the MIL model, 𝑃 𝑖 represents the instance features in the 𝑖-th pseudo bag, and 𝑌 the inherited label from the WSI.In addition to the classification loss, we incorporate a consistency constraint based on the segmentation category information derived from SAM. This constraint is based on the premise that global attention weights within the same segmentation category ought to be more similar, thereby enhancing the model's interpretability and its fidelity to the underlying pathology. The consistency loss calculation is as follows:EQUATIONwhere sim represents a similarity function between the attention weights 𝑎𝑡𝑡𝑛 𝑖 and 𝑎𝑡𝑡𝑛 𝑗 , and 𝛿 is a Kronecker delta function that equals 1 if the segmentation categories 𝑠 𝑖 and 𝑠 𝑗 are the same, and 0 otherwise. The attention weights are derived from the MIL model's attention mechanism, and 𝑠 𝑖 , 𝑠 𝑗 are the segmentation categories from SAM.The losses at both the instance level and the bag level, along with the consistency loss, are aggregated for overall model optimization, ensuring that the attention mechanism is not only guided by the MIL classification objective but also respects the intrinsic histological patterns identified by SAM,EQUATIONwhere 𝛼 and 𝛽 are scaling factors.The MIL framework implemented in this work builds upon the widely used AB-MIL [17] architecture prevalent in the WSI domain. 
wsi preprocessing: 
 In order to integrate the segmentation of SAM into the preprocessing workflow of WSIs, it is necessary to adapt the traditional preprocessing protocols. We adopt the data preprocessing protocol Patching & SAM Segmentation: After completing foreground segmentation, for each slide, our algorithm meticulously crops patches of size 512 × 512 from within the segmented foreground contours. It utilizes the HDF5 hierarchical structure to store the image patches along with their coordinates and the slide metadata in a stacked data format. Moreover, each slide is subject to segmentation within the segmented foreground using SAM, with the segmentation information being saved in HDF5 format.Feature & SAM Info Extraction: For each slide, we employ deep convolutional neural networks (the ResNet50 model pretrained on ImageNet, or other feature extractors) to compute local feature representations for each image patch, transforming each 512 × 512 patch into a 1024-dimensional feature vector. In addition to the original feature extraction, utilizing the SAM segmentation information saved in the previous stage, a global feature representing each segmentation category is extracted for regions within each category, termed group features, and this information is explicitly annotated and saved in the feature file. Furthermore, the area information of each patch within the SAM segmentation is recorded to provide guidance for subsequent training.In addition to employing the original process for preprocessing WSIs, we offer a method to generate the aforementioned SAMguided feature files and related information from previously extracted feature files, reducing the time spent on redundant feature extraction. Please check Supplementary Material for more details.We use CAMELYON-16 [2] (C16), and TCGA-NSCLC to evaluate the performance on diagnosis and sub-typing tasks. For details on the dataset, please refer to the Supplementary Material.Based on previous work [27, 34] , model performance is evaluated using accuracy, area under the curve (AUC), and F1 score. AUC is the primary performance metric in binary classification tasks, and only the AUC and F1 score are reported in the ablation experiments. We adopt ResNet50 [13] pre-trained with ImageNet-1k as the feature extractors. Supplementary Material offers more details.We mainly compare with AB-MIL [17] , DSMIL [20] , CLAM-SB [27] , CLAM-MB [27] , TransMIL [34] , DTFD-MIL [46] , IBMIL [23] , and MHIM-MIL [37] . In addition, we compared two traditional MIL pooling operations, Max-pooling and Mean-pooling. Due to differences in the datasets, the results of all other methods were reproduced using the provided official code under identical settings.As demonstrated in Table 1 , most of the MIL models in WSI classification tasks focus on how to better utilize local patch features for learning, and many effective models have been designed to improve performance. For instance, MHIM [37] enhances the training for student models by mining hard examples and applying masking to examples based on attention scores during the training process. However, none of the previous work explicitly introduces spatial context; rather, all modeling relies solely on local patch features. The model focuses only on the visual features within each patch obtained through patching, potentially leading to a decrease in performance. Our proposed SAM-MIL explicitly incorporates spatial context and achieves significant performance improvements (96.08% AUC on CAMELYON-16 and 96.01% AUC on TCGA) in both datasets by leveraging segmentation prior information from SAM to guide the training. The performance surpasses that of both MHIM [37] and IBMIL [23] , which require two-stage training, underscoring the importance of spatial context information for model training. This not only demonstrates that the direct introduction of spatial context has a positive effect on model training, but also reflects that our designed model is capable of effectively utilizing the additional spatial context to aid in model training. AUC on CAMELYON-16 and 95.84% AUC on TCGA). This efficacy underscores the value of incorporating spatial contextual information and of applying varied mask ratios across different categories to significantly reduce redundant information. Overall, our proposed instance-level masking strategies have achieved excellent results, confirming their effectiveness across different datasets.Impact of Different Mask-Ratio Function. Specifically, in the SAM-Guided Group Mask strategy, masks were applied to different categories' groups at varying ratios based on their area. Calculation of the corresponding ratio from the category's area is facilitated by a designated function. In our experiments, three different functions were evaluated: constant, linear, and adjusted sigmoid. The experimental results for these functions are shown in Table 4 . First, a fixed ratio showed limited improvements due to data imbalance. A linear function, influenced negatively by extreme values, led to poor performance. The adoption of an adjusted sigmoid function, addressing the drawbacks of the first two methods by adjusting its center-point and slope, resulted in superior performance across datasets.Our proposed SAM-Guided Feature Extractor's group features are suitable not only for our designed MIL framework but also for application to various mainstream MIL models, thereby achieving immediate performance improvements. In our experiments, features extracted by ResNet50 [13] and PLIP [16] were processed under the guidance of SAM and validated in the CAMELYON-16 dataset. Besides the previously mentioned baseline AB-MIL, we selected two advanced MIL models [34, 37] to assess the effectiveness of our group features. The corresponding experimental results are displayed in both methods. Furthermore, it is observed that, due to the lack of pre-training on medical images, the features extracted using ResNet50 are somewhat inferior in performance compared to those extracted using PLIP. However, after supplementing with group features based on spatial context extraction, the features extracted by ResNet50 demonstrated a greater improvement in performance. This potentially suggests that the group features we proposed provide significant guidance for models lacking prior knowledge. Overall, our SAM-Guided Feature Extractor can serve as a plug-and-play module applicable to various mainstream MIL models.To more intuitively verify the accuracy and effectiveness of the SAM segmentation results, we conducted a comparative visualization of the original tumor and normal slides as shown in Figure 5 . The images are arranged from high to low resolution, respectively showcasing SAM's segmentation capabilities from macro to micro levels. From the comparison in the figure, we can observe that the areas outlined in blue lines (tumor regions) in the tumor slides are successfully segmented by SAM, indicating that SAM's segmentation of WSIs is both effective and accurate visually. Similarly, in the normal slides, SAM also segments the WSIs based on visual information, providing additional visual segmentation prior information. On a macro level, whether in tumor or normal slides, SAM is capable of extracting a large number of visually similar regions (which may contain a large number of similar instances leading to redundancy). On a micro level, SAM shows good sensitivity to small local regions, which may be quite important for classification tasks. The experimental results and visualizations also demonstrate that SAM can serve as an effective spatial context extractor for guiding model learning by extracting semantic-free segmentation features, effectively defining the relations of instances. The potential groups segmented by SAM efficiently distinguish between normal and tumor tissues, ensuring the reliability of the extracted spatial context. In addition to the comparison of SAM's segmentation results, we will provide more visualization information in the Supplementary Material to help better understand the model. 
conclusion: 
 In this work, we rethink the impact of spatial contextual information for MIL-based WSI classification tasks. We demonstrate that explicit spatial contextual information is beneficial for MIL classification, and that independent local features may cause the model to neglect the connections between patches and higher-level tissue architecture relations. To address this issue, we employ the visual segmentation foundation model SAM to introduce spatial context by extracting spatial contextual information from image hierarchies in WSIs. Meanwhile, we design multiple components to explicitly introduce the extracted spatial context into the MIL model, thereby guiding the classification of WSIs. Specifically, we develop a SAM-Guided Group Masking strategy to mask instances using spatial contextual information, and extract the representative group features from each category, thereby compensating for the loss of information caused by the masking operation and providing macroscopic feature information. In addition, we introduce spatial context-based consistency loss constraints to enhance the spatial contextual information during pseudo-bags training. The experimental results demonstrate the superiority and versatility of the SAM-MIL framework relative to other methods and further highlight the positive impact of the explicitly introduced spatial context on the MIL model.  
 
 Appendix:  


Decision: Overall, all reviewers are satisfied with the response given by the authors, and are glad to see that the quality of the paper has been improved substantially.

Reviews: review: paper_summary: paper_summary: This paper proposes a spatial contextual aware MIL approach (SAM-MIL) for whole slide image classification. Current MIL-based methods usually focus on the local small patches, leading to a significant loss of global contextual spatial information. SAM-MIL explicitly models the spatial information by utilizing SAM and proposes several spatial contextual information mining strategies at both the bag level and instance level.
strength_weakness: strength_weakness: 1. The writing and logical structure of this paper are good and easy to read.
2. Improving the MIL's framework performance by utilizing the SAM-guided spatial contextual information is novel in the field of digital pathology.
3. Extensive comparable experiments and ablation studies verify the effectiveness of the proposed SAM-MIL.
limitations: limitations: 1. Modeling the spatial context in the WSI to enhance the performance of the MIL framework is a hot topic.  Some latest graph-based MIL-related work is missing [1-4]. 
2. SAM is pre-trained in the natural image. The specific discussion about why it can be directly used to segment the WSI needs to be analyzed.
3. SAM relies on visual or textual prompts to perform segmentation. However, I couldn't find a description of the prompts used in this paper. How does the author utilize SAM to generate the segmentation mask? 
4. There are some other methods to segment the WSI and provide spatial information, such as K-means and SLIC [5]. A comparison and discussion between SAM-based methods and these traditional methods is necessary.
5. In the Camelyon-16 dataset, the baseline results (e.g., DTFD-MIL and TransMIL) match those reported in the paper [6]. Why are the MHIM-MIL results different? 
6. A paired t-test statistical analysis (p-value > 0.5) is necessary to evaluate the significance of the performance improvement.
7. Why are there white Spaces in the upper and left sides of the segmentation results in the lower left corner of Figure 5?

Reference:
[1] Hou W, Yu L, Lin C, et al. H^2-MIL: exploring hierarchical representation with heterogeneous multiple instance learning for whole slide image analysis[C]//Proceedings of the AAAI conference on artificial intelligence. 2022, 36(1): 933-941.
[2] Lee Y, Park J H, Oh S, et al. Derivation of prognostic contextual histopathological features from whole-slide images of tumours via graph deep learning[J]. Nature Biomedical Engineering, 2022: 1-15.
[3] Chan T H, Cendra F J, Ma L, et al. Histopathology whole slide image analysis with heterogeneous graph representation learning[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023: 15661-15670.
[4] Li J, Chen Y, Chu H, et al. Dynamic Graph Representation with Knowledge-aware Attention for Histopathology Whole Slide Image Analysis[J]. arXiv preprint arXiv:2403.07719, 2024. 
[5] Achanta R, Shaji A, Smith K, et al. SLIC superpixels compared to state-of-the-art superpixel methods[J]. IEEE transactions on pattern analysis and machine intelligence, 2012, 34(11): 2274-2282.
[6] Tang W, Huang S, Zhang X, et al. Multiple instance learning framework with masked hard instance mining for whole slide image classification[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023: 4078-4087.
paper_summary: paper_summary: This paper proposes an MIL framework, named SAM-MIL, that emphasizes spatial contextual awareness and explicitly incorporates spatial context by extracting comprehensive, image-level information. The design of group feature extraction based on spatial context and a SAM-guided group masking strategy are used to mitigate class imbalance issues. A dynamic mask ratio for different segmentation categories is utilized. SAMMIL divides instances to generate additional pseudo-bags, and introduces consistency of spatial context across pseudo-bags to enhance the model’s performance. Experimental results on the CAMELYON-16 and TCGA Lung Cancer datasets demonstrate the proposed SAM-MIL model is effective.
strength_weakness: strength_weakness: 1.	Overall, this paper is written clearly.
2.	The topic is interesting and significant.
3.	Many experiments are provided to support the conclusions.
limitations: limitations: 1.	In the Title and Abstract, medical images should be mentioned to better understand whole slide image.
2.	This paper emphasizes spatial contextual awareness, but the concept is general and almost all methods in this field perform spatial contextual awareness.
3.	In experiments, many details are missing, such as some settings on patches, bags, and multiple instance learning.
4.	My main concern is whether or not SAM is effective for segmenting medical slide images.
paper_summary: paper_summary: This paper proposes a novel MIL framework to address the problem that current methods usually lead to a significant loss of global spatial context and restrict the model’s focus to merely local features.
strength_weakness: strength_weakness: The figures in this paper are beautiful. Additionally, the paper demonstrates several advantages. First, it significantly improves the performance of multi-instance learning (MIL) models on the CAMELYON-16 and TCGA Lung Cancer datasets by utilizing the Segment Anything Model (SAM) to extract spatial context information. Second, it presents a comprehensive evaluation, demonstrating superior performance compared to mainstream MIL methods, thereby validating the effectiveness of incorporating spatial context in whole slide image (WSI) classification. Third, the enhancement of the training set by creating pseudo-bags while maintaining spatial context consistency is a noteworthy contribution that further improves model robustness and classification accuracy.
limitations: limitations: The experiments were inadequate and poorly analyzed, describing only the general phenomena, but not the causes and reasons behind them. Besides, the dataset scope is relatively narrow, focusing primarily on the CAMELYON-16 and TCGA Lung Cancer datasets, which may not fully represent the diversity of WSI data encountered in clinical settings. The computational complexity of the proposed SAM-MIL framework is high, potentially limiting its practical application in resource-constrained environments. Additionally, the generalizability of SAM to different types of medical images remains unproven, raising concerns about its versatility and applicability in broader medical contexts.