NcIEMIL: Rethinking Decoupled Multiple Instance Learning Framework for Histopathological Slide Classification

Authors: qiehe  sun
doukou  jiang
renao  yan
tian  guan
zhiqiang  cheng
Venue: MIDL_io_2024_Conference
Published: 2024-01-29 14:49:20

Abstract: On account of superiority in annotation efficiency, multiple instance learning (MIL) has proved to be a promising framework for the whole slide image (WSI) classification in pathological diagnosis. However, current methods employ fully- or semi-decoupled frameworks to address the trade-off between billions of pixels and limited computational resources. This exacerbates the information bottleneck, leading to instance representations in a high-rank space that contains semantic redundancy compared to the potential low-rank category space of instances. Additionally, most negative instances are also independent of the positive properties of the bag. To address this, we introduce a weakly annotation-supervised filtering network, aiming to restore the low-rank nature of the slide-level representations. We then design a parallel aggregation structure that utilizes spatial attention mechanisms to model inter-correlation between instances and simultaneously assigns corresponding weights to channel dimensions to alleviate the redundant information introduced by feature extraction. Extensive experiments on the private gastrointestinal chemotaxis dataset and CAMELYON16 breast dataset show that our proposed framework is capable of handling both binary and multivariate classification problems and outperforms state-of-the-art MIL-based methods. The code is available at: https://github.com/polyethylene16/NcIEMIL.

Main paper: 
 introduction: 
 Histopathological slide examination which commonly requires a lot of time and effort from pathologists is seen as the "gold standard" in the clinical diagnosis (Aeffner et al., 2017; Cai et al., 2021) . And computational pathology seeks to reduce the burden on physicians by employing algorithms (Kather et al., 2019; Skrede et al., 2020; Greenwald et al., 2022) .Tissue samples are stained, scanned, and stored as digital images with different magnifications in a pyramid structure known as the whole slide image (WSI). The lower layers of WSI are utilized to study the tumor heterogeneity in cell morphology, necessitating the feeding of billions of pixels into networks at once. However, GPUs struggle to handle such many parameters simultaneously (Tellez et al., 2019) . In this case, multiple instance learning (MIL) that decouples the computational process is more effective (Quellec et al., 2017) .Slides are meshed into numerous patches at a specific magnification. "patch" and "slide" correspond to "instance" and "bag" in MIL, respectively. A bag's characteristic is a collection of instance attributes. For slide classifcation, MIL frameworks are decoupled into two independent parts (Ilse et al., 2018) , with a deep extractor being employed to understand salient features of instances, and a shallow aggregator responsible for the integration and mapping of bags. Since the above networks are trained separately, the computational cost is greatly reduced. Specifically, on the one hand, patching enables instances to be fed into the extractor in batches, avoiding the necessity to perform large matrix operations in a single pass. On the other hand, the global pooling layer at the end of the extractor substantially reduces dimensionality, facilitating straightforward linear transformations for the assembled bag-level representations. However, upon revisiting the aforementioned process, we identified information redundancy originating from multiple sources:• Gradient operation and backpropagation. Slide-level labels serve as the exclusive source of supervised signals, and their gradients are solely fed back into the shallow aggregator. Conversely, the extractor is not constrained, resulting in a lack of understanding regarding the task-relevant property of the instances.• Instance-level feature concatenation. In slides, the distribution of key instances tends to be sparse. However, when forming a bag embedding, all instances are involved, which leads to many irrelevant instances being of interest during aggregation.• Redundancy in the channel dimension. For a given task, instances can be perceived as manifolds in a low-dimensional space. However, extractors often embed instances in high-dimension to prevent information loss, regardless of their utility.For this reason, we sought to optimize the interference terms mentioned above and proposed a Non-crucial Information Elimination-based MIL (NcIEMIL) architecture. In customizing the extractor, we followed the approach of Campanella et al. (Campanella et al., 2019) by associating the corresponding bag-level annotation with the single instance that the extractor deems most likely to be positive, thus making full use of limited supervised signals. Subsequently, we exclude irrelevant instances based on the ranking of positive probability and forming a more refined bag embedding. The noise of channel dimension is specifically addressed during aggregation. We design a bi-parallel aggregator, introducing channel attention to weigh each embedding dimension, aiming to approximate the lowdimensional manifold. Simultaneously, we maintain the advantage of spatial attention, dynamically modeling the influence of different instances. To comprehensively validate the framework's effectiveness, we conducted experiments on the CAMELYON16 dataset and the retrospectively collected BgIM gastric mucosal biopsy dataset. 
related work: 
 Initially employed for drug activity prediction (Dietterich et al., 1997) , multiple instance learning is widely used in the whole slide image research (Campanella et al., 2019; Kanavati et al., 2020; Chen et al., 2021a; Marini et al., 2022) . It aligns with the characteristics of pathology data, which is large but unlabeled, intending to reduce the daily workload of pathologists. MIL is categorized into instance-and embedding-level methods (Ilse et al., 2018) . Extensive research has been conducted on the latter as it is well-suited for more complex application scenarios, and our method also belongs to this category.Due to the absence of a standard database, MIL extractors typically employ neural networks pre-trained on ImageNet (Deng et al., 2009) to map instances into a hidden space, retaining basic knowledge of color and texture (Lu et al., 2021; Shao et al., 2021; Zhang et al., 2022) . However, histopathological images significantly differ from natural images (Wang et al., 2022; Kang et al., 2023) , a distinction increasingly emphasized with the advancement of selfsupervised learning and extended to MIL (Li et al., 2021; Chen et al., 2022) . Additionally, the semi-decoupled framework selects instances deemed important by the aggregator (Qu et al., 2022; Yu et al., 2023) , providing supervision to the extractor but being prone to introducing errors. In contrast, we fully utilize slide-level labels, referencing instance-level methods (Campanella et al., 2019) to impart more determinism to the extraction network.To identify the active region and eliminate extraneous interference, the instances were manually recalibrated by introducing extra supervision (Wang et al., 2019) . Class activation mapping (CAM) (Zhou et al., 2016) was used to indicate lesion regions on thumbnails of WSI, although with some imprecision (Chen et al., 2021b) . In this paper, we calculate the positive scores and select discriminative instances accordingly.Max-and avg-pooling serve as the simplest aggregation functions, representing two extreme assumptions in the MIL formulation that are prone to opposite judgments due to biases in the extraction process. Recognizing the diverse contributions of each instance, attention mechanism is employed to efficiently aggregate representations of instances (Ilse et al., 2018; Lu et al., 2021) . Due to the great success of Transformer (Vaswani et al., 2017) , self-attention (SA) has emerged as an alternative (Shao et al., 2021; Zheng et al., 2022) . However, its quadratic complexity is called into question due to the significantly larger number of tokens in MIL compared to other domains. In this paper, by eliminating most non-discriminative instances, our aggregator is still constructed on top of SA, maintaining computational efficiency within an acceptable range. 
multiple instance learning formulation: 
 For clarity, we present the formulation of MIL. Considering a binary problem as an example, for a given bag X = {x 1 , x 2 , • • • , x n } and its corresponding label Y ∈ {0, 1}, where x i denotes the i-th instance of X and n is the numbr of instances. We assume that the true label of x i is y i ∈ {0, 1}, which is not known in practice. Then MIL can be described as:EQUATIONAssuming the instances within X are independently and identically distributed, Eq. 1 can be generalized (Zaheer et al., 2017) as:Y = S(X) = g( i f (x i )) (2)where S(•) is a scoring function, and f (•) and g(•) are two suitable transformations. MIL is decoupled into two steps. In the embedding-level approach, f (•) is responsible for feature embedding. The summation function transforms into concatenation, and g(•) is an aggregation function that maps the bag-level representation to the category space.In practice, two different networks, f θ and g τ , perform the duties of f (•) and g(•), where θ and τ are learnable parameters. We observed that a substantial amount of irrelevant information was forced to be retained or introduced during the process. Hence, we introduce Non-crucial Information Elimination (NcIE) to alleviate this trend, as shown in Fig. 1 .Weakly-supervision for Extraction. The gradient backpropagation stops between the aggregator and the extractor due to physical patching and splicing. As a result, the extractor remains unconstrained. Taking inspiration from the work of Campanella et al. (Campanella et al., 2019) , we incorporate the ground truth Y into the pre-training of the extractor. In particular, we append a mapping head h θ ′ to f θ to generate the positive probability P (x i ) = P(h θ ′ (f θ (x i )) = y * |x i ) for the instance x i , where y * represents the positive class. The instance with the highest positive probability contributes to the parameter update, and the loss can be expressed as:EQUATIONwhere ŷ is the score of the instance that is most likely to be positive. n is not a constant as the foreground areas of slides vary, and θ and θ ′ are updated simultaneously.Discriminative Instance Selection. In histopathological sections, only a small portion is pertinent to the problem. Numerous irrelevant instances introduce noise for the aggregator, leading to a drastic shift in attention (Yan et al., 2023) . We precisely filter potential discriminative instances by assigning each instance from the same bag a corresponding score that is produced by h θ ′ , and then re-ranking instances in descending order:EQUATIONThe first K and the last K instances of X are chosen to create a new pseudo-bagX ′ = {x ′ 1 , • • • , x ′ K , x ′ n-K+1 , • • • , x ′ n }.Including the latter K instances aims to implicitly construct negative samples for discriminative instances while avoiding extreme data distribution.Hybrid Attention-Based Aggregator. According to Eq. 1, for a given task, ideally, f (x i ) should be a low-dimensional manifold indicative of potential categories. However, f θ (x i ) is commonly a high-dimensional vector, implying redundancy in the channel dimension of bag embedding. Therefore, we use squeeze and excitation (Hu et al., 2018) to capture channel validity for it dynamically:u channel = sigmoid(W 2 σ(W 1 • 1 2K 2K k=1 u(k))) • u (5) where u = || x k ∈X ′ f θ (x k ) ∈ R 2K×ddenotes the bag embedding, obtained by concatenating instance embeddings, with d being the embedding dimension. σ is an activation function, while W 1 and W 2 are linear transformations. We maintain self-attention to model the influence of instances and their spatial correlation:u spatial = softmax(QK T / √ d)V Q = (v||u)W Q , K = (v||u)W K , V = (v||u)W V (6)where W Q , W K and W V are learnable, and v ∈ R 1×d is a learnable embedding that represents a virtual instance used for classification. The above process is illustrated in Fig. 1 (c). We designed an aggregator based on hybrid attention in a "parallel-merge" structure.The intersection of attention is achieved through cross-attention:u ′ = softmax(u spatial W ′ Q (u channel W ′ K ) T / √ d)u channel W ′ V (7)where W ′ Q , W ′ K and W ′ V are also learnable linear projections. Then the aggregated v ′ is extracted from the spatial dimension of u ′ ∈ R (1+2K)×d and converted to the result by a fully-connected layer. Breast Dataset. Camelyon16 (Bejnordi et al., 2017 ) is a publicly unbalanced dataset focused on differentiating between cancer and non-cancer cases for metastasis detection in breast cancer, which consists of 270 slides for training and 129 for testing. Following pre-processing, we acquired approximately 2.6 million patches at 20× magnification, averaging 7,346 patches per slide.BgIM Gastric Mucosa Dataset. Intestinal epithelial metaplasia is a common lesion of the gastric mucosa, occurring in many chronic gastric diseases, and is typically considered a precancerous condition (Correa et al., 2010) . We retrospectively gathered 220 gastric mucosal biopsies and assessed the 4-grading results for intestinal epithelial metaplasia with two pathologists. On average, 1,320 patches were extracted per slide at 40× magnification. More details are shown in Tab. 1 and Fig. 2 . 
experiment setup and evaluation metrics: 
 We used the OTSU algorithm to create foreground masks for all WSIs, producing nonoverlapping 256 × 256 patches. For CAMELYON16, we performed a 5-fold cross-validation on the official training set and reported the average performance on the official test set. We artificially divided the BgIM dataset into an 8:2 ratio for training and testing, respectively. We report the average performance across five different sets of random seeds. Metrics used include accuracy (ACC), area under the curve (AUC), and F1-score.We used swin-tiny (Liu et al., 2021) as the backbone of the extractor. When pre-training, a projection head was added at its end to obtain a positive probability, and a batch size of 128 was employed. In the aggregation phase, the bag embedding was increased from 768 to 1024 dimensions using a fully connected layer. Because the number of focus instances was fixed at 2K (K = 512 for CAMELYON16 and K = 128 for BgIM), the mini-batch was set to 4, differing from 1 in other comparison methods. We chose SGD as the optimizer with a learning rate of 2e-4 and weight decay of 1e-5, and the loss function used was cross-entropy. All experiments were conducted on an RTX 3090. Our NcIEMIL was compared with MIL-RNN (Campanella et al., 2019) , ABMIL (Ilse et al., 2018) , CLAM (Lu et al., 2021) , DSMIL (Li et al., 2021) , TransMIL (Shao et al., 2021) , and ILRA-MIL (Xiang and Zhang, 2022) . Tab. 2 presents the results on the two datasets. For CAMELYON16, our model shows improvements of 1.24%, 1.55%, and 1.94% in ACC, AUC, and F1-score, respectively, compared to the optimal model. While in the cumulative grading task with BgIM, our model exhibits greater effectiveness, surpassing the most effective ILRA-MIL by 9.99%, 4.59%, and 14.44% in ACC, AUC, and F1-score, respectively. Additionally, we reported significance verification to indicate that our method is significantly better than the baseline methods, as shown in appendix A. Furthermore, we introduce two non-MIL methods: navie and fully-supervised approaches. The naive image-wise approach involved directly classifying downsampled WSI thumbnails using a swin-tiny network, akin to the approach in (Chen et al., 2021a) . The fully-supervised approach used a batch of patches with real labels to train a patch-level swin-tiny network. The results of the slides were obtained by the mean value of all instances. The results show that the proposed method captures slice image details and performs much better than the naive method, but is slightly inferior to the fully supervised method due to the extractor's lack of factual knowledge. To demonstrate the accuracy of the discriminative instance selection, we also performed the corresponding visualizations, as shown in appendix B.Ablation results are presented in Tab. 3. Initially, we validated the effectiveness of channel attention by removing the blue box in Fig. 1(c ). The results demonstrate that, although modest, channel attention does enhance performance. Subsequently, we modified the way of selecting focus instances, replacing bi-directional sampling with single-directional sampling and random sampling (single sampling and random sampling in Tab. 3). The results indicated that bi-directional sampling significantly outperformed the other two approaches, validating our hypotheses regarding data distribution and implicit contrasts in bag embedding. We then tested different extractors, including replacing the original one with a network pre-trained on ImageNet and pre-trained by self-supervised learning (Wang et al., 2022) . The selection of focal instances remains unchanged. Results show that task-specific relevant weakly supervised training of the extractor is effective. We also performed ablation exper- iments on K. We introduced two values for K: a small K (K = 128 for CAMELYON16 and K = 32 for BgIM) and a medium K (K = 288 for CAMELYON16 and K = 72 for BgIM), and assessed their impacts on the results. The results show that a larger K has more advantages but is limited by the slide area. 
conclusion: 
 In this paper, we reconsider the decoupled MIL framework and assess the noise sources.To reduce information redundancy, we reshape the focal instance selection by employing weakly supervised training extractors and then create a hybrid attention-based aggregator. We collected a gastric mucosal biopsy dataset, BgIM, to validate the method's effectiveness.Extensive experiments on CAMELYON16 and BgIM demonstrate that our method's performance is comparable to the state-of-the-art. However, the approach still has shortcomings.For instance, the weakly supervised training of aggregators tends to bring positive instances with different semantic information closer in the feature space. Therefore, optimizing this process is a consideration for future work. metaplasia grade 1, the projection head incorrectly assumed that negative mucosal images were positive. However, it still retrieved all patches where intestinal metaplasia actually occurred. In contrast, the projection head's judgment was more accurate for slides with intestinal metaplasia grades 2 and 3. 
 
 Appendix:  


Decision: The paper presents a multiple instance learning (MIL) based approach for whole slide image (WSI) classification in pathological diagnosis, introducing a novel framework called Non-crucial Information Elimination-based MIL (NcIEMIL). The proposed method involves feature extraction, selection based on probability score, and aggregation using spatial and channel attention mechanisms to predict image labels. Results on two datasets demonstrate improved performance compared to existing MIL approaches.

Overall, the paper presents a promising approach for WSI classification using MIL, with improvements in feature selection and aggregation mechanisms. Addressing the weaknesses mentioned by reviewers will strengthen the paper's contribution to the field.

Reviews: review: paper_summary: paper_summary: The paper presents a multiple instance learning based WSI classification method. Overall pipeline consists of:
+ feature extraction
+ feature selection based on probability score
+ feature aggregation based on spatial and channel attention mechanism to predict the level of the image

Results are provided on two datasets, and the proposed MIL approach performs better the compared MIL approaches.
main_review: main_review: The authors have addressed the previous, and have include all the required results (such as comparison to non-MIL based methods and ablation study of the k). I retain the previous rating, favouring the acceptance.
strength_weakness: strength_weakness: Writing:
+ The writing can be improved in the *methodology* section
     + *Figure-1* should have a more elaborate caption (providing a more detailed description)
     + $e_s$ (embeddings), $\textbf{u}$, and $\textbf{v}$ should be explained in detail and how are they derived
     + The dimensions of the above should also be included for better readability

Results:
 + The results with non-MIL based approach should also be included
+  The ablation study for $k$ can also be included
questions: questions: Please refer to the *weaknesses* section
paper_summary: paper_summary: The paper proposes a novel framework called Non-crucial Information Elimination-based MIL (NcIEMIL) to improve whole slide image (WSI) classification in pathological diagnosis, particularly focusing on histopathological slide examination.
main_review: main_review: The paper generally is of good quality. I still have doubts about the added benefits of the proposed changes. The authors addressed some of my comments. I modified my ranking from Borderline to Borderline accept.
strength_weakness: strength_weakness: 1. Normally the instance-level feature concatenation is not an issue an issue:
1.1. Either information about which areas are relevant is available priorly and filtered before the feature extractor.
1.2. If the irrelevant instances are forwarded into the aggregator the attention mechanism can filter these out by learned weights without a major computational effort. 
2. Scores of baselines on Camelyon16 are much lower than reported in original papers and also other methods that used these frameworks. Why?
questions: questions: See weaknesses and detailed comments.
paper_summary: paper_summary: The authors tackle the whole slide image classfication task using using a multiple instance learning framework, and propose several modifications to current state-of-the-art methods to resolve some of their limitations. They provide image-level label at the feature selection stage, overcoming the limits of fully- or semi- decoupled methods. They also carefully design a way to better select and and aggregate the most relevant feature for slice classification. This allows them to outperform state-of-the-art methods on a publicly avaible dataset and on a private dataset.
main_review: main_review: I would tend to recommand accepting the paper although I am still not totally sure about the rating I give to this work. On the one hand, the method follows classical MIL workflow (feature  extractor then classification) but on the other hand the design choices gives better results than several baselines and the model is evaluated on a public dataset. The paper is overall of good quality and the authors have address the concerns that have been raised with care and details.
strength_weakness: strength_weakness: - It's not easy to understand the differences between the model proposed in this work and the one upon which it is based (Campanella et al., 2019). 

- The comparison study with state-of-the-art method may benefit from including the reference method of (Campanella et al., 2019).

- The analysis of results, discussion and conclusions paragraphs are quite short and would benefit from being more complete.
questions: questions: - The authors are invited to make more explicit the differences between their model and the one of (Campanella et al., 2019). This would help clarify whether their paper contains significant methodological novelties, or whether it is an application of method of (Campanella et al., 2019) to a new task/database.

- If the proposed model is not exactly the same as (Campanella et al., 2019), why not include it in the methods compared in Table 1 ? This would allow to understand the value of the additions that are made compared to (Campanella et al., 2019) (if there are any).

- The performance of the proposed model is systematically superior to the baselines. However, standard errors are also quite high: have you performed statistical tests to validate that the increase in performance of your model is indeed significant ?

Comments: title: Reference
Reference
comment: **Reference**:

[1] Yan, Renao, et al. "Shapley Values-enabled Progressive Pseudo Bag Augmentation for Whole Slide Image Classification." arv preprint arv:2312.05490 (2023).
**Reference:**

[1] Campanella, Gabriele, et al. "Clinical-grade computational pathology using weakly supervised deep learning on whole slide images." Nature medicine 25.8 (2019): 1301-1309.

[2]Wang, **yue, et al. "Transformer-based unsupervised contrastive learning for histopathological image classification." Medical image analysis 81 (2022): 102559.
Thank you for recognizing our work and for your valuable suggestions. And we hope the following statements address your concerns (As the image cannot be compiled in openreview, please see <https://github.com/polyethylene16/NcIEMIL/blob/main/rebuttal/wvfJ.pdf> for details) :

- **Differences between MIL-RNN [1] and our work.** Our work differs significantly from MIL-RNN. Firstly, our approach stems from addressing informational redundancies within embedding-level methods and aims to improve them. In contrast, MIL-RNN primarily focuses on instance-level methods, where techniques like RNN and random forest serve as improvements over basic aggregation functions. Secondly, while MIL-RNN primarily deals with binary classification tasks in pathology, the BgIM dataset we collected is quadruple classified. Thirdly, although both our work and MIL-RNN involve pre-training the feature aggregator, a key distinction lies in strategy during training. MIL-RNN randomly samples instances within a bag, whereas we utilize all instances, leading to a longer inference but avoiding information loss. Fourthly, our aggregator structure represents a significant innovation in itself. Finally, in terms of instance filtering, MIL-RNN selects an equal number of discriminative instances as the number of recurrences, which is influenced by the mechanism of RNN. In contrast, we adopt a different approach by filtering a larger number of non-focal instances to create bag embeddings through bidirectional sampling. To illustrate this, based on your suggestion, we appended the MIL-RNN results on both datasets as shown in the table below. And NcIEMIL proved to be more effective.

| Method | CAMELYON16 ACC | CAMELYON16 AUC | CAMELYON16 F1-Score | BgIM ACC | BgIM AUC | BgIM F1-Score |
|--------|----------------|----------------|---------------------|----------|----------|---------------|
| MIL-RNN| $82.64_{2.17}$ | $85.02_{2.73}$ | $80.64_{1.94}$      | $53.81_{3.56}$ | $80.19_{1.86}$ | $48.55_{2.81}$ |
| NcIEMIL| $\mathbf{86.05_{1.55}}$ | $\mathbf{89.68_{2.10}}$ | $\mathbf{85.26_{1.54}}$ | $\mathbf{85.23_{0.95}}$ | $\mathbf{95.87_{0.60}}$ | $\mathbf{81.20_{0.94}}$ |

- **Significance verification.** We reported the p-values of two-sample t-tests on ACC, AUC, and F1-Score between our method and baselines respectively. $p < 0.05$ indicates that our method is significantly better than the baseline method.

| Method | CAMELYON16 ACC | CAMELYON16 AUC | CAMELYON16 F1-Score | BgIM ACC | BgIM AUC | BgIM F1-Score |
|--------|----------------|----------------|---------------------|----------|----------|---------------|
| MIL-RNN| $0.0086$ | $0.0194$ | $0.0006$ | $5.25e-5$ | $0.0001$ | $1.91e-5$ |
| ABMIL  | $0.0011$ | $0.0060$ | $0.0017$ | $8.51e-7$ | $7.48e-5$ | $1.95e-7$ |
| CLAM-MB| $0.0318$ | $0.0089$ | $0.0193$ | $0.0008$ | $0.0008$ | $3.09e-5$ |
| DSMIL  | $0.0086$ | $0.0059$ | $0.0015$ | $4.92e-5$ | $0.0001$ | $4.63e-5$ |
| TransMIL| $0.1091$ | $0.1874$ | $0.0451$ | $0.0010$ | $0.0003$ | $0.0005$ |
| ILRA-MIL| $0.1893$ | $0.0581$ | $0.1191$ | $0.0002$ | $0.0012$ | $0.0002$ |

- **Extra extractor.** Based on your suggestion, we report the effect of self-supervised training [2] of swin-tiny as a feature extractor as follows:

| Ablation item | CAMELYON16 ACC | CAMELYON16 AUC | CAMELYON16 F1-score | BgIM ACC | BgIM AUC | BgIM F1-score |
|---------------|-----------------|-----------------|----------------------|----------|----------|---------------|
| w/ ImageNet weight  | $85.12_{1.58}$ | $87.02_{3.87}$ | $84.13_{1.22}$ | $80.00_{1.90}$ | $92.49_{0.97}$ | $70.41_{1.05}$ |
| w/ ctranspath weight| $85.73_{1.69}$ | $88.45_{1.61}$ | $84.84_{1.64}$ | $77.62_{4.15}$ | $92.70_{0.66}$ | $70.16_{5.87}$ |
| NcIEMIL               | $\mathbf{86.05_{1.55}}$ | $\mathbf{89.68_{2.10}}$ | $\mathbf{85.26_{1.54}}$ | $\mathbf{85.23_{0.95}}$ | $\mathbf{95.87_{0.60}}$ | $\mathbf{81.20_{0.94}}$ |

- **Time-consumption of pre-training.** We completely agree with your perspective. Indeed, the pre-training process of the feature extractor does require some time, falling somewhere between directly loading ImageNet weights and engaging in self-supervised learning. The feature extractor typically converges within 80 to 120 epochs. However, there are ways to address this drawback. For example, one approach could involve randomly sampling within the negative bags while keeping the positive bags untouched. Additionally, performing distributed computation could also help alleviate the time consumption.

- **Visualization.** According to your suggestion, we present the top-3 instances and the negative top-3 instances from the 4-grades slides of BgIM and normal, micro-, and macro-metastasis slides of CAMELYON16, respectively. This section will be included in the appendix.

Thanks again for your comments, and we will adjust the wording of section 2.2, although there is little relevant research on instance filtering. All additional experimental results will be added to the paper.
Thank you for your invaluable feedback and comments. We are sorry that the lack of specificity in the writing confused you. We hope the following statements address your concerns (As the image cannot be compiled in openreview, please see <https://github.com/polyethylene16/NcIEMIL/blob/main/rebuttal/9snc.pdf> for details) :

- **Filtering of irrelevant instances and upper bounds on the capacity of the attention mechanism.** For issue 1.1: Before feature extraction, our method is consistent with other baseline methods, which all use the OTSU algorithm to remove background patches including blank areas, fat areas, etc. While filtering of instance-level features occurs between extraction and aggregation. Conventional embedding-level methods extract high-dimensional features from patches, which will all be considered by the aggregator. However, our proposed method first sorts all patches using positive scores, and then filters out the top k ones (regarded as potential positive instances) and the negative top k ones (which are regarded as potential negative instances) as input to the aggregator, which helps reduce the noisy signal brought by too many negative instances [1]. To further prove the effectiveness of our proposed method, we consider the Camelyon16 dataset, which is a binary classification task of WSI distinguishing the presence and absence of cancer, in which the WSI with cancer is regarded as a positive bag (patches with cancer are regarded as positive instances), the non-cancer WSI is regarded as a negative bag (patches without cancer are regarded as negative instances), and the attention mechanism-based methods will give each instance a positive-related importance score. However, in practice, they will mistakenly give high scores to negative instances sometimes. As shown in the figure below. We demonstrate the attention scores assigned to instances by ABMIL and the positive scores assigned to instances by our classifier (filter) in two micro-metastatic cancer slides. Negative instances are highlighted with black frames, while positive ones are marked in red. Instances are sorted in descending order from left to right by attention. For ease of observation, the attention is scaled by the softmax operator. In Test_10, three negative instances are assigned the second to fourth highest attention, but our method successfully eliminated these negative examples and correctly classified the WSI. In Test_84, negative instances containing a large number of lymphocytes received high attention in ABMIL, while positive instances were not in the high attention list, resulting in misclassification. Our method tended to eliminate these negative instances.

![Figure 1](1.pdf)

For issue 1.2: As shown in the following figure, for Test_84, irrelevant instances (negative instances) are fed into the aggregator of ABMIL, but the attention mechanism considers them relevant and leads to misclassification. We also show the results of ranking the positive scores generated according to our filter, where a large number of positive instances remain in the higher ranks despite the lower scores. These relevant instances are encapsulated in the new pseudo-bag.

![Figure 2](2.pdf)

- **Results of baseline methods.** We do not reproduce the results reported in the original article using the baseline model on the CAMELYON16 dataset for several reasons. Firstly, our experimentation involved a 5-fold cross-validation, which differed from that used in the baseline article. To enhance the credibility of our study, we provide detailed information regarding the division of the 5 folds, which can be accessed at [this link](https://github.com/polyethylene16/NcIEMIL/tree/main/fold). Secondly, despite employing the same preprocessing strategy, we did not generate the same number of patches as reported in the other articles. Finally, we did not employ any additional techniques during training; for instance, in the comments section of [this link](https://github.com/szc19990412/TransMIL/issues/25), a commenter only achieved slightly lower results by employing a trick (early-stop) not mentioned in the original article. During experiments, We directly used the training code provided by the authors and conducted with our partitioned dataset.


- **Modifications of the abstract.** We have updated the abstract to address poor readability and replace "We reconsider the task-specific irrelevance of most instances and the semantic redundancy of the instance representations" with "This exacerbates the information bottleneck, leading to instance representations in a high-rank space that contains semantic redundancy compared to the potential low-rank category space of instances. Additionally, most negative instances are also independent of the positive properties of the bag."

In response to the poor readability of this paper, we will adjust the paragraphs, including the abstract, in the final version of the paper. Thanks again for your comments!
Thank you for recognizing our work and for your valuable suggestions. And we quite agree with you, our point-to-point response is as follows (As the image cannot be compiled in openreview, please see <https://github.com/polyethylene16/NcIEMIL/blob/main/rebuttal/4Xqy.pdf> for details.) :

- **Writing improvement.** In response to your suggestion, we have described Fig. [fig1] in more detail as follows. Furthermore, exhaustive presentations of the sources and dimensions of embeddings, such as **u** and **v**, will be provided. For example, "where **u** = $\in \mathbb{R}^{2K \times d}$ denotes the bag, obtained by concatenating instance embeddings, with *d* being the embedding dimension", "**v**  $\in \mathbb{R}^{1 \times d}$ is a learnable embedding that represents a virtual instance used for classification" and "Then the aggregated **v'** is extracted from the spatial dimension of **u'**  $\in \mathbb{R}^{(1 + 2K) \times d}$. These changes will be updated in the final version.

![Overview of NcIEMIL. (a) Instances are fed into the extractor and projection head on a bag basis, and the instance with the highest probability is involved as a representative of its corresponding bag. (b) Instances within a bag are re-ranked, and bi-directional sampling is employed to generate bag embeddings. (c) A hybrid attention-based aggregator, comprising parallel spatial and channel attention mechanisms, is employed to mitigate information redundancy.](1.pdf)

- **Additional comparison experiments.** Following your suggestions, we have incorporated two additional methods into our comparative experiment. The naive image-wise approach involves directly classifying downsampled WSI thumbnails using a classification network, akin to the approach in \[1\]. The fully-supervised approach, on the other hand, uses a batch of patches with real labels to train a classification network. In CAMELYON16, we extracted positive and negative patches from within and without cancerous regions respectively, according to the pixel-level annotations provided officially. Regarding BgIM, we randomly selected 2,000 preprocessed patches and annotated them for fully supervised training. The results of the slides were obtained by the mean value of all instances. The results are presented in the following table and will be incorporated into Tab. 1 of the original paper. To be fair, swin-tiny was chosen for all the above classification networks to maintain consistency.

| Method           | CAMELYON16  ACC | CAMELYON16 AUC | CAMELYON16 F1-Score | BgIM ACC | BgIM AUC | BgIM F1-Score |
|------------------|----------------|----------------|----------------------|----------|----------|----------------|
| Naive            | $62.40_{1.42}$          | $62.16_{1.40}$          | $45.15_{5.28}$                | $52.98_{3.32}$    | $76.58_{1.28}$    | $44.27_{4.44}$          |
| Fully-supervised| $ \color{red}{91.32_{1.99}} $     | $ \color{red}{94.93_{0.87}} $     | $ \color{red}{90.73_{2.12}} $           | $ \color{red}{91.91_{1.17}} $| $ \color{red}{98.72_{0.33}} $| $ \color{red}{91.57_{2.61}} $     |
| NcIEMIL          | $\mathbf{86.05_{1.55}}$      | $\mathbf{89.68_{2.10}}$     | $\mathbf{85.26_{1.54}}$           | $\mathbf{85.23_{0.95}}$| $\mathbf{95.87_{0.60}}$| $\mathbf{81.20_{0.94}}$      |

- **Additional ablation experiments.** We also performed ablation experiments on *K*. The results are shown in the following table. Specifically, we introduced two values for *K*: a small *K* (*K*=128 for CAMELYON16 and *K*=32 for BgIM) and a medium *K* (*K*=288 for CAMELYON16 and *K*=72 for BgIM), and assessed their impacts on the results. The reason for not selecting a value larger than *K* in the original article (*K*=512 for CAMELYON16 and *K*=128 for BgIM) is because of the restricted number of instances in the smallest bag within the dataset.

| Ablation item   | CAMELYON16 ACC | CAMELYON16 AUC | CAMELYON16 F1-score | BgIM ACC | BgIM AUC | BgIM F1-score |
|-----------------|-----------------|-----------------|----------------------|----------|----------|----------------|
| w/ small *K*    | $85.75_{1.84}  $         | $\mathbf{90.06_{1.87}}$     | $84.87_{1.68}  $              | $83.33_{1.51}  $  | $95.21_{0.50}  $  | $80.09_{2.14}  $        |
| w/ medium *K*   | $85.36_{1.49}  $         | $89.46_{1.30}   $        | $84.01_{1.30}  $              | $84.28_{1.90}   $ | $94.90_{0.83}  $  | $80.00_{1.95}   $       |
| NcIEMIL         | $\mathbf{86.05_{1.55}}$      | $89.68_{2.10} $          | $\mathbf{85.26_{1.54}}$           | $\mathbf{85.23_{0.95}}$| $\mathbf{95.87_{0.60}}$| $\mathbf{81.20_{0.94}}$      |

[1] Chen, Chi-Long, et al. "An annotation-free whole-slide training approach to pathological classification of lung cancer types using deep learning." Nature communications 12.1 (2021): 1193.