How Effective Can Dropout Be in Multiple Instance Learning ?
Authors: Wenhui Zhu
Peijie Qiu
Xiwen Chen
Zhangsihao Yang
Aristeidis Sotiras
Abolfazl Razi
Yalin Wang
Venue: ICML 2025 poster (2025)
Keywords: Multiple Instance Learning
Dropout
Optimization
Whole Slide Image
Computer Vision
Primary area: deep_learning
TL;DR: The MIL methods need a specific Dropout.

Abstract: Multiple Instance Learning (MIL) is a popular weakly-supervised method for various applications, with a particular interest in histological whole slide image (WSI) classification. Due to the gigapixel resolution of WSI, applications of MIL in WSI typically necessitate a two-stage training scheme: first, extract features from the pre-trained backbone and then perform MIL aggregation. However, it is well-known that this suboptimal training scheme suffers from "noisy" feature embeddings from the backbone and inherent weak supervision, hindering MIL from learning rich and generalizable features. However, the most commonly used technique (i.e., dropout) for mitigating this issue has yet to be explored in MIL. In this paper, we empirically explore how effective the dropout can be in MIL. Interestingly, we observe that dropping the top-k most important instances within a bag leads to better performance and generalization even under noise attack. Based on this key observation, we propose a novel MIL-specific dropout method, termed MIL-Dropout, which systematically determines which instances to drop. Experiments on five MIL benchmark datasets and two WSI datasets demonstrate that MIL-Dropout boosts the performance of current MIL methods with a negligible computational cost. The code is available at \url{https://github.com/ChongQingNoSubway/MILDropout}.

How Effective Can Dropout Be in Multiple Instance Learning?
Wenhui Zhu
Peijie Qiu 2
Xiwen Chen Zhangsihao Yang' Aristeidis Sotiras?
Abolfazl Razi
Yalín Wang'
Abstract
Baseline
MIL-Dropout
Multiple Instance Learning (MII.) is popular weakly-supervised method for various applica- tions, with particular interest in histological whole slide image (WST) classification. Due to the gigapixel resolution of WSI, applications of MIL WSI typically necessitate two-stage training scheme: first, extract features from the pre-trained backhone and then perform MII. ag- gregation. However, it is well-known that this suboptimal training scheme suffers from "noisy" feature embeddings from the backbone and in- herent weak supervision, hindering MIL from learning rich and generalizable features. How- ever, the most commonly used technique (i.e.. dropout) for mitigating this issue has yet to be explored in MIL. In this paper, we empirically explore how effective the dropout can be in MIL. Interestingly, we observe that dropping the top- k most important instances within a bag leads to better performance and generalization even under noise attack. Based this key observa- tion, we propose a novel MIL-specific dropout method, termed MIL-Dropout, which systemat- ically determines which instances to drop. Ex- periments on five MIL. benchmark datasets and two WSI datasets demonstrate that MIL-Dropout boosts the performance of current MIL meth- ods with negligible computational cost. The code is available at https://github. com/ Chonggi rgNesuubwey/MTTDDooooo
Sharp Minimum
Flat Minimum
mini-balch gradieut
hole-dalaset ground true gradient
gradient
Figure I. An illustrative example comparing the convergence tra- jectories of the baseline ABMIL without dropout (Left) and AB- MIL with the proposed MIL-Dropouu (Right). ABMIL without dropout likely follow incorrect gradient direction initially and eventually converge to a sharp minimum. In contrast, ABMIL with the proposed MIL--ropout typically achieves lower gra- dient direction error and reaches flatter minimum with better generalization.
bag), making it a weakly- classification method due the absence of instance-level labels. MIL has been ap- plied to various fields (Zhu al., 2023b; Chen et al., 2024b), including digital pathology (Ilse al., 2018; Li et al., 2021; Shao et al., 2021), video analysis (Babenko et al., 2010; Quellec et al., 2017), and time series classification (Farly et al., 2023; Chen et al., 2024b). In particular, MIL is a defacto standard for histological whole slide image (WSI) classification in digital pathology. This because WSIs are gigapixel images, making them challenging process using traditional machine learning deep learning meth- ods. A WSI is treated as a bag comprising collection of smaller tiles, each regarded as an instance. The MIL model takes a collection of WSI tiles as input and then assigns a single label to an entire WSI (e.g., malignant). Due to a large number of instances within a bag, deep learning based MIL methods (Wang al., 2018b; Ilse al., 2018; Shao et al., 2021; Zhang et al., 2022: Zhu et al., 2024; Qiu et al., 2023) typically necessitate two-stage learning scheme, where features are first extracted by a pre-trained backbone and then combined by an MIL aggregator. However, this suboptimal training scheme suffers from "noisy" feature embeddings, typically resulting reduced performance.
1. Introduction
Multiple instance learning (MIL) has gained significant at- tention in machine learning applications. MIL assigns single label (bag-level label) to collection of instances (a
Equal contribution 'Arizona State University. USA. Washington University in Louis, USA. *Clemson University, USA.. Correspondence to: Wenhui Zhu @ww.u55@@su.cuu>>
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

How Effective Can Dropout Be in Multiple Instance Learning? ?
Follow-up methods also argue that this issue is attributed to overfitting and the inability of the MII. aggregator learn rich representation in such a suboptimal learning scheme (Li et al., 2021). Although it is not formally discussed, (Li et al., 2021) randomly drops input instances (i.e., DropInstance in this paper) in their implementation, which partially evi- dences the effectiveness of this simple dropout-like method in regularizing MIL. Motivated by this observation, we ar- gue that formal and thorough investigation of the role of the dropout method in MIL is necessary.
demonstrated that bag-level MIL methods generally outper- form their instance-level counterparts (Shao et al., 2021; Wang et al., 2018b). Recent advancements in bag-level MIL methods have primarily focused on improving instance-level MIL. aggregation. This has been achieved through the incor- poration of attention mechanisms (IIse et al., 2018), trans- formers (Shao et al., 2021; Xiang & Zhang, 2023), pseudo bag (Zhang et al.. 2022), and non-local attention (Li et al., 2021) effectively capture correlations between instances. In addition, some studies have begun focusing on negative mining within instances, using masked approaches to un- cover more positive instances and incorporating a complex teacher-student model (Tang et al., 2023). However, the rationale behind masking instances remains unclear, and these methods cannot be casily generalized a wider range of MIL tasks. In contrast, we focus on demonstrating the ef- fectiveness of dropout in regularizing MIL to achieve better performance and generalization. Instead of enhancing MIL. aggregation, we focus on integrating the proposed a general MIL-Dropout into various MIL aggregators to boost their performance.
For this purpose, we first conduct experiments on various dropout strategies in MIL. Our empirical investigation in- dicates that dropping the top-k most important instances (i.e., top-k DropInstance) leads to an increase in classifica- tion accuracy. At first glance, this appears counterintuitive, whereas an in-depth analysis reveals its theoretically in- triguing properties. First, Different from findings in other deep learning models (Liu et al., 2023; Jastrzebski et al., 2020; Kavis et al., 2022), dropping top-k most important instances can help MIL reduce the gradient direction error until converging (see example in Fig. 1 and evidence in Fig. 4). Second, it encourages the MIL model to converge to flatter minimum, which enhances generalizability (e.g.- robustness to noise attack). Despite its simplicity, dropping important instances not only results in performance gain but also shows theoretical guarantees. However, determining the importance of instances and selecting which ones to drop remains an open question.
Dropout methods. Dropout has been empirically shown to be an effective technique for mitigating overfitting in variety of computer vision studies (Hinton et al., 2012; Srivastava et al., 2014). Numerous follow-up studies have extended the concept of dropout, exploring a wide range of methodologies. Notable examples include spatial- dropout (Tompson et al., 2015), dropout based on con- tiguous regions (Ghiasi et al., 2018), and attention-based dropout (Choe & Shim, 2019). However, the application of dropout in MIL has not been explored. In the context of MII, where the MIL aggregator operates on instance embed- ding features devoid of spatial context, the dropouts based on spatial context are not applicable. Despite sharing simi- larities with the attention- dropout, our MIL-Dropout is tailored to MII. applications, which drops instances in- stead of neurons in other applications (e.g., natural image classification). It is worth noting that this paper also serves as a theoretical supplement to PDL (Zhu et al., 2023a).
Leveraging the advantageous properties of top-k DropIn- stance, we propose novel MIL--rropout method that sys- tematically determines the importance of instances and iden- tifies which ones to drop. This approach helps reduce gra- dient direction errors and achieves better generalization by converging to flatter local minima (see example in Fig (Right) and evidence in Fig. 3). Our method involves two key components: (i) a novel everaging-based attention mech- anism to efficiently determine instance importance and (ii) query-based instance selection mechanism to select the instance set to drop. Extensive experiments on various MII. benchmarks demonstrate that our method can be seamlessly integrated into existing MIL frameworks to effectively im- prove their performance with negligible computational cost.
3. Preliminary.
The MIL is usually treated as a binary classification prob- lem, the goal is to learn a direct mapping from a bag of N instances X {ar,, Y E 10. In most real scenarios, the instance-level labels {un n 1..., ,N are unknown, making it weakly- supervised problem:
2. Related Work
1, .N to a binary label
MIL. Methods. The introduction of MI-Net (Wang et al., 2018b) has significantly elevated the prominence of bag- level MIL methods, which utilize only bag-level labels for supervision. This approach addresses the ambiguity asso- ciated with propagating bag-level labels to individual in- stances inherent in instance-level MIL methods (Feng & Zhou, 2017; Hou al., 2016; Xu et al., 2019). Particularly in WSI classification, empirical studies have consistently
0, iff Σn =
Y
(I)
1, otherwise.
We considered embedding- based MII. as an example due
2

---

How Effective Can Dropout Be in Multiple Instance Learning
sharp
3.1. Analysis of Dropout in MIL
1.00
-0.30
In an MIL framework, Dropout is typically applied the shallow feature extractor fa. For simplicity, considering fa an MLP with L layers, the feature map at the I-th layer of fo is a 2-dimensional matrix f E R Nx where D() represents the embedding dimension. In this scenario, consider randomly zeroing out either entries (Hinton et al., 2012) or entire instances (Tompson et al., 2015) in f For brevity, we denote the former as DropNeuron and the latter as Dropinstance. For a given rate p E [o, 1], both DropNeuron and DropInstance can be defined as performing an element-wise masking operation over the feature map f(() at the 1-th layer of fa:
(5 4p 4
Loss fronston Ex
--0.9-
0.25
0.00
-0.50
-0.22
-0.75
-0.23
-1.0
-0.5
0.0
0.5
1.0
1.000
-100
-0.5
0.0
0.5
1.0
Figure An conceptual illustration for a and sharp minimum in ID curvature (Left) and 2D landscape (Right) of the loss func-
tion Cg.
f() M
(4)
to its popularity. The standard workflow of an embedding- based MIL involves (i) projecting instances into feature em- beddings via an instance-level feature extractor and (ii) ag- gregating the instance-level features into bag-level predic- tion through an MIL aggregator. Specifically, the instance- level feature extractor typically consists of pretrained backbone feature extractor frackbone (e.g., ResNet) and a trainable shallow feature extractor So parameterized by (e.g., multi-layer perception), due to the intractability to optimize the backbone (Li al., 2021). The input in- stances X is projected into D-dimensional feature vectors 1.... N) E ND by applying the instance-level feature extractors: V fo faackccoo (X)). The MIL aggregator is then applied to the instance-lever feature embeddings to obtain the bag-level probabilistic prediction Y E [0,
where denotes element wise multiplication, and M R D is the binary Dropout mask at the I-th layer. the regime of DropNeuron, each entry M in M are from Bernoulli distribution: M Bernoulli(p). In con- trast, each row in M has the same entry and is sampled from a Bernoulli distribution in the case of DropInstance: M Bernoullipp) However, how and when impose Dropout remains open question in the context of MIL, which has yet to be thoroughly explored by previous MIL works. In this section, we provide insight into these two aspects by conducting empirical and theoretical analyses in the relatively complex WSI classification (e.g. Came- Iyon using representative ABMIL framework. Please refer Appendix A for the details of these investigation experiments. Our investigation reveals two main findings: (i) DropInsiance leads to flatter local minima and better gen- eralizability compared with DropNeuron, and (ii) DropIn- stance helps reduce gradient direction errors in learning trajectory, improving data fitting and performance.
D
V {ua,
Ý = fu Ln NN)..
where f.) a bag-level classifier parameterized by w, MIL pooling function parameterized by vb The learning of an MIL can be achieved by optimizing the binary cross-entropy loss over trainable parameters {8, w.
and Pv: is sermutation-invarian
3.2. DropNeuron vs DropInstance.
A common way to assess the effectiveness Dropout is to examine the robustness an MIL after applying Dropout. For this purpose, we compute the sharpness of MIL model (Dinh al., 2017), which pertains to how sensi- tive the model's performance is changes in the model parameters. We consider the e-sharpness (Dinh et al., 2017) as the sharpness measure, which is defined using the local curvature of the loss function. Let B2 0) be an Euclidean ball centered on a minimum 0 with radius e. Then, for a non-negative valued loss function Lo, the e-sharpness proportional to the following quantity:
Without loss of generality, we consider attention-based based MIL pooling (ABMIL) (Ilse et al., 2018)<< as an example. This is because most embedding-baasd
MIL models (Zhang et al., 2022; Shao al., 2021) fall within the regime of ABMIL. Mathematically, ABMIL pooling function is defined as
N
Pe 1,... NJ) =
Σ On Un-
(3)
with Oz = =oftmax(u1) tanh(wz2o?))
max C.0ss0'B2tll (B2le.0) -L
e-sharpness x
(5)
1+C Cs
where ab = {wi E Dx1, rameter, and on implies the importance of the n-th instance.
W2 E DxL
/ is the trainable pa-
Geometrically, the e-sharpness is proportional to the height of the area enclosed by the curvature of Co in Fig. 2(eef)).
3

---

How Effective Can Dropout Be in Multiple Instance Learning
0.70
2.95
AUC
0.90
Baseline (n-tha-neeen-..0 010)
-1-
Drop Neuron je-muppsssse..00 026:
Dropiiiance j=siiappsss--0000
Gausslan
Poisson
Pepper
(a) Lo Landscape [DropNeuron]
(b) La Landacape [Dropinstance]]
(c)
Peformance against noise attacking
Figure 3. The landscape of the loss function Lo for different dropout strategies (a) DropNeuron and (b) DropInstance well (c)
the performance of MIL models against different noise attacks. We mark the Euclidean ball (2 (e,, around the optimal parameter 6"
(see Eq. 5) in subpanel figure (a) and (b) with a red circle. We observe that the landscape of Co in the Dropinstance scenario leads flatter minima compared to DropNeuron, which also results in better performance in AUC.
0.90
Gradient Direction Error (GDE)
Base Inc |Arca under GDE = 4119]
Drop Random Instances (Ares under GDE 4737
0.54
Drop Buttom-k irstances (Area under GDE 4324]
Top instanoes under
0.88
0.52
0.1
0.88
0.48
AUC
-0.0
Loss
0.46
-011
0.82
0.42
-022
0.40
heration=1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
0.80
0.3
Figure 4. The comparison of change of GDE (Left) over the first 10,000 iterations well performance and loss (line plot) and AUC (bar plot) when using different instance dropout strategies (Right), where the area under GDE is the area enclosed by GDE and the x-axis. Dropping the top-k instances shows the smallest GDE, training loss and highest AUC among all four strategies.
In the 2D landscape La in Fig. 2(ighh,), the loss function Co changes more rapidly for sharper minima. As suggested by (Keskar al., 2016; Foret al., 2021; Zhang & Xu, smaller volume enclosed by the
sharpness analysis of the MII. model (see Fig. (b)), where DropInstance leads to flatter local minima that typically have better generalizability. We conjecture this may be be- cause Dropinstance is similar to data augmentation, which generates random instance combinations to improve the per- formance. However, this observation prompts a question: Are all instances worth being dropped out, or the improve- ment due to specific combinations after Droplnstances?
2024), for specific curvature of Co typically indicates flatter local minima (see Fig. 2), which leads to better generalizability. By applying the second-order Taylor approximation, e-sharpness in Eq. 5 can be rewritten as
Col22 e-sharpness 2(1 + Le)
(6)
3.3. How to impose Dropout.
which enables us to directly compute the c-sharpness. We assess the robustness of the trained MIL model against three different types of noise (i.e., Gaussian, Poisson, and Salt Pepper noise) by attacking the input tiles/patches with them. The implementation of this experiment is detailed in Appendix A. We observe that adding Dropout lowers the e-sharpness value and results in flatter minima (Fig. 3 (a) and (b)), which also improves the bag-level classification performance (Fig. 3 (c)). In particular, adding DropInstance leads to the best performance. This is also supported by the
Here, we further investigate how to apply DropInstance. Previous studies have revealed that the effectiveness al- gorithms modules (e.g.eeeeee.pppppp.o.o (roppout can be reflected by the gradient direction error (GDE) gradient variance during model optimization (Liu et al., 2023; Jastrzebski et al., 2020; Kavis al., 2022). The gradient direction error quantifies the dissimilarity between the mini-batch gradient getep and whole dataset gradient g:

---

How Effective Can Dropout Be in Multiple Instance Learning? ?
well instances similar them. The most important step revolves around selecting these instances in a systematic way. In a previous investigation, the attention map, which indicates the importance of each instance, is obtained from Eq. (I1). However, we argue that a key issue with this attention map is that it is produced by previous optimiza- tion iteration. Due to the instability of MIL training under weak supervision. top-k instance selection based on this attention map may introduce additional noise, especially the very beginning of the training. In addition, the top-k most important instances at different layers of the feature extractor fe can potentially be different. To mitigate this issue, we propose a non-parametric attention mechanism to select the top-k instances for dropout. Specifically, we note that various methods for natural image processing employ average pooling to compress channel information and pro- duce an attention map (Woo et al., 2018; Park et al., 2018). Inspired by this, we extend average pooling to our method to determine the importance of instances at the I-th layer of the feature extractor fo, i.e., On on
GDE
1
Σ
1 2
(gtepp 9) (1 2
G
1 l12 |l0lla
Sup
where G is a set of mini-batch gradients. Leveraging GDE, we investigate the impact of three different DropIn- stance strategies, including dropping (i) top-k instances, (ii) bottom-k instances, and (iii) random instances. Here, top/bottom-k k refers to dropping top/bottom & most impor- tant instances based on the importance of instances (i.e., at- tention scores in any attention based MIL framework) sorted in descending order. For this purpose, we plot the change of GDE as function of training iterations. As shown in Fig. 4 (Left), the GDE of standard MIL does not decrease during model learning. Likewise, the GDE of the MIL models that apply random and bottom-k instance dropout even increases across training iterations. This suggests that the parameter updates are not moving in the correct direc- tion, potentially compromising the fitting of the MII. model. Among the four aforementioned DropInstance strategies, only dropping the top-k instances leads decreased GDE by model training. Further comparison of the aforemen- tioned four DropInstance strategies in terms of classification performance suggests that a smaller GDE typically leads to better classification performance and lower training loss as shown in Fig. 4 (Right).
(l)
a sigmoid ((argpool( n = L, .N))
D
1
with avgpool(f
(7)
D
Here, the average pooling is applied to each instance at em- hedding dimension D)I) after passing a sigmoid activation function to obtain the importance weight of each instance. (I) Based on the instance importance on we first select the top-k most important instances and then group the embed- ding features f()) at the I-th layer of fo into two groups: (i) top-k ( and (ii) remaining feature f(1).
3.4. Explanation.
In summary, empirical results show that dropping the top-k most important instances typically leads to better perfor- mance and gradient error direction. In the context of MIL, positive bag defined by the presence of at least one positive instance. However, in practice, the positive instance within each bag diversity leads many models to learn a single set of highly similar positive instances to make deci- sions. Dropping top-k instances deactivates these learned discriminative instances (zeroing them out in the feature map). With the bag label remaining unchanged, this forces the network to seek additional positive instance representa- tions. From a gradient perspective, mini-batch gradients are generated by different sub-networks that are closer to the whole dataset gradient. This indicates that dropping these top-k instances allows the network to continue learning ef- fectively and enhances performance, explaining why the top-k strategy results in lower gradient error.
We argue that simply dropping the top-k instances is in- sufficient. This is because instances within bag may not be independent and identically distributed (i.i.d)) in most real- world applications (Tu et al., 2019a; Ilse et al., 201 8). This is particularly true for WSI classification, where pathol- ogists consider both the contextual information around a single area and the correlation information between differ- ent areas when making decisions. Based on these facts, we also drop instances that are highly similar to the selected top-k instances to further encourage the model to learn di- verse representations. For this purpose, we propose a query mechanism to select Gi number of instances that is highly similar to the selected top-k instances from the remaining features f (1) RR based on their similarity:
4. Proposed MIL-Dropout
Contingent on the above discovery, we introduce a sim- ple yet effective dropout method (termed MIL-Dropout), which can be easily plugged into existing MIL frameworks. MIL-Dropout mainly leverages the instance-basee dropout operation to drop the top-k most important instances, as
Sr = I I 1
vi [K].EE - K]. (8)
where Si denote the similarity score between p4: and
f R. features, with higher St,, indicating higher similarity.
5

---

How Effective Can Dropout Be in Multiple Instance Learning ?
The top-k instances are then combined with their K G similar instances form the final dropout instance set in- dexed by A, containing NA instances. Then the remain- ing instances are indexed by A, containing Na instances. Following the convention in Eq. (4), DrI.-Dropout can be achieved by masking out the selected instance set A:
Algorithm 1 MIL-Drropout Mechanism
Input: Input feature map f = (0),-.. UND, K and G
Output: Processed Bag f(i) with MIL-Dropout
I: Initial: M <- kxDD (# Initial mask)
(fT (T) (0)
2: Select top-k important instances:
split()()), A [K] (Eq. 7)
_-= = x O M)
(9)
Compute the similarity matrix between rest instances
with MA 0 and M,, A = 1.
1 and top-K instances 5
(# Obtain G instances from f that are most similar
Here, we add normalization term 7 to stabilize the training. Note that each top-k instance may have duplicate similar instances, necessitating deduplica- tion. Therefore, the final number of dropout instances is typically less than K + (K G). Similar other dropout methods (Srivastava et al., 2014; Hinton et al., 2012), we do not apply MIL- MIL-Dropout during inference. The algorithmic taxonomy of the MII--Dropout is shown in Algorithm 1.
every lop-K instance)
4: for i to K do
5:
A = arg MaX L Sig (Eq. 8)
6: A + AUA,
7: end for
8: M [A :] =
9: pdd t -((())
o MM (Eq. 9)
10: return fill) (# Masking and normalization)
4.0.1. COMPLEXITY ANALYSIS.
Our MrI--rroouut only adds additional two hyperparame- ters (i.e., top-k number K and similarity instance number G), without involving additional learnable parameters. In each iteration, the main overhead is computing the mask, which is presented as,
270 samples and a testing set of 129 samples. The TCGA- NSCLC dataset primarily identifies two subtypes of lung cancer: lung squamous cell carcinoma and lung adenocar- cinoma. As outlined in (Li et al., 2021), I), 1037 WSIs were divided into a training set of 744 WSIs, a validation set of 83 WSIs, and testing set of 210 WSIs. Following the threshold (Li et al., 2021) and OTSU (Zhang et al., 2022) preprocessing methods, each WSI was divided into non-overlapping 224 x 224 patches at a magnification of x20. Two sets of patches were extracted using different frameworks: ResNet-50 from DTFD-MIL, yielding 1024- dimensional vectors per patch, and the SimCL contrastive learning framework from DSMIL,, yielding 5 312-dimensional vectors per patch.
O(N Oog N) + ( + OOK(N - K)) N)
Top-k Sorting
Compute Similarity
Subsequent Sorting
The complexity is substantially reduced to the fact that O(ND()) due to K is typically much smaller than N, c.g.,
K- = 20, whereas N = 5000 ~ 10000.
5. Experimental designs
5.1. MIL benchmarks.
The benchmark datasets include MUSK1, MUSK2, FOX, TIGER, and ELEPHANT, which are commonly used to evaluate and compare the performance of MII. algorithms. MUSKI and MUSK2 (Dietterich et al., 1997) focus on molecule classification, with each bag containing instances representing atoms. Conversely, FOX, TIGER, and ELE- PHANT (Andrews et al. 2002) involve image classification, where each bag represents images and contains instances that represent patches within those images. As consistent with the experimental protocols outlined in (Li et al., 2021), all experiments are conducted five times with a 10-fold cross-validation. We report the mean (+t std) for all MIL benchmark datasets.
5.3. Baselines.
On five MII. benchmark datasets, we mainly plug MII.- Dropout into ABMIL as proof of concept, while com- paring with other baseline methods: mi-Net, MI-Net, MI- Net with DS, MI-Net wth RC (Wang et al., 2018b) AB- MIL, ABMIL- (Ilse et al., 2018), GNN-MIL (Tu et al., 2019b), DP-MINN (Yan et al., 8), NLMIL (Wang et al., 2018a), ANLMIL (Zhu et al., 2019), and DSMIL (Li et al., 2021). On the WSI datasets, we plug MII MIL--ropout into four state-of-the-an MIL aggregators and their variants to valid the effectiveness of the proposed method, i.e., ABMIL (Ilse et al., 2018), DSMIL (Li et al., 2021), TransMIL (Shao et al., 2021), and DTFD-MIL (Zhang et al., 2022).
5.2. WSI datasets.
5.4. Evaluation metrics and implementation details.
The CAMELOYON'S dataset aims to identify metastatic breast cancer in lymph node tissue and consists of high- resolution digital WSIs. It is divided into a training set of
For the MIL benchmark dataset, we performed five times with a 10-fold valodaval for each method and reported
6

---

How Effective Can Dropout Be in Multiple Instance Learning? ?
Table 1. Performance comparison on MII. henchmark datasets. Each experiment is performed five times with 10-fold cross-validation. We reported the mean of the classification accuracy (# the standard deviation of the mean).
Methods
MUSK1
MUSK2
FOX
TIGER
ELEPIANT
mi-Net
0.889 + 0.039
0.858 = 0.049
0.613 + 0.035
0.824 = 0.034
0.858 = 0.037
MI-Net
0.887 1 0.041
0,859 + 0.046
0.622 0.038
0.830 0.032
0.862 = 0.034
MI-Net with DS
0.894 + 0.042
0.874 + 0.043
0.630 0.037
0.845 0.039
0.872 = 0.032
MI-Net with RC
0.898 = 0.043
0.873 = 0.044
0.619 0.047
0.836 0.037
0.857 = 0.040
ABMIL
0.892 = 0.040
0.858 = 0.048
0.615 0.043
0.839 0.022
0.868 0.022
ABMIL-Gated
0,900 + 0.050
0.863 = 0.042
0.603 0.029
0.845 0.018
0.857 = 0.027
GNN-MIL
0.917 1: 0.048
0.892 1 0.011
0.679 0.007
0.876 0.015
0.903 1 0.010
DP-MINN
0.907 1: 0.036
0,926 + 0.043
0.655 0.052
0.897 = 0.028
0.894 + 0.030
NLMIL
0.921 + 0.017
0,910 = 0.009
0.703 0.035
0.857 = 0.013
0.876 = 0.011
ANLMIL
0.912 + 0.009
0.822 = 0.084
0.643 - 0.012
0.733 = 0.068
0.883 = 0.014
DSMIL
0.932 1 0.023
0.930 1 0.020
0.729 1 0.018
0.869 + 0.008
0.925 1 0.007
ABMIL
0.964 + 0.033
0.954
0.019
0.789
0.043
0.917
0.036
0.934 = 0.046
MIL--Dooouu
ABMIL-Gated
0.967 + 0.019
0.958 + 0.021
0.788 0.016
0.919 = 0.033
0.927 + 0.033
MIL-Dropout
the mean of the classification accuracy and the standard deviation. For two WSI datasets, we report the classifica- tion mean of accuracy, F1, and AUC were reported with standard deviation based on running five times of each ex- periment without fixing random seed. All baselines were implemented with the parameter configurations specified in their original papers. To incorporate the MIL dropout, we employed a unified specific architecture involving adding three fully connected layers as the shallow feature extrac- tor fo before entering MIL aggregation. Furthermore, We reported all detailed experiment settings in Appendix B.
MIL-Dropout into current MIL aggregators CAME- LYON16 and TCGA-NSCLC WSI datasets. Experimen- tal results show that MIL-Dropout boosts the performance of four different types of MIL aggregators across these two WSI datasets with features extracted using different means. Specifically, applying MII--rooout results in aver- age improvements of 2.3% in accuracy, 2.46% in Fl score, and 2.48% in AUC, respectively, dataset. Similarly, on the TCGA-NSCLC dataset, MIL.- Dropout leads to average increases of 2.05% in accuracy, 2.44% in Fl score, and 1.80% in AUC. Remarkably, in- tegrating MIL-Dropout into simple attention- based MIL aggregators (ABMIL and ABMIL-Gated) achieve a per- formance that is comparable or even surpasses state-of- the-art methods. This shows that even ABMIL can signifi- cantly improve representation learning with MIL-Dropout, addressing underfitting and enhancing latent feature discov- ery. Interestingly, we observe that MIL aggregators trained with features extracted using SimCLR outperforms that trained with features extracted by SimCLR on the CAME- LYONI6 dataset, but shows similar performance on the TCGA-NSCLC dataset. With the proposed MIL--Drooout the performance of both ImageNet and self-supervised pre- trained backbone is largely indistigguishable under optimal conditions. On the TCGA-NSCLC dataset, both extractors result in similar performance, suggesting that more training samples improve the generalization of self-supervised fea- ture extractors. However, our MIL-Dropout can enhance the generalization of different feature extractors without an additional self-supervised training stage.
the CAMELYON16
6. Results
6.1. MIL. benchmark results.
The integration of the proposed MIL-Dropout into ABMIL and ABMIL-Gatee Icads to superior performance compared to all prior state-of-thh-ar (SOTA) methods across five MIL benchmark datasets, as shown in Table 1. Notably, ABMIL augmented with MII.-Dropout achieves SOTA accuracy of 78.9% on the FOX dataset and 93.4% on the ELEPHANT dataset, along with an average accuracy improvement of 9.72% across the five datasets compared to the baseline ABMIL. Similarly, ABMIL-Gated with Dropout reaches SOTA accuracy of 96.7% on MUSKI, 95.8% on MUSK2, and 91.9% on TIGER, demonstrating an average accuracy improvement of 9.82% across the five datasets over the baseline ABMIL-Gate. Through extensive experiments, we observe that integrating our proposed Dropout into the two simplest bags embedding MIL aggregation (ABMIL and ABMIL-Gated) can also achieve SOTA performance.
6.3. Ablation analysis.
6.2. WSI results.
We perform ablation studies two key hyperparameters in MIL-Dropout: the number of top-k instances (K) and the number of instances similar to the top-k instances (G). These analyses are carried out on the CAMELYON 16 and
As shown in Table 2, we perform a comparative study to evaluate the effectiveness of integrating the proposed
7

---

How Effective Can Dropout Be in Multiple Instance Learning ?
0.66
0.6
Annotated MSI
ABHIL wio Hil: Droporo
ABHI Hil- Mil-Dropouu
0.62
AUC
0.88
0/90
0.86
0.84
NSCLD
0.62
10
20
30
3
5
10
20
30
40
(a) Number of top-k instances (K)
(b) Number of similarity instances (G)
(c) Attention map visualization
Figure 5. Ablation studies on the number of top-k instances K (a) and similarity instance S (b) using CAMELYON 16 and TCGA-NSCLC datasets. (c) Attention map from ABMIL without and with MIL-Dropout, with tumor regions outlined in red. Brighter eyan in columns two and three indicates higher tumor probability (higher attention score) for corresponding locations.
Table Comparison of performance before and after plugging MIL-Dropout into four different types of MIL aggregators and their variants C and TCGA-NSCIL datasets. A denotes the performance gains after the integration of MII.-Dropout. The classification accuracy (%), Fl score (%), and AUC (%) are reported (# the standard deviation of the mean) by running each experiment five times.
CAMELOYONI6
TOGA-NSCLC
ImageNet Preirained
SimCLR Pretrained
ImaggeNet Peetrained
SinCLR Pretrained
Ascuracy
Accuracy
Accuracy
AUC
Accuracy
86.3
85.021.0
86.0
85.6
812113
86.6 1 1.4
87.5
87.5
9.4
87
18.1006
93.8 108
ABMIL
+MIL Dropout
87.2 1.0
86.420.8
90.1 0.8
88.6 1.1
1.4
8.2
91.1 91
1.3
05
91
915++0
9.1
+3.6
43.2
36.9
85.2=1.2
86.2 + 1.2
84.3 1.1
834+10
87.9 1.6
87.9 0.9
02
89.0 1.2
89.0 1.2
9.74
ABMIL-
+MIL Dropout
9
89.621.2
90.0
9.33
Gated
A
+3.5
+3.9
+4.6
+3.4
+4.2
+1.5
+2.1
+2.1
+2.5
+1.8
+1.8
+1.4
85.5+
84.3=
82.0
85.9
84.1 94.2+0.3
DSMIL
+II Dropout
87.92 1.5
$6.821.6
90.62 1.2
85.6 0.9
84.8 0.5
$7.6 = 0.8
89.9 0.6
90.0s 0.5
9..3
86.9 0.4
88.3
93.9 0.3
+1.6
+2,8
8..7 2.1
83.312,9
86.5 2.4
86.8 1 1.0
85.9 1.2
89.7 1 0.6
87.0 0.6
93.3 0.7
88.2 2.1
8..3
9..6
TransML
+ MIL Dropout
86.0
84.921.5
90.3
88.5
91.6 9.4
A
+1.3
+1.3
+2,9
+2.9
+2.8
+0.6
+1.1
+1.6
+1.0
+2,8
+3.7
+1.6
8.6
88.2 033
87.4
88.0
67.6
DTFD-- MILLAPS)
+MIL Dropout
85.7++1 1.4
79.1 2.2
89.9 0.6
88.5 0.7
84.2 0.6
92.5 + 1.0
M0.3 0.4
90.0 + 0.4
0.1
91.5 + D.4
91 X 0.4
96.1 U.2
A
+1.6
+3.6
+1.6
+1.5
+2.9
+18
+2.0
10.4
+3.9
14.0
63.0
84.7 1.8
78.3
87.8 0.8
87.7 = 1.5
8
88.4 0.9
87.4 1.0
87.3e L.D
9..
85.1 1.2
84.9
91.0 1.0
DTTEIS
MIIL Dropout
86.5+0.9
89.8
89.5
91.6 84
88.+00.
9.
N7.5
93.2
MILIMaxS)
+1.8
+2.7
+2.0
+1.8 +2.4
+3.2
+1.4
+1.1
+1.2
+2.4
+3.3
+2.
TCGA-NSCLC datasets, using features extracted by ResNet- 50 pretrained on ImageNet. The optimal values for K varies from dataset to dataset, with K 10 on CAMELYON16 and K 30 TCGA-NSCICC (Fig. 5(a) and (b)). This suggests that TCGA-NSCLC may have large variability in instances. Additionally, too-large K and G may disrupt MIL tasks by discarding too much information, whereas a too-small K and G may not be sufficient enough to drop easily identified positive instances and encourage MIL to discover richer features. a result, we seek to find an equilibrium point that balances these effects.
miss some true positive regions or show a lower impor- tance. These results show evidence that MIL-Dropout can potentially help uncover more rich and generalizable lesion instedd of relying on single group of surely iden- tified positive instances to make decisions. This aligns with the underlying rationale and theoretical guarantees behind the success of MIL-Dropout.
7. Conclusion
In this paper, we thoroughly analyze the role of dropout-like methods in MIL. Our observations empirically demonstrate that simply dropping certain instances can significantly en- hance the performance and generalization of existing MIL. methods. We further propose an MIL-Dropout to system- atically determine the instances to drop by leveraging an attention aatention-based mechanism to weigh instance importance and a query -base mechanism select instance sets. Experi-
6.4. Lesion localization.
As shown in Fig. (c), ABMIL with MIL-Dropout provides better localization with a higher attention score the posi- tive instances compared to ABMIL without MIL-Dropout. In contrast, the attention map of vanilla ABMIL may either
8

---

How Effective Can Dropout Be in Multiple Instance Learning ?
ments on several MIL datasets demonstrate the effectiveness of our MII.-Dropout. We hope our findings can guide the design of MIL models to learn more rich and generalizable features.
References
Andrews, S., Tsochantaridis, I., and Hofmann, T. Support vector machines for multiple-instane instance learning. Advances in neural information processing systems, 15, 2002.
Acknowledgment
Babenko, B., Yang, M.-H., and Belongie, S. Robust object tracking with online multiple instance learning. IEEE transactions on pattern analysis and machine intelligence, 33(8):1619-1692, 2010.
This material is based upon the work supported by the National Science Foundation under Grant Number CNS-
2204721. is also supported by our collaborative project with MIT Lincoln Lab under Grant Numbers 2015887 and 700061 12889. Additionally, this work was partially supported by National Institutes of Health, United States (201EY032125 and R21 AG065942) and the State of Ari- zona via the Arizona Alzheimer Consortium.
Chen, R. J., Ding, T., Lu, M. Y., Williamson, D. F., Jaume, G., Chen, B., Zhang, A., Shao, D., Song, A. H., Shaban, M., et al. Towards a general-purpose foundation model for computational pathology. Nature Medicine, 2024a.
Chen, T., Kornblith, S., Norouzi, M., and Hinton, G. A simple framework for contrastive learning of visual rep- resentations. In International conference on machine learning, pp. 1597-1607. PMLR, 2020.
Impact Statement
This paper proposes a novel dropout regularization method for multiple instance learning, derived from an empirical and theoretical study. The potential socictal implications of MlI--Dropout can be categorized into three main areas:
Chen, X., Qiu, P., Zhu, W., Li, H., Wang, H., Sotiras, A., Wang, Y., and Razi, A. Timemil: advancing multivari- ate time series classification via a time-aware multiple instance learning. In Proceedings of the 41st Interna- tional Conference Machine Learning, pp. 7190-7206, 2024b.
(i) Theoretical View. Starting from a generalization per- spective, we investigate two commonly used dropout meth- ods in MIL and drop instance. We find that drop instance not only provides stronger generalization but also improves performance. Building on this, we analyze dif- ferent instance-dropping strategies from an optimization standpoint and conclude that dropping the top-k instances achieves a lower gradient-direction error and reaches a flat- ter minimum with better generalization. Our findings and theoretical insights also fill an important gap for subsequent research on regularization in MIL.
Choe, and Shim, H. Attention- dropout layer for weakly supervised object localization. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 2219-2228, 2019.
Dietterich, T. G., Lathrop, R. H., and Lozano- Pérez, T. Solv- ing the multiple instance problem with axis-parallel rect- angles. Artificial intelligence, 89(1-2)31-71, 1997.
(ii) Applicability. Our dropout method can be seamlessly integrated with existing MIL approaches and applied to all MII. algorithms, whether pathology or other weakly supervised classification tasks in natural images.
Dinh, L.,, Pascanu, R., Bengio, S.,, and Bengio, Y. Sharp min- ima can generalize for deep nets. In International Con- ference on Machine Learning, pp. 1019-1028. PMI.R, 2017.
(iii) Interpretability. From Interpretability perspective, our method can enhance the localization capability in MII. based weak supervision and help ensure model robustness. This, in turn, facilitates the identification of vulnerabilities and reduces the risk of adversarial attacks.
Early, J., Cheung, G. K., Cutajar, K., Xie, H.,, Kandola, J., and Twomey, N. Inherently interpretable time series clas- sification via multiple instance learning. arXiv preprint arXiv:2311.10049 2023.
Limitation. At present, we have not extended this method broader range of tasks, such as video frame detection or 3D point cloud classification. Moreover, the current MIL dropout approach does not account for instance uncertainty, which could be a promising direction for guiding instance- drop strategics. We plan to explore these aspects in future work.
Feng, J. and Zhou, Z.-H. Deep miml network. In Pro- ceedings of the AAAI conference on artificial intelligence, volume 31, 2017.
Foret, P., Kleiner, A., Mobahi, H., and Neyshabur, B. Sharpness- aware minimization for efficiently improving generalization. International Conference on Learning Representations, 2021. URL https: openreviee.. net/forum? ddmmdddmmmmmmmmmmmmmmmmmmmm

---

How Effective Can Dropout Be in Multiple Instance Learning ?
Ghiasi, G., Lin, T.-Y., and Le, Q. V. Dropblock: regular- ization method for convolutional networks. Advances in neural information processing systems, 31, 2018.
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z.., Gimelshein, N., Antiga, L., et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
Hinton, G. E., Srivastava, N., Krizhevsky, A., Sutskever, I., and Salakhutdinov, R. R. Improving neural networks by preventing co adaptation of feature detectors, arXiv preprint arXiv: 220. 2012.
Qiu, P., Xiao, P., Zhu, W., Wang, Y., and Sotiras, A. Sc-mil: Sparsely coded multiple instance learning for whole slide image classification. arXiv preprint AXXiv:33/1.00044 2023.
Hou, L., Samaras, D., Kurc, T. M., Gao, Y., Davis, J. E., and Saltz, J. H. Patch-based convolutional neural network for whole slide tissue image classification. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2424 2433, 2016.
Qu, I.., Yang, 7.,, Duan, M., Ma, Y., Wang, S., Wang, M., and Song, Z. Boosting whole slide image classification from the perspectives of distribution, correlation and mag- nification. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 21463 -4443,,,3,, 2023.
Huang, Y. Zhao, W., Fu, Y., Zhu, L., and Yu, L. Unleash the power of state space model for whole slide image with local aware scanning and importance resampling. IEEE Transactions on Medical Imaging. 2024.
Quellec, G., Cazuguel, G., Cochener, B., and Lamard, M. Multiple-instance learning for medical image and video analysis. IEEE reviews in biomedical engineering, 10: 233,234,22 2017.
Ilse, M., Tomezak, J., and Welling, M. Attention- baased deep multiple instance learning. In International conference on machine learning, pp. 2127-2136. PMLR, 2018.
Shao, Z., Bian, H., Chen, Y., Wang, Y., Zhang, J., Ji, X., et Transmil: Transformer based correlated multiple instance learning for whole slide image classification. Advances in neural information processing systems, 34:2136-2147 2021.
Jastrzebski, S., Szymczak, M., Fort, S., Arpit, D., Tabor, J., Cho, K. and Geras, K. The break-even point on optimization trajectories of deep neural networks. arXiv preprint arXiv:2002.09577 099002,...99................ 2020.
Song, A. H., Chen, R. J., Jaume, G., Vaidya, A. J., Baras, A. S., and Mahmood, F. Multimodal prototyping for can- cer survival prediction. arXiv preprint arX1v:2407.00224. 2024.
Jaume, G., Vaidya, A., Chen, R. J., Williamson, D. F., Liang, P. P., and Mahmood, Modeling dense multimodal interactions between biological pathways and histology for survival prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11579-11590, 2024.
Srivastava, N. Hinton, G., Krizhevsky, A., Sutskever, I., and Salakhutdinov, R. Dropout: A simple way to prevent neural networks from overfitting. Jour- nal of Machine Learning Research, 15(56):1929-15 1958, 2014. URL
Kavis, A., Skoulakis, S., Antonakopoulos, K., Dadi, L. T., and Cevher, V. Adaptive stochastic variance reduction for non-convex finite-sum minimization. Advances in Neural Information Processing Systems, 15:23524-23538, 2022.
wwwmmll..rgpppp///////// srivastaval 4a html.
Tang, W., Huang. S., Zhang, X., Zhou, F., Zhang, Y., and Liu, B. Multiple instance learning framework with masked hard instance mining for whole slide image clas- sification. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4078-4087, 2023.
Keskar, N. S., Mudigere, D., Nocedal, J., Smelyanskiy, M., and Tang, P. T. P. On large-batch training for deep learning: Generalization gap and sharp minima. arXiv 2016.
preprint arXiv:1609.04880
Li, B., Li, Y., and Eliceiri, K. W. Dual-stream multiple instance learning network for whole slide image clas- sification with self- supf-super contrastive learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 14318-14328, 2021.
Tompson, J., Goroshin, R., Jain, A., LeCun, Y., and Bre- gler, C. Efficient object localization using convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 648-656, 2015.
Liu, Z., Xu, Z., Jin, J., Shen, Z., and Darrell, T. Dropout reduces underfitting. In International Conference on Machine Learning, pp. 2233-22248. PMLR, 2023.
Tu, M., Huang, J., He, X., and Zhou, B. Multiple instance learning with graph neural networks. arXiv preprint arXiv: (90..0008.... 2019a.
Park, J., Woo, S., Lee, J..Y., and Kweon, I. S. Bam: Bottle- neck attention module. arXiv preprint arXiv:1807.06514 2018.
Tu, M., Huang, J., He, X., and Zhou, B. Multiple instance learning with graph neural networks. arXiv preprint arXiv: / 04881, 2019b.
10

---

How Effective Can Dropout Be in Multiple Instance Learning ?
Wang, X., Girshick, R., Gupta, A., and He, K. Non-local neural networks. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 7794- 7803, 2018a.
Zhu, W., Chen, X., Qiu, P., Sotiras, A., Razi, A., and Wang. Y. Dgr-mil: Exploring diverse global representation in multiple instance learning for whole slide image classifi- cation. In European Conference on Computer Vision, pP. 333-351. Springer, 2024.
Wang. X., Yan, Y., Tang, P., Bai, X., and Liu, W. Revisiting multiple instance neural networks. Pattern Recognition, 74:15-24, 2018b.
Zhu, Z Xu, M., Bai, S., Huang, T., and Bai, X. Asymmetric non-local neural networks for semantic segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 593-602, 2019.
Woo, S., Park, J., Lee, J.-Y., and Kweon, I. Cbam: Convolutional block attention module. In Proceedings of the European conference computer vision (ECCV), pp- 3-19, 2018.
Xiang, J. and Zhang, J. Exploring low-rank property in multiple instance learning for whole slide image classi- fication. In The Eleventh International Conference on Learning Representations, 2023.
Xu, G., Song, Z., Sun, Z., Ku, C., Yang. Z., Liu, C., Wang, S., Ma, J., and Xu, W. Camel: weakly supervised learn- ing framework for histopathology image segmentation. In Proceedings of the IEEE/CVF International Conference computer vision, pp. 10682-10691, 2019.
Yan, Y., Wang, X., Guo, X., Fang. J., Liu, W., and Huang, J. Deep multi-instance learning with dynamic pooling, In Asian Conference on Machine Learning, pp. 662-677. PMLR, 2018.
Zhang, H., Meng, Y., Zhao, Y., Qiao, Y., Yang, X., Coup- land, S. E., and Zheng. Difd-mil: Double-tier feature distillation multiple instance learning for histopathology whole slide image classification. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18802-18812, 2022.
Zhang, Z. and Xu, Z.-Q.J. J. Implicit regularization of dropout. IEEE Transactions on Pattern Analysis and Machine In- telligence, 2024.
Zheng, T., Jiang, K., and Yao, H. Dynamic policy-driven adaptive multi-instance learning for whole slide image classification. In Proceedings of the IEEE/CVF Confer- ence Computer Vision and Pattern Recognition, pp. 8028-8037, 2024.
Zhu, W., Qiu, P., Chen, X., Dumitrascu, O. M., and Wang, Y. Pdl: Regularizing multiple instance learning with progres- sive dropout layers. arXiv preprint arXiv:2308. /01 12, 2023a.
Zhu, W. Qiu, P., Lepore, N., Dumitrascu, O. M., and Wang, Y. Self-supervised equivariant regularization rec- onciles multiple-instance learning: Joint referable dia- betic retinopathy classification and lesion segmentation. In 18th International Symposium on Medical Information Processing and Analysis, volume 12567, pp. 100-107. SPIE, 2023b.

---

How Effective Can Dropout Be in Multiple Instance Learning
Dropout
Linear ReLU (1024-2512)
Linear ReLU (512->256)
Linear + ReLU (256~>128)
MIL Aggregatto (e.g. ABMIL, TransMIL, DTFD-MIL.)
Figure ABMIL aggregation with Dropout setting within investigation experiments
A. Investigation Experiment Design
We implementation all experiments node of cluster with NVIDIA V100 (32GB). We use Pytorch Library (Paszke et al., 2019) with version of 1.13. The investigation experiments employ the same set of CAMELYONI6 extractor features; for the pre-processing stage, we utilized Otsu's 's thresholding method to localize tissue regions within each slide image (WSI). Subsequently, non-overlapping patches, each measuring 256 x 256 pixels at 20X magnification, were extracted from these localized tissue regions. total, 3.7 million patches were obtained from the CAMELYON-I6 dataset (Zhang et al., 2022). Following the previous work (Zhang et al., 2022), The pre-trained Resnet-50 on ImageNet serves the feature extractor for the embedding patch 1024-dimensional vector. The investigation experiments the training/testing provided by CAMELYON16 officials.
For these two experiments, we uniformly adopted the ABMIL (Ilse et al., 2018) with cross-entropy loss, and the Lookahead optimizer was employed with a learning rate of le-4 and weight decay le-4. To integrate Dropout into ABMIL, we employed three fully connected layers and ReLU before ABMIL aggregation 1024-1112 -512-2256-+18)) After applying the ReLU activation function to each layer, dropout regularization introduced,as shown Figure 6.
A.1. Detail of Generalization experiment
We conducted this experiment to investigate the CAMELYON16 dataset. We train the model on the original training set and test its generalization on the test data with artifact noise to mimic the domain shift between training data and the unseen test in real- world scenarios. We consider some common types of noise in the imagining process. Specifically, add five levels for each type of noise: i) for Gaussian noise {0.01, 0.05, 0.1, 0.2, 0.3). ii) for Poisson noise, set its scale to {1.,5., 5., 10., 20., 25.). and iii) for Salt and Pepper noise, set the probability of both salt and pepper to {0.01, 0.03, 0.05, 0.07, 0.09). Therefore, we have 15 different noisy test sets now. Fig.7 shows the exemplary patch with different noises. Apparently, model with better generalization should be able to perform better most of these datasets.
Now, let us elaborate on how to derive Eq. 5 to Eq. 6. The Taylor expansion for L(l') presented in Eq. 5 can be expressed as,
T((0' ~ I(O) VT(O)T
0) (0'-0)) (0) -0 VV)-)))0)(000'' -) 0) O - 0).
1
Due to a local minimal, V((0) 0, thereby,
L((0) L()) ; -) T L0
(10)
The maximum of a quadratic form occurs when (0' 0) aligns with the eigenvector corresponding to the largest eigenvalue of L(), therefore,
max (at(e.) (L'))) L(0))
L())2 2(1 + L(0))
1 L(0)
where [[)]1) denotes the spectral norm (largest eigenvalue).
12

---

How Effective Can Dropout Be in Multiple Instance Learning
Gaussian
Poisson
Pepper
Figure 7. The exemplary patch with different noise types and strengths.
Table 3. Parameters setting for baseline MIL methods. Here employed the AdamW, W, RAdam, CosineAmnealingLR to follow these MIL methods papers setting.
ABMIL
ABMIL-
DSMIIL
TransMII.
DTFD-MII.
Optimizer
AdamW
AdamW
AdamW
RAdam
Adam
Learning rate
le
le
le
le-4
2e
Weight decay
1e-4
le-4
5e-3
5e-3
2e-3 -3
Optimizer scheduler
LookAhead
LookAhead
CosineAnnealingL.R
LookAhead
MultiStepL.R
Loss function
CrossEntropy
CrossEntropy
CrossEntropy
CrossEntropy
CrossEntropy distille loss
A.1.1. GRDIENT MEASURE EXPERIMENT
We conducted this experiment based on the CAMELYON16 training and testing The primary objective of fitting the training data is to minimize the loss across the entire training set, rather than focusing any individual mini-batch. Following in (Liu et al., 2023), We compute the gradient for given model over the entire training set, setting DropInstance to inference mode to capture the full model's gradient. Subsequently. we evaluate the deviation of the actual mini-batch gradient 9step from this whole-dataset "ground-truth" gradient ÿ. We define the average cosine distance from all Jstep to as the gradient direction error (GDE):
GDE
1 G
Σ gung
(Asuup- g) 11 l2 |lā]|2
Here, we employ the same architecture as shown in Fig 6 for integrating Dropout. In this manuscript, we investigate four different DropInstance modes: top-k, bottom-k, random DropInstance, and without DropInstance. For the CAMELYONI6 dataset, the value of is consistently set to 20. The terms "top" and "bottom" refer to the ABMIL aggregation attention map
13

---

How Effective Can Dropout Be in Multiple Instance Learning
MIL- Dropout
Linear + ReLU
Linear + ReLU
Linear + ReLU
MIL. Aggregatto (e.g. ABMIL, TransMIL, DTFD-MIL.)
(1024->512 Or 512->256)
(1024-2512
(1024->512 Or 128->64)
Or 256->128)
Figure 8. Different aggregation methods with MIL--Dropout setting within main experiments
obtained from the previous iteration:
N
(vU( n1 n== 1, ,N)) Σ
Ox Un.
(11)
n=1
with On softmax(uit (w) tanh(w?e?")
where an implies the importance of the n-th instance. In this context, the top and bottom refer to the ranking of attention scores in descending order, with the top corresponding to the highest attention scores and the bottom to the lowest. In addition, we conducted performance comparison on the testing set. We observed that these models converge after approximately 10,000 iterations and achieve optimal performance prior to this point. In the manuscript, we present the highest AUC achieved before 10,000 iterations based the four different DropInstance modes.
B. Main Experiment Details
We provide details of integrating MIL-Dropout into existing MIL methods, including ABMIL (Ilse et al., 2018), DTFD- MIL (Zhang et al., 2022), TransMIL (Shao et al., 2021), DSMIL (Li et al., 2021).
B.1. MIL Benchmark experiments setting
B. 1.1. EMBEDDING FEATURES
For the MUSKI and MUSK2, A bag is constructed for each molecule, as instances. Each instance is represented with a 166-dimensional embedding feature. The FOX, TIGER, and ELEPHANT datasets contain 200 bags of instance features. Each instance is associated with 230-dimensional embedding feature.
B. 1.2. DETAIL OF INTEGRATING MIL DROPOUT INTO EXISTING MIL ARCHITECTURES
We employed three fully connected layers in shallow feature extractor fe, each comprising 256, 128, and 64 hidden units. Each layer was equipped with ReLU activation function and subsequently appended by MIL Dropout layer (refer to Fig 8). Following the passage of features through the fe. the ABMIL, ABMIL-Gate, and DSMIL aggregators processed the 64-dht 14-dimensional embedding feature as their input. To compare triginal-based models, all aggregators adhere to the configuration outlined in paper (Li et al., 2021), with distinct embedding features of dimensions 230 and 166 directly employed as inputs. All experiments used the Adam optimizer with 2e -4 learning rates and 5e -3 weight decay and trained on cross-entropy loss for 40 epochs.
B.2, WSIs experimental setting
B. PREPROCESSING WSI
Following the threshold (Li et al., 2021) and OTSU (Zhang et al., 2022) preprocessing methods, each WSI was divided into non-overlapping 224 224 patches at a magnification of x20. This results in total of 3.7 million patches (~ 9000 per
14

---

How Effective Can Dropout Be in Multiple Instance Learning
bag) from the 1 dataset and 8.3 million patches (~~ 8000 per bag) from the TCGA Lung Cancer dataset.
B.2.2. EMBEDDING NETWORK PRETRAINED
Two sets of patches were extracted using different frameworks: (i) ImageNet Pretrained: ResNet-50 from DTFD- MIL, yielding (224-dimensional vectors per patch, and (i) SimCLR Pretrained: the SimCL contrastive learning framework from DSMIL, yielding 512-dimensional vectors per patch. The self-supervised SimCLR manner employed the contrastive learning framework (Chen et al., 2020) to pretrain the projector based on the training set, wherein contrastive loss training was implemented between extracted patches and corresponding two random data lata-auumentttion counterparts (Li al., 2021).
B.2.3. DETAIL OF INTEGRATING MIL DROPOUT INTO EXISTING MIL. ARCHITECTURES
We followed the parameter settings outlined in the original literature for the baseline experiments the two WSI datasets, as shown in Table 3. To integrate MIL Dropout into these MIL methods, we employed three fully connected layers in shallow feature extractor fo. each comprising 512 256, 256 128, and 128 / 64 hidden units. Each layer was equipped with a ReLU activation function and subsequently appended by a MIL Dropout layer as shown in Fig The input dimensions of these networks were adjusted to 64 while keeping the other parameters unchanged. All the experiments were trained on 200 epochs. The MIL Dropout parameters were identical the paper description (section Implementation details). The experiments integrated with MIL Dropout also followed the parameters in Table 3.
C. Rebuttal Extra Experiment
C.1. UNI feature (Chen et al., 2024a)
Consistent with requirements from other reviewers, we have replicated the relevant experiments here for clarity. Because UNI was pre-trained on TCGA and Camelyon data-raising the possibility of data leakage-wa conducted additional experiments using the independent EBRAINS dataset. The results of these experiments are shown below.
Model
Accuracy
Fl
A Accuracy
ABMIL
65.4
68.7
+MIL Dropout
70.4
73,2
+5.0
+4.5
TransMIL.
67.4
74.4
+MIL Dropout
71.3
79.4
+3,9
+5.0
DSMIL
67.4
74.4
+MIL Dropout
69.3
76.0
+1.9
+1.6
DTFD
53.4
63.6
+MIL Dropout
64.8
69.8
+11.4
+6.2
Although our MIL dropout still offer improvements when the better patch features are implemented, the performance gains may be less substantial (about 1.5 to 0.5). Nevertheless, as demonstrated by the experiments, foundation model features often do not perform optimally on private datasets, making our approach particularly suitable for such scenarios.
C.2. Additional Experiments
Currently, experiments primarily utilize shallow feature extractors. We also attempted to integrate MIL dropout into the classifier (add more linear layers for integrating MIL Dropout) following the MIL module (actually the same thing we felt), and the performance remained largely unchanged. Furthermore, integrating MIL dropout into a transformer architecture was -ikely
unsuccessful likely because MIL dropout interferes with layer normalization.
MIL dropout fails converge for integration into transformer blocks.
C.3. Extra Comparison
Our MIL dropout can seamlessly integrated with other MIL methods, as it is orthogonal to them. We conducted experiments using our patch features and observed performance improvements, further demonstrating the flexibility and
15

---

How Effective Can Dropout Be in Multiple Instance Learning
Table Model (running times on CAMELYON16 Imagenet)
Model
Accuracy
F1
AUC
TransMIL
84.7
83.3
86.5
+MIL Dropout in shallow extractor
86.0
84.9
89.4
+MIL Dropout in classifier
87.6
82.5
88.7
+MII. Dropout in Transformer
64.5
52.5
52.5
effectiveness of our dropout.
Table Model (running 3 times on CAMELYONI6 Imagenet), [(Zheng et al., 2024; Qu al., 2023)/The full open source code has not been relcased, and we believe our MIL dropout can be integrated into these methods.
Model
Accuracy
AUC
Accuracy
A AUC
PAM (Huang et al., 2024)
85.0
83.2
86.7
+MIL Dropout
86.2
84.3
87.7
+1.2
+1.0
DPSF (Zheng et al., 2024)
(Qu et al., 2023)
C.4. Questions/Suggstionn
C.4.1. EXPERIMENTAL SETUP:
Figure 5(c) reports ablation results on two datasets using ImageNet- features. All experiments in Table 2 use the optimal values of K and G identified in Figure 5(c).
C.4.2. PERFORMANCE DEGRADATION WITH HIGH DROP RATE:
Dropping a large proportion of instances inevitably degrades performance - not only for our proposed MII.-Dropout but also for standard/vanilla dropout.
When most features or instances are zeroed out, the model struggles to distinguish positive from negative instances (ReLU outputs zero for both), while the bag-level label remains positive. This prevents effective learning.
In our Camelyon16 experiments, aggressive dropout led to the failure of convergence for all methods; See the following results.
Table 6. Dropout Comparison on Camelyon16
Model
Accuracy
ABMIL
86.3
+Dropout(p=0.4)
66.2
Dropout (=))088t(pp088))))))
53.4
+Dropout (D=0.44)
63.1
+Dropout (D=0.8)
51.3
+Our MIL-Dropout =k 100, G 5)
59.9
+Our MIL-Dropout (k 400, G=
55.7
C.4.3. OPTIMAL DROPOUT THRESHOLD:
By restricting dropout to at most 10% of the average instance count (either via Top-K selection or a dropout probability p == 0.1), we observe consistent performance improvements across datasets.
16

---

How Effective Can Dropout Be in Multiple Instance Learning
C.5. Extra WSI task/datasets Evaluation:
TASK & DATASETS:
We evaluated our method on survival prediction using two TCGA datasets (LUAD and BRCA). following the protocols of (Jaume al., 2024; Song et al., 2024).
C.5.1. EVALUATION METRIC:
Concordance index (C-index) with %. The K = 10, G 10 for MIL-Dropout.
C.5.2. RESULTS:
Table 7. Survival Prediction Results on TCGA-LUAD and ICGA-BRCA
Model
TCGA-LUAD +MIL-Dropout
A
TCGA-BRCA
+MIL-Dropout
A
ABMIL
65.7
67.7
+2.0
72.8
75.2
+2.4
DSMIL
61.4
63.4
+2.0
68.8
72.1
+3.3
TransMIL
64.3
67.2
+2.9
72.1
74.7
+2.6
DTFD (AFS)
62.0
65.3
+3.3
71.6
74.6
+2.5
DTFD (MaxS)
65.9
68.8
+2.4
72.8
75.7
+2.9
These results demonstrate that our MIL Dropout consistently improves all baseline methods on WSI-based survival prediction.
17


Reviews:
[{"review_id": "9gqaNyBqeK", "reviewer": "Reviewer_3cHV", "summary": "This paper mainly studies the Dropout scheme for Multi-Instance Learning (MIL). First, this work reveals the superiority of DropInstance over traditional Dropout scheme through empirical experiments on a WSI dataset. Then, this work investigates the suitable scheme for DropInstance, Top-K instance dropping. Finally, a MIL-Dropout approach is proposed for MIL. This approach leverages average pooling to determine the importance of instances, which eases the dependence on instance's attention scores, followed by dropping the Top-K instances to implement the aforementioned Top-K instance dropping. Experiments on classical MIL benchmark and WSI datasets demonstrate the effectiveness of the proposed method.\n\n## update after rebuttal\n\nThe authors' rebuttal fails to directly answer my questions and concerns. My detailed responses to the authors' rebuttal could be found in the thread. Thus, my major concerns remain as follows:\n- Experimental settings are not aligned with the common settings in WSI classification, which leads to the concerns about the convincingness of the experimental results.\n- Inherent limitations of the proposed method, e.g., being sensitive to the setting of K and relying on careful fine-tuning, could largely limit the impact of this work.\n- Some theoretical claims lack convincing justifications. In a nutshell, I don't think the theoretical claims could be made from the empirical experiments on only one dataset. Convincing justifications are need to support the authors' theoretical claims.", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**We greatly appreciate the reviewer’s valuable feedback. Below is our response:**\n\n1. **Sub‑optimal** refers to using a feature extractor that isn’t trained jointly with the MIL module, resulting in non‑optimal embeddings. Even strong feature extractors (self‑supervised training foundation models) achieve only ~0.60 accuracy on EBRAINS (e.g., UNI).  Prior work has begun exploring end‑to‑end training strategies, which empirically outperform two‑stage approaches [1].\n\n2. **Claim of Theoretical analysis:** \nWe will revise the text to be more precise and accurately reflect the level of theoretical discussion presented in the paper. \nOur claims are based on established theoretical and empirical findings (Keskar et al., 2016; Liu et al., 2023):\n   - Smaller sharpness leads to flatter local minima, which generally improves generalization.\n   - Gradient variance is widely used to evaluate the optimizers.\n   \n\n3. **Analysis of additional datasets:**  \n   We observed the same phenomenon on **TCGA** in ViT pre-trained on ImageNet. Models like UNI or CONCH were trained on TCGA and Camelyon, introducing data leakage; thus, downstream evaluation on those datasets is not meaningful. In Following ViT based experiments, DropInstance shows the same effect described in the manuscript: \n\n   |Method (TCGA - ViT)|Sharpeness (ε=1.00e-03)|\n   |:-----|----------------------:|\n   |DropNeuron|1.36e-7|\n   |**DropInstance**|**1.16e-8**|\n   |Baseline|1.06e-6|\n\n4. **Top-K:**   Standard dropout also degrades performance when a large proportion of features or instances are zeroed out. If all positive instances are dropped, positive and negative instances become indistinguishable (ReLU outputs zero), while the bag label remains positive, preventing the model from learning. In our Camelyon16 experiments, aggressively dropping instances caused all methods to fail to converge.\n   |Model|Accuracy|\n   |:---|---:|\n   |ABMIL|86.3|\n   |+Dropout(p=0.4)|66.2|\n   |+Dropout(p=0.8)|53.4|\n   |+Dropout1D(p=0.4) |63.1|\n   |+Dropout1D(p=0.8)|51.3|\n   |+Our MIL Dropout (k = 100, G= 5)|59.9|\n   |+Our MIL Dropout (k = 400, G= 5)|55.7|\n\n   However, limiting dropout to no more than 10% of the average instance count—either by selecting Top‑K instances or using a dropout \n   p=0.1—consistently improved performance. Our ablation study confirms this result and will add more details to the \n   revised manuscript.  \n\n5. **Extra-Experiments:**  \nBelow are the results on Camelyon16 (Imagenet pretrained) using the mentioned latest method. We add MIL‑Dropout via two linear layers before feeding features into the MIL module.  \n|Model|Accuracy|F1|AUC|Δ Accuracy|Δ F1|Δ AUC|\n|:---|------------:|-------:|-------:|-----------:|----:|----:|\n|ILRA|85.8|81.4|88.4|—|—|—|\n|+MIL Dropout|87.2 |86.4 |90.1 |+1.4|+5.0|+1.7|\n|CAMIL|84.8|82.6|86.8 |—|—|—|\n|+MIL Dropout|86.4|85.7|91.2 |+1.6|+3.1|+4.4|\n|R2T|83.1|81.6|84.5|—|—|—|\n|+MIL Dropout|85.6|84.7|87.4 |+2.5|+3.1|+2.9|\n\n   Because UNI was trained on TCGA and Camelyon, using those features risks data leakage. To avoid this, we evaluate UNI features on the EBRAINS dataset by running the experiment three times.  \n    |Model|Accuracy|F1|Δ Accuracy|Δ F1|\n    |:---|------------:|-------:|-----------:|----:|\n    |ABMIL|65.4|68.7|—|—|\n    |+MIL Dropout|70.4|73.2|+ 5.0|+ 4.5|\n    |TransMIL|67.4|74.4|—|—|\n   |+MIL Dropout|71.3|79.4|+ 3.9|+ 5.0|\n    |DSMIL|67.4|74.4|—|—|\n   |+MIL Dropout|69.3|76.0 |+ 1.9|+ 1.6|\n   |DTFD|53.4|63.6|—|—|\n   |+MIL Dropout|64.8|69.8|+ 11.4|+ 6.2|\n   |ILRA|64.8|74.4|—|—|\n   |+MIL Dropout|66.8|76.0|+ 2.0|+ 1.6|\n   |CAMIL|60.7|68.5|—|—|\n   |+MIL Dropout|69.1|75.6|+ 8.4|+ 7.1|\n   |R2T|57.9|66.1|—|—|\n   |+MIL Dropout|64.8|69.8|+ 6.9|+ 3.7|\n\n   Our MIL Dropout consistently improves other MIL methods, with the largest gains on noisier UNI features—showing its effectiveness when patch features are suboptimal.\n\n5. **Relation to arXiv:2308.10112**  \n   PDL is an unpublished work, and our MIL Dropout complements and extends the theory-grounded analysis behind PDL. While PDL methods are largely intuition‑driven and derive DropInstance from nonlinear interpolation without deeper analysis, our work demonstrates that DropInstance leads to flatter minima and better generalization. We further show that Top‑K dropout reduces gradient variance. We appreciate the reviewer’s pointer to this paper and will include a detailed discussion in the revised manuscript.\n\n    |Model|Accuracy|F1|AUC|\n    |:---|---:|---:|---:|\n    |ABMIL|86.3|85.0|86.0|\n    |+Our MIL Dropout|87.2|86.4|90.1|\n    |+PDL|86.1|84.3|85.2|\n    |+Concrete Dropout|85.3|84.2|84.0|\n    |+guided dropout|82.9|81.2|81.4|\n    |+alpha-dropout|83.7|81.8|78.1|\n\n    Due to space constraints, we provide a comparison of our method against PDL and other dropouts on Camelyon16 (ImageNet).\n\n---\n**Reference**  \n[1] Li et al. (2023). Task‑specific fine‑tuning via variational information bottleneck for weakly‑supervised pathology WSI classification.", "claims_and_evidence": "Some claims are not supported by convincing evidence. My major concerns lie on\n- The effectiveness of Top-K instance dropping: It is only verified on WSI classification task and one dataset. I am curious about whether it could be generalized to more tasks or datasets, so as to demonstrate the generalization of the proposed method.\n- No theoretical analysis is conducted to support this paper's core argument. However, the authors claim \"whereas an in-depth analysis reveals its theoretically intriguing properties.\" at lines 65 and 66 on page 2.", "methods_and_evaluation_criteria": "**Methods**: Most designs seems reasonable. However, these designs are completely driven by the empirical results on a single dataset. This raises critical concerns about the generalization of the proposed method.\n\n**Evaluation Criteria**: seems reasonable.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "Yes, I have checked the experimental design of this paper. My concerns are as follows:\n- In the experiments, WSIs are processed by ResNet-50 to obtain bag features for MIL. However, this way is out-of-date. Current papers in this field generally adopt more advanced pretrained foundation models like UNI (Nature Medicine, 2024) and CONCH (Nature Medicine, 2024). In other words, the experimental settings are not aligned with the typical settings in the field of MIL-based WSI classification. This calls into question the effectiveness of the proposed method.\n- MIL-based WSI classification is an active topic that is going rapid developments. This paper only includes the MIL baselines proposed in 2023 or earlier years. Many state-of-the-art MIL methods for WSI classification are not compared in this paper, *e.g.*, CAMIL (ICLR, 2024) and R2T (CVPR, 2024).", "relation_to_broader_scientific_literature": "No obvious relation", "essential_references_not_discussed": "It is not clear the relation of this paper with the reference [https://arxiv.org/pdf/2308.10112]. I have given this concern in my comments to the authors. Hope the authors could clarify this in the rebuttal.", "other_strengths_and_weaknesses": "Strengths:\n- This paper investigates the Dropout scheme for MIL. This topic is under-explored in MIL as far as I know.\n- Some empirical studies are conducted to justify the proposed scheme. Most of them are intuitive and easy to understand.\n\nWeaknesses:\n- Lack of theoretical analysis to prove the generalization of the proposed MIL-Dropout method.\n- Some statements are not rigorous for academic papers and should be reworded.\n- The experimental settings are not typical and fail to align with the common settings in WSI classification tasks.\n\nIn summary, I acknowledge that the authors investigate an interesting problem in MIL. However, most claims are only supported by empirical results, which are not convincing enough to prove the generalization of the proposed method. Essential theoretical analysis would be better. Moreover, the experimental settings in WSI classification tasks are not aligned with common settings, which call into question the real performance of the proposed method. Given these critical issues, I think that the current paper can't get published in ICML and it should be improved substantially in terms of theoretical analysis and experiments.", "other_comments_or_suggestions": "Please see *Questions For Authors*", "questions_for_authors": "My main questions and concerns are as follows:\n- For the statement, \"However, this suboptimal training scheme suffers from ”noisy” feature embeddings, typically resulting in reduced performance.\", written on page 1, I am curious about why the authors assert the training scheme is sub-optimal. As far as I know, the two-stage learning scheme, patch feature extracting + MIL aggregator, is a standard framework for WSI classification.\n- For the statement, \"an in-depth analysis reveals its theoretically intriguing properties\", written on page 2, I fail to find any theoretical analysis related to this part. Could the authors provide more details for this statement?\n- For the statement, \"dropping important instances not only results in performance gain but also shows theoretical guarantees.\", written on page 1, I also fail to find theoretical guarantees; instead, the empirical results on one WSI dataset are observed.\n- For the statement, \"achieves better generalization by converging to flatter local minima (see example in Fig 1 (Right) and evidence in Fig. 3)\", written on page 1, I am curious about whether this is the same in other WSI datasets or this is the same when the pretrained model is changed to UNI or CONCH but not ResNet-50.\n- For the Top-K instance dropout, there are two severe limitations:\n  - How to determine K. Would the performance be sensitive to this? How can I believe this method is a stable one for MIL?\n  - If the number of positive instances are less than K, what will happen?\n- A pretrained ResNet-50 model is used to extract features from patches. This setting is out-of-date. Could the authors provide the main results for modern pretrained models like UNI (Nature Medicine, 2024) or CONCH (Nature Medicine, 2024).\n- The baselines presented in Table 2 are from the work published in 2022 or earlier. Could the authors provide the results for state-of-the-art MIL baselines, like ILRA (ICLR, 2023), CAMIL (ICLR, 2024) and R2T (CVPR, 2024).\n- It is not clear the relation of this paper with the reference [https://arxiv.org/pdf/2308.10112].\n\nReferences:\n\n[1] UNI: Towards a General-Purpose Foundation Model for Computational Pathology\n\n[2] CONCH: A visual-language foundation model for computational pathology\n\n[3] ILRA: Exploring low-rank property in multiple instance learning for whole slide image classification\n\n[4] CAMIL: Context-Aware Multiple Instance Learning for Cancer Detection and Subtyping in Whole Slide Images\n\n[5] R2T: Feature re-embedding: Towards foundation model-level performance in computational pathology", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8i2EipsR9J", "reviewer": "Reviewer_ds4G", "summary": "The authors study how dropout can be used specifically for multiple instance learning (MIL), with a focus on its use for WSI-level prediction tasks within computational pathology.\n\nThey propose a MIL-specific dropout method named MIL-Dropout. It entails dropping the top K instances, along with G instances which are highly similar to these top K instances. MIL-Dropout can be applied to various MIL approaches such as ABMIL and TransMIL.\n\nThe proposed MIL-Dropout is evaluated on five MIL benchmark datasets, and on two WSI-level classification datasets. When combined with various MIL approaches (e.g. ABMIL), the performance is consistently improved at least to some extent.\n\n\n\n\n##########\n\n##########\n\n##########\n\n##########\n\nUpdate after the rebuttal:\n\nI remain borderline on this paper.\n\nI think the proposed MIL-Dropout potentially could be at least quite useful, but I tend to agree with reviewer 3cHV that there are some remaining concerns.\n\nI will thus keep my current score of \"_2: Weak reject (i.e., leaning towards reject, but could also be accepted)_\".", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**We sincerely thank the reviewer for the valuable feedback, and for taking the time to read and engage with our paper. Below we address the reviewer’s concerns point by point.**\n\n---\n\n### 1.**Questions/Suggestions**: \n\n- **Experimental setup:**  \n  Figure 5(c) reports ablation results on two datasets using ImageNet‑pretrained features. All experiments in Table 2 use the optimal values of K and G identified in Figure 5(c).\n\n- **Performance degradation with high drop rate**:  \n  - Dropping a large proportion of instances inevitably degrades performance — not only for our proposed MIL‑Dropout but also for standard/vanilla dropout.  \n  - When most features or instances are zeroed out, the model cannot distinguish positive from negative instances (ReLU outputs zero for both), while the bag-level label remains positive. This prevents effective learning.  \n  - In our Camelyon16 experiments, aggressive dropout led to the failure of convergence for all methods; See the following results.\n     |Model|Accuracy|\n     |:---|---:|\n     |ABMIL|86.3|\n     |+Dropout(p=0.4)|66.2|\n     |+Dropout(p=0.8)|53.4|\n     |+Dropout1D(p=0.4) |63.1|\n     |+Dropout1D(p=0.8)|51.3|\n     |+Our MIL-Dropout (k = 100, G= 5)|59.9|\n     |+Our MIL-Dropout (k = 400, G= 5)|55.7|\n- **Optimal dropout threshold:**  \n  - By restricting dropout to at most 10% of the average instance count (either via Top‑K selection or a dropout probability _p_ = 0.1), we observe consistent performance improvements across datasets.\n\n> **We appreciate the reviewer’s suggestion and will add more detailed explanations in the revised manuscript, and will also correct the spelling errors you pointed out.**\n\n### 2.**Extra WSI task/datasets Evaluation**: \n\n- **Task & Datasets:**  \n  We evaluated our method on survival prediction using two TCGA datasets(LUAD and BRCA), following the protocols of [1,2].\n\n- **Evaluation Metric:**  \n  Concordance index (C‑index) with %. The K = 10, G = 10 for  MIL-Dropout.\n\n- **Results:**  \n| Model       | TCGA‑LUAD | +MIL-Dropout | Δ | TCGA‑BRCA | +MIL-Dropout| Δ |\n|-------------|----------:|-------------:|-----------:|----------:|-------------:|-----------:|\n| ABMIL       | 65.7      | 67.7         | +2.0       | 72.8      | 75.2         | +2.4       |\n| DSMIL       | 61.4      | 63.4         | +2.0       | 68.8      | 72.1         | +3.3       |\n| TransMIL    | 64.3      | 67.2         | +2.9       | 72.1      | 74.7         | +2.6       |\n| DTFD (AFS)  | 62.0      | 65.3         | +3.3       | 71.1      | 73.6         | +2.5       |\n| DTFD (MaxS) | 65.9      | 68.3         | +2.4       | 72.8      | 75.7         | +2.9       |\n\n\n> **These results demonstrate that our MIL Dropout consistently improves all baseline methods on WSI-based survival prediction**.\n\n### 3.**Foundation Model Features**: \n\n- **UNI Features:**  \nAs same as requested by Reviewer 3, we evaluated our method using UNI features. Because UNI was pretrained on TCGA and Camelyon data—which poses a risk of data leakage—we conducted additional experiments using the independent EBRAINS dataset.\n    |Model|Accuracy|F1|Δ Accuracy|Δ F1|\n    |:---|------------:|-------:|-----------:|----:|\n    |ABMIL|65.4|68.7|—|—|\n    |+MIL Dropout|70.4|73.2|+ 5.0|+ 4.5|\n    |TransMIL|67.4|74.4|—|—|\n   |+MIL Dropout|71.3|79.4|+ 3.9|+ 5.0|\n    |DSMIL|67.4|74.4|—|—|\n   |+MIL Dropout|69.3|76.0 |+ 1.9|+ 1.6|\n   |DTFD|53.4|63.6|—|—|\n   |+MIL Dropout|64.8|69.8|+ 11.4|+ 6.2|\n   |ILRA|64.8|74.4|—|—|\n   |+MIL Dropout|66.8|76.0|+ 2.0|+ 1.6|\n   |CAMIL|60.7|68.5|—|—|\n   |+MIL Dropout|69.1|75.6|+ 8.4|+ 7.1|\n   |R2T|57.9|66.1|—|—|\n   |+MIL Dropout|64.8|69.8|+ 6.9|+ 3.7|\n\n> **Our MIL Dropout consistently improves other MIL methods.**\n---\nWe hope our response addresses your concern. If you have any further questions, please let us know. Thanks!\n---\n\n**References**  \n[1]. Jaume, G., et al. (2024). *Modeling dense multimodal interactions between biological pathways and histology for survival prediction.* CVPR.  \n[2]. Song, A. H., et al. (2024). *Multimodal prototyping for cancer survival prediction.* ICML.", "claims_and_evidence": "Overall, yes.", "methods_and_evaluation_criteria": "Overall, yes.", "theoretical_claims": "N/A.", "experimental_designs_or_analyses": "The experimental setup is OK overall, but could include one or two more WSI-level datasets/tasks. Could also include a more recent histopathology foundation model as the patch-level feature extractor.", "relation_to_broader_scientific_literature": "OK discussion of related work in Section 2.", "essential_references_not_discussed": "N/A.", "other_strengths_and_weaknesses": "Strengths:\n- The studied problem is quite interesting, if dropout can be used to consistently improve the performance of e.g. ABMIL models within computational pathology, this would definitely be valuable.\n- The proposed MIL-Dropout method makes sense overall, and can be applied to various common MIL approaches.\n- The performance of various MIL approaches is consistently improved at least to some extent on two WSI-level classification tasks (Table 2).\n\n\n\nWeaknesses:\n- The paper could be a bit more well written, it contains quite a few typos etc.\n- The experimental evaluation only includes two WSI-level datasets. No proper external validation.\n- The WSI-level experiments don't utilize any of the recent histopathology foundation models as patch-level feature extractors.\n- The proposed MIL-Dropout is not compared against any other dropout variant as a baseline.", "other_comments_or_suggestions": "Questions/Suggestions:\n- For Figure 5.c, could you show more examples? Is this actually a general trend?\n- The results in Table 2, are these with optimized values for K and G? I.e., different values for CAMELOYON16 vs TCGA-NSCLC, different values for ImageNet model vs SimCLR model, for ABMIL vs TransMIL etc? If so, might these reported performance gains for MIL-Dropout not be somewhat overstated? Because, if K and G are chosen just a bit differently, the performance might be worse than for the non-dropout baselines?\n\n\n\n\nMinor things:\n- \"3. Preliminary.\" --> \"3. Preliminary\"?\n- Line 66: \"First, Different\" --> \"First, different\".\n- Line 68: \"we focus on integrating the proposed a general\" --> \"we focus on integrating the proposed general\"?\n- Line 83: \"the dropouts based\" --> \"the dropout methods based\"?\n- Line 139: \"The learning of an MIL can be achieved\" --> \"The learning of a MIL model can be achieved\"?\n- Line 158: \"In an MIL framework\", I would personally write \"a MIL framework\" instead, since I say \"MIL\" like the word \"mill\", but maybe that's just me.\n- Line 137: \"reduce gradient direction errors in learning trajectory\" --> \"reduce gradient direction errors in the learning trajectory\"?\n- Figure 4 caption: \"smallest GDE, training loss , and\" --> \"smallest GDE, training loss, and\".\n- Line 205: \"(e.g.Dropout)\" --> \"(e.g., Dropout)\"?\n- Line 269: \"In a previous investigation, the attention map, which indicates the importance of each instance, is obtained from Eq. (11)\", \"previous investigation\" --> \"previous iteration\"? Or what is this previous investigation, a previous paper?\n- Line 290: \"Our MIL-Dropout only adds additional two hyperparameters\" --> \"Our MIL-Dropout only adds two additional hyperparameters\"?\n- Line 299: \"The complexity is substantially reduced to the fact that O(N D(l)) due to K is typically much smaller than N\", I don't understand, reformulate?\n- Line 317: \"MIL aggregators and their variants to valid\" --> \"MIL aggregators and their variants to validate\".\n- Line 644: \"A.1.1. GRDIENT MEASURE EXPERIMENT\", typo.\n- Figure 5 caption, \"and similarity instance S\" --> \"and similarity instances G\"?", "questions_for_authors": "1. Could you add at least one more WSI-level dataset/task? E.g. another TCGA dataset, or the PANDA dataset?\n\n2. Could you add results when using a pathology foundation model as patch-level feature extractor, e.g. UNI? Does MIL-Dropout consistently improve the performance also in this setting?\n\n\n\n\nJustification of overall recommendation:\n\nQuite interesting and solid paper. Nothing groundbreaking, but the proposed MIL-Dropout could definitely be at least quite useful. However, I think the current experimental evaluation for WSI-level tasks should be extended a bit. Thus, I am currently leaning towards reject.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mTntRpFSNb", "reviewer": "Reviewer_hvzj", "summary": "This paper investigates the effectiveness of Dropout in Multiple Instance Learning (MIL), particularly for histopathological whole-slide image classification. To address the overfitting issue in conventional two-stage MIL training (feature extraction + aggregation) caused by noisy embeddings and weak supervision, the authors propose a counter-intuitive MIL-Dropout approach: By discarding top-k important instances within bags, this method reduces gradient directional errors and converges to flatter optima. The framework contains two novel modules: 1) An averaging-based attention mechanism for efficiently determining instance importance, and 2) A query-based instance selection strategy for identifying instances to discard. Experiments on five MIL benchmark datasets and two WSI datasets demonstrate that MIL-Dropout significantly enhances existing MIL methods’ performance with negligible computational overhead.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Thank you very much for taking the time to review our work and for providing such valuable feedback. In response to the reviewer's concerns, we address the following point:**\n\n1. **Related Literature:** \n\n   Regarding the sampling-related papers mentioned by the reviewer, our method differs, and it is orthogonal to those methods, allowing for integration with them. In the revised version of our paper, we will cite these papers and include an extended discussion in the related works section.\n\n2. **Computational Analysis.** \n\n   Our MIL dropout incurs a negligible computational overhead during training, similar to other dropout methods, and it is not used during inference.\n\n   As mentioned in the manuscript (refer to the complexity analysis), the proposed dropout is not used during inference, adds no extra parameters, and has a negligible impact on training time.\n\n    |Based on Camelyon16(one epoch)|FLOP(M)|Params(K)|Training time(sec)|Inference time(sec)|\n    |---|---:|---:|---:|---:|\n    |AEML|872.26|174.27|5.86|3.06|\n    |+ Dropout|872.26|174.27|5.86|3.06|\n    |+ Our MIL Dropout（K=20,G=10）|872.26|174.27|5.86|3.06|\n\n\n3. **UNI Feature.**  \n   Consistent with requirements from other reviewers, we have replicated the relevant experiments here for clarity. Because UNI was pre-trained on TCGA and Camelyon data—raising the possibility of data leakage—we conducted additional experiments using the independent EBRAINS dataset. The results of these experiments are shown below.\n    |Model|Accuracy|F1|Δ Accuracy|Δ F1|\n    |:-|-:|-:|-:|-:|\n    |ABMIL|65.4|68.7|—|—|\n    |+MIL Dropout|70.4|73.2|+ 5.0|+ 4.5|\n    |TransMIL|67.4|74.4|—|—|\n   |+MIL Dropout|71.3|79.4|+ 3.9|+ 5.0|\n    |DSMIL|67.4|74.4|—|—|\n   |+MIL Dropout|69.3|76.0 |+ 1.9|+ 1.6|\n   |DTFD|53.4|63.6|—|—|\n   |+MIL Dropout|64.8|69.8|+ 11.4|+ 6.2|\n   |ILRA|64.8|74.4|—|—|\n   |+MIL Dropout|66.8|76.0|+ 2.0|+ 1.6|\n   |CAMIL|60.7|68.5|—|—|\n   |+MIL Dropout|69.1|75.6|+ 8.4|+ 7.1|\n   |R2T|57.9|66.1|—|—|\n   |+MIL Dropout|64.8|69.8|+ 6.9|+ 3.7|\n\n   Although our MIL dropout can still offer improvements when the better patch features are implemented, the performance gains may be less substantial (about 1.5 to 0.5). Nevertheless, as demonstrated by the experiments, foundation model features often do not perform optimally on private datasets, making our approach particularly suitable for such scenarios.\n\n4. **Visualization**  \n   Although MIL dropout dynamically selects the top-k and similar instance group G in each training iteration, our MINIST toy experiments (using the target instance \"9\") confirm that both the top-k selection and G effectively focus on the digit \"9\". We plan to include these visualizations in the supplementary material.\n\n5. **Additional Experiments**  \n   Currently, our experiments primarily utilize shallow feature extractors. We also attempted to integrate MIL dropout into the classifier (add more linear layers for integrating MIL Dropout) following the MIL module(actually the same thing we felt), and the performance remained largely unchanged. Furthermore, integrating MIL dropout into a transformer architecture was unsuccessful—likely because MIL dropout interferes with layer normalization. \n\n    |Model (runing 5 times on CAMELOYON16 Imagenet)|Accuracy|F1|AUC|\n    |:--|--:|-:|-:|\n    |TransMIL|84.7|83.3|86.5|\n    |+MIL Dropout in shallow extractor|86.0|84.9|89.4|\n    |+MIL Dropout in classifier |87.6|82.5|88.7|\n    |+MIL Dropout in Transformer|64.5|52.5|52.5|\n    \n   **MIL dropout fails to converge for integration into transformer blocks.**\n\n5. **Extra Comparision:**\n\n   Our MIL dropout can be seamlessly integrated with other MIL methods, as it is orthogonal to them. We conducted experiments using our patch features and observed performance improvements, further demonstrating the flexibility and effectiveness of our dropout.\n   |Model (runing 3 times on CAMELOYON16 Imagenet)|Accuracy|F1|AUC|Δ Accuracy|Δ F1|Δ AUC|\n   |:---|-:|---:|----:|---:|----:|----:|\n   |PAM[1]|85.0|83.2|86.7|—|—|—|\n   |+MIL Dropout|86.2|84.3|87.7 |+1.2|+1.1|+1.0|\n   |DPSF[2]|—|—|—|—|—|—|\n   |[3]|—|—|—|—|—|—|\n \n\n> **[2][3]: The full open-source code has not been released, and we believe our MIL dropout can be integrated into these methods.**\n---\n### **We appreciate your time and hope that our response addresses your concerns. We will also add a discussion of the literature references/citations (as you mentioned) in the revision. If you have any further questions, please let us know.**\n---\n\n**References**  \n[1].Unleash the Power of State Space Model for Whole Slide Image with Local Aware Scanning and Importance Resampling (TMI 2024) \n\n[2].Dynamic Policy-Driven Adaptive Multi-Instance Learning for Whole Slide Image Classification (CVPR 2024)\n\n[3].Boosting Whole Slide Image Classification from the Perspectives of Distribution, Correlation and Magnification. (ICCV 2023)", "claims_and_evidence": "Yes, the claims made in the manuscript are largely supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "Yes, the proposed method(s) and/or evaluation criteria (e.g., benchmark datasets) are appropriately justified for the current problem or application.", "theoretical_claims": "Yes, I have examined the validity of the theoretical claims underlying the proposed approach as presented in the article, primarily the Top-K instance selection method described in Section 4 and the associated query mechanism for selecting G number of instances. After examining the references cited in that section (Woo et al., 2018; Park et al., 2018; Tu et al., 2019a; Ilse et al., 2018), I confirm that the theoretical basis of the proposed method is logically sound and free from issues.", "experimental_designs_or_analyses": "Yes, I have checked the validity of experimental designs and analyses from Section 6.1 to Section 6.4. The experimental design and analysis conducted for the proposed method in the article are methodologically sound and empirically valid.", "relation_to_broader_scientific_literature": "The MIL-Dropout proposed in this paper enhances existing MIL models by discarding Top-K important instances, whose fundamental contribution lies in establishing a novel instance sampling strategy. This aligns with findings from prior studies (e.g., [1][2][3][4]) that proposed diverse instance sampling strategies to improve model performance. However, it should be emphasized that the proposed sampling strategy primarily operates through shallow feature extractors in conventional MIL frameworks, constituting a methodological distinction from earlier approaches.\n\n[1] Unleash the Power of State Space Model for Whole Slide Image with Local Aware Scanning and Importance Resampling (TMI 2024)\n[2] Dynamic Policy-Driven Adaptive Multi-Instance Learning for Whole Slide Image Classification (CVPR 2024)\n[3] MG-Trans: Multi-Scale Graph Transformer With Information Bottleneck for Whole Slide\nImage Classification (TMI 2023)\n[4] Boosting Whole Slide Image Classification from the Perspectives of Distribution, Correlation and Magnification. (ICCV 2023)", "essential_references_not_discussed": "To the best of my knowledge, the key technical contribution of the proposed MIL-Dropout in this paper is unique: By discarding Top-K important instances through shallow feature extractors in existing MIL frameworks, it facilitates model convergence towards superior performance. Although prior work (e.g., [1] proposing bag filtering eliminate partial instances, [2] directly discarding top instances) has made analogous contributions, the novel dropping strategy developed in this study operates within inner neural network layers, demonstrating distinct technical innovation.\n\n[1] Dgmil: Distribution guided multiple instance learning for whole slide image classification. (MICCAI 2022)\n[2] Bi-directional weakly supervised knowledge distillation for whole slide image classification(neurips 2022)", "other_strengths_and_weaknesses": "Strengths:\n1.The paper introduces a novel application of dropout in MIL by targeting the top-k important instances, diverging from traditional neuron-level dropout.\n2.The proposed method is designed as a plug-and-play module that can be seamlessly integrated into the majority of existing MIL frameworks\n\nWeaknesses:\n1.The proposed method exhibits dependence on parameter selection (K and G), which may hinder its generalization to broader WSI datasets. When encountering new datasets, re-optimization of these parameters may be required to achieve optimal performance.\n2.While 10-fold cross-validation was applied on five MIL benchmarks, this was not conducted for the WSI datasets, which may hinder the evaluation of the proposed MIL-Dropout’s robustness in real-world clinical diagnostic scenarios.", "other_comments_or_suggestions": "Some minor issues:\n1.The computational complexity analysis lacks concrete experimental validation. For instance, there is no quantification of the additional memory overhead when integrating MIL-Dropout into existing frameworks like ABMIL or TransMIL, nor is there systematic evaluation of its impact on training and inference time.\nThe experiments did not incorporate the features from the latest foundation model (e.g., UNI and CONCH). These advanced features may narrow the performance gap between existing MIL methods with and without MIL-Dropout integration, thus diminishing the significance of this work.", "questions_for_authors": "1.Can the K instances selected by MIL-Dropout and their associated G instances be visualized? Do these selected instances exhibit pathological relevance or interpretability?\n2.Is the proposed MIL-Dropout limited to shallow feature extractors in existing MIL methods? For example, can it be integrated into other modules like the Transformer in TransMIL?\n3.What advantages does MIL-Dropout offer over existing instance sampling strategies (e.g., test-time sampling in [1], dynamic sampling in [2], Bag Filter in [3])? Can MIL-Dropout outperform these approaches?\n\n[1] Unleash the Power of State Space Model for Whole Slide Image with Local Aware Scanning and Importance Resampling (TMI 2024)\n[2] Dynamic Policy-Driven Adaptive Multi-Instance Learning for Whole Slide Image Classification (CVPR 2024)\n[3] Boosting Whole Slide Image Classification from the Perspectives of Distribution, Correlation and Magnification. (ICCV 2023)", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NIrgq84PMx", "reviewer": "Reviewer_QeRD", "summary": "The authors investigate the effectiveness of dropout in Multiple Instance Learning (MIL), asserting that removing the top-k most important instances within a bag enhances performance and generalization, even under noise attacks. Additionally, they introduce a dropout method, termed MIL-Dropout, which integrates into existing MIL frameworks.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for taking the time to review our work and for providing many valuable comments. Please find our detailed responses to the reviewer’s concerns below:\n\n---\n\n\n1. **Unclear Contributions:** \n  Our investigation reveals two key findings:  \n    - **Finding 1:** DropInstance produces flatter local minima and improves generalizability compared to DropNeuron.  \n    - **Finding 2:** DropInstance mitigates gradient direction errors during the learning trajectory, thereby enhancing data fitting and \n  overall performance, which we have summarised in the manuscript.\n\n2. **Insufficient Literature Review:** \n\n    We agree with the reviewer’s comment regarding the limited discussion of related work. In the revised version, we will include a more comprehensive review of relevant studies.\n\n\n3. **MIL Background:** \n\n   Due to space limitations in the main paper, we will provide additional background information on Multiple Instance Learning (MIL) in the supplementary material.\n\n4. **Experimental Results:** \n\n    The results reported in Tables 1 and 2 reflect the performance changes achieved by incorporating our proposed dropout method, which is designed to enhance the performance of existing MIL approaches. We will also add further visualizations in the supplementary material to illustrate the differences in model performance before and after applying our method.\n\n5. **Legend Clarification:** \n\n    We will revise the legend in the updated version to improve clarity.\n\n\n6. **Insufficient Evaluation:** \nIn line with suggestions from other reviewers, we have conducted additional experiments, as shown below.\n\n    - **Because UNI was trained on TCGA and Camelyon, using those features risks data leakage. To avoid this, we evaluate UNI features on the EBRAINS dataset by running the experiment three times**.  \n    |Model|Accuracy|F1|Δ Accuracy|Δ F1|\n    |:--|--:|---:|---:|--:|\n    |ABMIL|65.4|68.7|—|—|\n    |+MIL Dropout|70.4|73.2|+ 5.0|+ 4.5|\n    |TransMIL|67.4|74.4|—|—|\n   |+MIL Dropout|71.3|79.4|+ 3.9|+ 5.0|\n    |DSMIL|67.4|74.4|—|—|\n   |+MIL Dropout|69.3|76.0 |+ 1.9|+ 1.6|\n   |DTFD|53.4|63.6|—|—|\n   |+MIL Dropout|64.8|69.8|+ 11.4|+ 6.2|\n   |ILRA|64.8|74.4|—|—|\n   |+MIL Dropout|66.8|76.0|+ 2.0|+ 1.6|\n   |CAMIL|60.7|68.5|—|—|\n   |+MIL Dropout|69.1|75.6|+ 8.4|+ 7.1|\n   |R2T|57.9|66.1|—|—|\n   |+MIL Dropout|64.8|69.8|+ 6.9|+ 3.7|\n\n    - **Below are the results on Camelyon16 (Imagenet pre-trained) using the mentioned latest method. We add MIL‑Dropout via two linear layers before feeding features into the MIL module**.  \n   |Model|Accuracy|F1|AUC|Δ Accuracy|Δ F1|Δ AUC|\n   |:-|--:|---:|--:|--:|--:|--:|\n   |ILRA|85.8|81.4|88.4|—|—|—|\n   |+MIL Dropout|87.2 |86.4 |90.1 |+1.4|+5.0|+1.7|\n   |CAMIL|84.8|82.6|86.8 |—|—|—|\n   |+MIL Dropout|86.4|85.7|91.2 |+1.6|+3.1|+4.4|\n   |R2T|83.1|81.6|84.5|—|—|—|\n   |+MIL Dropout|85.6|84.7|87.4 |+2.5|+3.1|+2.9|\n\n    - **We evaluated our method on survival prediction using two TCGA datasets(LUAD and BRCA), following the protocols of [4,5]**.\n   | Model       | TCGA‑LUAD | +MIL-Dropout | Δ | TCGA‑BRCA | +MIL-Dropout| Δ |\n   |--|-:|-:|--:|--:|---:|----:|\n   | ABMIL       | 65.7 | 67.7 | +2.0  | 72.8   | 75.2   | +2.4 |\n   | DSMIL       | 61.4  | 63.4 | +2.0   | 68.8  | 72.1  | +3.3 |\n   | TransMIL    | 64.3  | 67.2| +2.9   | 72.1 | 74.7    | +2.6  |\n   | DTFD (AFS)  | 62.0 | 65.3  | +3.3  | 71.1  | 73.6  | +2.5 |\n   | DTFD (MaxS) | 65.9 | 68.3   | +2.4  | 72.8      | 75.7  | +2.9       |\n\n    - **Our MIL dropout can be seamlessly integrated with other MIL methods, as it is orthogonal to them. We conducted experiments using our patch features and observed performance improvements, further demonstrating the flexibility and effectiveness of our dropout.**\n   |Model (runing 3 times on CAMELOYON16 Imagenet)|Accuracy|F1|AUC|Δ Accuracy|Δ F1|Δ AUC|\n   |:---|----:|---:|----:|---:|----:|----:|\n   |PAM[1]|85.0|83.2|86.7|—|—|—|\n   |+MIL Dropout|86.2|84.3|87.7 |+1.2|+1.1|+1.0|\n   |DPSF[2]|—|—|—|—|—|—|\n   |[3]|—|—|—|—|—|—|\n\n> **[2][3]: The full open-source code has not been released; we believe our MIL dropout can be integrated into these methods.**\n\n---\nWe hope our response addresses your concern. please let us know if you have any questions. Thanks!\n---\n**References**  \n[1].Unleash the Power of State Space Model for Whole Slide Image with Local Aware Scanning and Importance Resampling (TMI 2024) \n\n[2].Dynamic Policy-Driven Adaptive Multi-Instance Learning for Whole Slide Image Classification (CVPR 2024)\n\n[3].Boosting Whole Slide Image Classification from the Perspectives of Distribution, Correlation and Magnification. (ICCV 2023)\n\n[4]. Jaume, G., et al. (2024). Modeling dense multimodal interactions between biological pathways and histology for survival prediction.CVPR.  \n[5]. Song, A. H., et al. (2024). Multimodal prototyping for cancer survival prediction.ICML.", "claims_and_evidence": "Based on the experimental analysis, the manuscript claims that MIL-Dropout enhances the performance of existing MIL methods across five MIL benchmark datasets and two WSI datasets while incurring negligible computational cost.", "methods_and_evaluation_criteria": "Yes, proposed methods supports the claims, but required more evaluation.", "theoretical_claims": "Yes.", "experimental_designs_or_analyses": "The experimental analysis on five MIL benchmark datasets and two WSI datasets demonstrates that MIL-Dropout enhances the performance of existing MIL methods.", "relation_to_broader_scientific_literature": "Integrating the proposed dropout approach can enhance the performance of Multiple Instance Learning (MIL).", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "- The manuscript fails to clearly articulate its key findings and contributions.  \n- The related work and literature survey are highly limited. The authors should engage more with existing research, outlining relevant findings that informed their approach.  \n- Additional background information on Multiple Instance Learning (MIL) should be included in the introduction.  \n- The manuscript does not effectively present the impact or effectiveness of dropout in MIL or how performance varies as a result.  \n- In Figure 4 (Right), the legend is missing.", "other_comments_or_suggestions": "- In Figure 4 (Right), the legend is missing.", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper focused on multiple instance learning (MIL) methods which have been very popular in recent years especially for digital pathology. The paper started with a preliminary study on a digital pathology dataset. Using gradient direction error, it showed that dropping out top-k important instances (with the highest attention values) led to a better fitting of the model. The paper proceeded and proposed a method to select top-k important instances, as well as other instances similar to the top-k, and drop these instances for bag representation calculation. The method is generic and can be applied to any MIL framework. Through experiments on classic MIL datasets and digital pathology datasets (Whole slide prediction), the proposed dropout method is shown to boost the performance of various existing MIL methods. 

The scores for this paper vary significantly. Some reviewers raised several concerns: (1) the experiments in digital pathology were limited and not sufficiently convincing, particularly in the context of leveraging state-of-the-art foundational model features; (2) the proposed method appears sensitive to hyperparameters such as K (an ablation study on key hyperparameters was provided in the original submission); and (3) the preliminary study was conducted on only a single dataset, making it less compelling. Some other reviewers were enthusiastic about the work. They found the core idea to be novel and appreciated its design as a plug-and-play module that can be seamlessly integrated into existing MIL frameworks.

After carefully reviewing the paper, the reviewers’ comments, and the authors’ rebuttal, the Area Chair leans toward recommending acceptance. While it is acknowledged that the evaluation on digital pathology is not as robust as it could be, the authors have made commendable efforts in the rebuttal phase. They provided additional comparisons using state-of-the-art MIL methods and incorporated features from foundational models. If these results are integrated into the final version of the paper, the empirical validation would be significantly strengthened.

Moreover, the paper offers a refreshing perspective compared to many current works in digital pathology. It focuses on the MIL methodology itself, introduces valuable insights, and proposes a simple yet effective approach that is easy to implement and potentially widely applicable across existing MIL frameworks. This conceptual contribution appears to outweigh its empirical limitations.
Decision: Accept (poster)