Sm: enhanced localization in Multiple Instance Learning for medical imaging classification
Authors: Francisco M Castro-Macías
Pablo Morales-Alvarez
Yunan Wu
Rafael Molina
Aggelos Katsaggelos
Venue: NeurIPS 2024 poster (2024)
Keywords: Multiple Instance Learning
Transformers
Graph Neural Networks
Medical Imaging
Weakly Supervised Learning
Primary area: machine_learning_for_healthcare
TL;DR: We draw attention to the localization task in MIL and propose Sm, a principled operator to account for local interactions among instances that yields enhanced performance.

Abstract: Multiple Instance Learning (MIL) is widely used in medical imaging classification to reduce the labeling effort. 
While only bag labels are available for training, one typically seeks predictions at both bag and instance levels (classification and localization tasks, respectively). Early MIL methods treated the instances in a bag independently. Recent methods account for global and local dependencies among instances. Although they have yielded excellent results in classification, their performance in terms of localization is comparatively limited. We argue that these models have been designed to target the classification task, while implications at the instance level have not been deeply investigated. Motivated by a simple observation -- that neighboring instances are likely to have the same label -- we propose a novel, principled, and flexible mechanism to model local dependencies. It can be used alone or combined with any mechanism to model global dependencies (e.g., transformers). A thorough empirical validation shows that our module leads to state-of-the-art performance in localization while being competitive or superior in classification. Our code is at https://github.com/Franblueee/SmMIL.

Sm: enhanced localization in Multiple Instance Learning for medical imaging classification
Francisco M. Castro-Macias CITIC-UGR
Pablo Morales-álvarez Dept. of Statistics and Operations Research
Dept. of Comp. Science and A. University of Granada
CITIC-UGR
University of Granada
Yunan Wu
Rafael Molina
Dept. Elect. and Comp. Engineering Northwestern University
Dept. of Comp. Science and A. University of Granada
Aggelos K. Katsaggelos
Dept. of Elect. and Comp. Engineering Northwestern University
Abstract
Multiple Instance Learning (MIL) is widely used in medical imaging classification to reduce the labeling effort. While only bag labels are available for training, one typically seeks predictions at both bag and instance levels (classification and localization tasks, respectively). Early MIL methods treated the instances in a bag independently. Recent methods account for global and local dependencies among instances. Although they have yielded excellent results in classification, their performance in terms of localization is comparatively limited. We argue that these models have been designed to target the classification task, while implications at the instance level have not been deeply investigated. Motivated by a simple observation that neighboring instances propose novel, principled, and flexible mechanism to model local dependencies. It can be used alone combined with any mechanism model global dependencies (e.g., transformers). A thorough empirical validation shows that our module leads state-of-the-an performance in localization while being competitive or superior in classification. Our code is at sittps://githhi htw.g.ttt..oo..o....ooooeooete.e www./oraannusse/enneoaoo
likely to have the same label we
Introduction
Over the last decades, medical imaging classification has benefited from advances in deep learning (44.
However, the performance of these methods drops when the number of labeled samples low, which is common in real- weal-world medical scenarios (1) To overcome this, Multiple Instance Learning (MIL) has emerged as a popular weakly supervised approach [14 g
35
In MIL, instances arranged in bags. At train time, label is available for the entire bag, while instance labels remain unknown. The goal to train a method that, given a test bag, can predict both at bag and instance levels (classification and localization tasks, respectively). This paradigm well suited to the medical imaging domain (28) In cancer detection from Whole Slide Images (WSIs), the WSI represents the bag, and the patches are the instances. In intracranial hemorrhage detection from Computerized Tomographic (CT) scans, the full scan represents the bag, and the slices at different heights the instances. In these scenarios, making accurate predictions of instance
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

labels is extremely important from clinical viewpoint, as it translates into pinpointing the location of the lesion 7
Most successful approaches in MIL build on the attention-based pooling 17 a permutation -nnvaaaaioonn operator that assigns an attention value to each instance independently. This method has been extended in different ways while maintaining the permutation-invariant property [2] 25 39). The aforementioned works pose a problem: the dependencies between the instances, which are important when making a diagnosis, are ignored. To account for this, TransMIL 32 proposed to model global dependencies using transformer encoder. The idea is to use the self-attention mechanism to introduce interactions between each pair of instances. Based on it, other transformer-basee approaches have emerged, also focusing on global dependencies [9] [2 37.. More recently, several works have also incorporated natural local interactions, which are those between neighboring instances [14 40 41).
Although these methods accounting for dependencies have resulted in excellent performance the bag level, the evaluation at the instance level has received less attention and the results are not comparatively good so far, see the very recent [14]. In this work, we argue that recent MIL methods have been designed with the classification task in mind, and we propose a new model that focuses on both the classification and localization tasks. Specifically, we propose a novel and theoretically grounded mechanism to introduce local dependencies, hereafter referred to as the smooth operator Sm. This is flexible module that can be used alone on top of classical MIL approaches, or in combination with transformers to also account for global dependencies. In both cases, we show that the proposed operator achieves state-of-thee auf-the-art localization results while being competitive in classification. We compare against total amount of eight methods, including very recent ones [14] 40. We utilize three different datasets of different nature and size, covering two different medical imaging problems (cancer detection in WSI images and hemorrhage detection in CT scans).
Our main contributions are: (i) we provide unified view of current deep MIL approaches; (ii) we propose a principled mechanism to introduce local interactions, which is a modular component that can be combined or not with global interactions; and (iii) we evaluate up to eight state-of-the-art MIL methods on three real-world MIL datasets in both classification and localization tasks, showing that the proposed method stands out in localization while being competitive or superior in classification.
2 Related work
In this work, we tackle the localization task in deep MIL methods using existing concepts and techniques from deep MIL and Graph Neural Networks (GNNs) theory.
Deep Multiple Instance Learning. As explained by Song et al. [4]. deep MIL methods can be divided into two broad categories, namely instance-based or ambedding-based depending the level at which a specific aggregation operator is applied. In this paper, we focus on the umbedding-basee category, and in particular attention-basel ones.
Isse et al. proposed attention-based pooling to weigh each instance in the bag. To improve it, different modifications were proposed, including the use of clustering layers 255, grouping the instances in pseudo-bags (39), and using similarity measures to critical instances to compute the attention values (21). However, these methods ignore the existing dependencies between the instances in a bag. To address this, Shao et al. (32) proposed to use a transformer-bast
architecture and
the PPEG position encoding module. This has been extended with different transformer variations, including the deformable transformer architecture [22), hierarchical attention (37], and regional sampling [9. Recently, these methods have been improved include spatial information in different ways, including the use of a Graph Convolutional Network (GCN) before the transformer 41
nnighbor-constraing attention module [14], and a spatial-encoding transformer architecture
40
In the studies mentioned above, the objective is to obtain increasingly better bag-level results, while the evaluation at the instance level is usually performed qualitatively. In contrast, our work addresses both the instance localization task and the bag classification task, as both are of great importance for making diagnosis. Moreover, our work is not limited to WSI classification; it is also valid for other medical imaging modalities.
Graph Neural Networks. Our motivation - that neighboring instances are likely to have the same label - a well-established assumption within the machine learning community, often referred to as the cluster assumption [10 31). Since leveraged in 1984 by Ripley [30] in the context of spatial

---

X
X
f1
Giobal leacciioee
Deep Mil. unthod
Namaae
Agz
Globed it-
tornctioce
Local In- lrsactioe
XW
XX
XW
(a) General deep model.
MIL. (b) Instances are treated in- (c) Only global interac- (d) Global and local inter- actions.
dependently.
tions.
Figure I: (a) Unified view of deep MIL. models. Depending on how instances interact with each other in (a), we devise three different families of methods: (b), (c), (d).
statistics, it has been extensively used in spectral clustering 227, semi-supervised learning on graphs 3 and recently in GNNs 200 Our work builds upon seminal works in these areas.
The proposed smooth operator is derived considering Dirichlet energy minimization problem, similar to the work by Zhou and Schölkopf [2] and Zhou et al. [43]. This approach has been employed in recent years to obtain new GNN models. including the p-Laplacian layer (15), and PPNP layer [16]. Moreover, the Dirichlet energy has been studied in the context of GNNs to analyze the over-smoothing phenomenon 6 23
this regard, our bound on the decrease of the Dirichlet energy is analogous to the result derived by Li et al. [23] to study over-smoothing for GCNs. Our result, however, holds for the proposed mechanism, of which the graph convolutional layer is special type.
3 Background: A unified view of deep MIL approaches
We first describe the binary MII. problem tackled in this paper. Then, we provide unified view of the most popular deep MIL methods. As explained in Sec.. 2 we focus on embedding- based approaches.
In MIL, the training consists of pairs of the form (X, Y), where X NR is bag of instances and Y E {0, 1} is the bag label. We write X = xi...... x E RNP,, where x,, E R are the instances. Each instance X,, is associated to label yn {0, 1}, not available during training. It is assumed that = max ........ .................. J. i.c., a bag X is considered positive if and only if there is at least positive instance in the bag.
Given a previously unseen bag (e.g.. a medical image), the goal at test time is to: i) predict the bag label (classification task) and ii) obtain predictions or estimates for the instance labels (localization task). In general, deep MIL models output bag-level prediction Y, well as instance-level scalars fn that are used for instance-level prediction. This general process depicted in Fig. la In many approaches, these fn are the so-called attention values (e.g., ABMIL [17., TransMIL 33 CAMIL
(14), but they can be obtained different ways (c.g., through GraphCAM in GTP 4I).. Within the general process in Fig. la. deep MIL models can be categorized into three families, depending and Fig. Id
how instances interact with each other, see Fig. 1b. Fig. Ic
In the first family, shown in Fig. 1b the instances are encoded independently and then aggregated. The well-known ABMIL ABMIL, while still encoding each instance independently 25 39). ABMIL, on which we will rely to introduce our model, is depicted Fng 3al First, a bag embeddings H [h...... .n] ] E NDD is obtained by applying neural network independently each instance. Then, the iitenniion-bseed pooling computes the attention values and the bag embedding z according to
fits in this paradigm. Subsequent works introduce slight modifications to
F == t-nh (HW
f= Fw
AttPool (H) H Softmax (f),
where W WEDLXD weRL trainable weights. Last, Ý is obtained by applying linear classifier
on z.
The second family accounts for global interactions between instances, possibly long-range ones, see Fig. lc These works treat instances as tokens that interact through the self-attention mechanism. This way. global interactions between instances are learned. One of the most popular approaches in this family is TransMII. 32), which was later extended in different directions 9 22). The third
family complements the previous one with local interactions defined by fixed neighborhood, see
3

---

(a) Lahelled patches in WSI.
0
(b) Labelled slices in CT scan.
Figure 2: WSIs divided into patches. CT scans are provided as slices. They often show spatial dependencies: in a WSI, a patch is usually surrounded by patches with the same label, while in CT sean, a slice is usually surrounded by slices with the same label. The red color indicates malignan/hemoorhage patches/slices.
Fig. ld They differ how local interactions are represented, e.g., as a graph in CAMIL and GTP [4]] or using position-encoded feature map in SETMIL. 44
In most of these works, the localization task is assessed qualitatively, e-g., by visually comparing the attention maps. This contrasts with the classification task, which is always evaluated quantitatively. As evidenced by Fourkioti al. [14], this has translated into comparatively poor performance in terms of localization. We notice that current models have been designed to target the classification task, and they excel at that. However, their model design is not as careful about the instance-level implications. For example, CAMIL [4] does not leverage any local information to obtain the instance-leve attention values. Indeed, from their Eq. (8) one deduces that the a, values are obtained from the tile representations t., which have not undergone any local interaction. Observe that local interactions take place in Eq. (4) and Eq. (5) in their paper, but these only affect the bag-level predictions, not the instance-leved ones. Similarly, GTP [41] introduces local interactions through an off-the-shelf graph convolutional layer, the effect of which not investigated at the instance level. the following section, we propose principled approach to account for meaningful local interactions based on the Dirichlet energy. The idea is motivated by a natural property often observed in the instance-level labels of medical images: the similarity between neighboring instances.
4 Method: Introducing smoothness in the attention values
In medical imaging, instance labels are a priori expected to exhibit local dependencies with their neighboring instances: instance is likely to be surrounded by instances with the same label, see Fig. Recall that attention values are commonly used as a proxy to estimate these labels, so they should inherit this property. Based these observations, our intuitive idea is to favor a smoothness property on the attention values. To this end, Sec. [4. formalizes the notion of smoothness through the Dirichlet energy. Sec. 4.2] presents the proposed smoothing operator Sm, which encourages smoothness as well as fidelity to the original signal. Sec. [.3 proposes how to leverage Sm in the context of MIL, both in combination with global interactions (via transformers), and without them. We will build on top of the well-known and simple ABMIL to isolate the effect of Sm and avoid over-sophisticated models.
4.1 Modelling the smoothness
We represent each bag as a graph, where the nodes are the instances and the edges represent the spatial connectivity between instances. Formally, we suppose that each bag X E IR NDD has been assigned an adjacency matrix A [Ay]] E N.N defined by Auj > O instances x; and x, are neighbors, and A.j 0 otherwise. We assume that the adjacency matrix is symmetric, i.e. A.j Aji-
The Dirichlet energy a well-known functional that measures the variability of function defined on graph 42 43 In our case, we think of this function as the attention values E V recall As we shall see below, it will necessary to define the Dirichlet energy for multivariate graph functions. Given a multivariate graph function U = [ui,..., uy] E RNND defined on the bag graph, the Dirichlet energy of U is given by
Fig. la
Ed (U) = 1 E ΣN Auj |lu uy 11 = Trace (U LU).
(3)

---

where I1-1l2 denotes the Euclidean norm, L is the graph Laplacian matrix L = D - D, E R is the degree matrix, D Diag (D).... Dx), D,, = Σ, Ans- When D we obtain the definition for univariate graph functions, such as the attention values f.
Bounding &D on the attention values. In most deep MIL. approaches, the attention values f are obtained by applying neural network to instance-level features. One example is ABMIL (17]. which uses two-layer perceptron defined by [Eq. Noting that tanh is a Lipschitz function with Lipschitz constant equal to 1,, we arrive at the following chain of inequalities
ED (f) S 1l ED (F) S ||w 12 [I2 ED (H):
where |I-Ilp denotes the spectral norm. A more general result holds in the general case of an arbitrary multi-layer perceptron, see Appendix A for proof. The above chain of inequalities tells us that if we want ED (f) to be low, we can act on f itself or on previous layers (e.g, on or on H), constraining the norm of the trainable weights to remain constant. This constraint can be achieved using spectral normalization (26), and we study its influence in SSec. B.3 In the next subsection, we propose an operator that can be used on any of these levels (f, F, H) to reduce the Dirichlet energy of its output.
4.2 The smooth operator
Our goal now turns into finding operator Sm RN NxD that, given a bag graph multivariate
function U E RNDD, returns another bag graph multivariate function Sm (U) E R such that its Dirichlet energy is lower without losing the information present in the original U. Following seminal we cast this as an optimization problem,
works 42 43
Sm (U) = arg ming & (G),
(5)
3 {G) aED (G) (1 |T - GI
where a E [0, 1) accounts for the trade off between both terms, and ||-I|p- denotes the Frobenius norm. The first term in the above equation penalizes functions with too much variability, while the second term penalizes functions that differ too much from the original U. Note that this can be interpreted as a maximum a posteriori formulation, where the first term corresponds to the prior distribution and the second the observation model, see (30]. The objective function & is strictly convex, and therefore admits a unique solution, given by
Sm (U) (1+ -)) U,
(7)
where 7 = aa(1 a). Unfortunately, the expression in Eq. 77. although elegant, incurs prohibitive computational and memory costs, especially when the number of instances in the bag is large (which the case of WSIs). Instead, we can take an iterative approach, defining Sm (U) = G(T), with
G( U: Gt) = a (I L) G(t - 1) (1 (1-a) U, E (1..... TT.
(8)
As demonstrated by Zhou et al. Eq. 7 As studied by Gasteiger et al. [16]. it is enough to use small number of iterations T to approximate the exact solution. Therefore, in this work, we will adopt the iterative approach described by[Eq.8 8 Based on previous work [16), we will use T = 10, and a will be set as trainable parameter initialized at a = 0.5. See Sec. for a study on the effects of these hyperparameters and Fig. 12 for visual comparison of the effect that has on the attention maps.
the sequence (G(t)) converges to the optimal solution in
Theoretical guarantees via the normalized Laplacian. We present a result that informs us about the rate at which the Dirichlet energy decreases when applying Sm. Let us define X; max (1 (1.) An E A \ 1003.. where A (Ai.... } are the eigenvalues of the bag graph Laplacian matrix. Then, we have the following inequality,
ED (Sm (U)) S X,ED (U). The proof is inspired by Cai and Wang 66. see Appendix A IF < 1, then the smooth operator effectively decreases the Dirichlet energy. If we replace the Laplacian matrix by the normalized Laplacian matrix, L D I/2LD 1/2 it is known that its eigenvalues lie in the interval [0, 2), and then X; < holds. This motivates the use of the normalized Laplacian in our experiments.
The smooth operator Sm only introduces one parameter to be estimated, a. Also, it is differentiable with respect to its input. Therefore, can be integrated into simple attention- based MII. models, such as ABMIL, to account for local dependencies.
5

---

Sersbal
A
XL
X
Ir
Alt Poalna
Puile:
Xx Nenaton
bxy
hx
XX
(a) ABMIL..
(h) SmAP.
(c) SmTAP.
Figure 3: Smooth Attention Multiple Instance Learning. (a) The well-known model in [7]. which build upon. (b): only local interactions are considered by applying the proposed smooth operator Sm in the aggregation part. (c): both global and local interactions are considered by applying Sm both in the transformer and in the aggregation parts.
4.3 The proposed model
Here we propose how to leverage the operator Sm the context of MIL.. We build on top of the well-known ABMIL. First, we introduce SmAP, which integrates ABMIL with Sm and only accounts for local interactions. Second, we introduce SmTAP, which equips SmAP with a transformer encoder to account for global dependencies. The proposed models are depicted in Fig. 3b and Fig. 3c The details about the architecture we have used can be found in Sec. B.2
SmAP: Smooth Attention Pooling. This is represented in Fig. 3b First, the bag of embeddings H obtained in ABMIL [7], i.e. treating the instances independently. Then, the operator Sm is integrated within the attention pooling. Based on Eg. 4 this can be done on the attention values themselves or on previous representations. We consider three different variants: SmAP-llte, SmAP-mid, SmAAP-early They act, respectively. on f (the attention values themselves), on F (i.e. before entering the last layer), and on (i.e. before entering the attention pooling). Formally,
late: f Sm (tanh (HWT) w)
(10)
mid: = tanh (Sm (HW T)) w,
(11)
early:
AttPool (Sm (H)),
(12)
While SmAP-late and SmAP-mid act on the computation of the attention values, SmAP-carly acts on the embedding that passed the attention-based pooling, see 8 in Appendix C] We use
SmaP- by default. (Sec. 5.3 shows that results do not differ much among configurations.
SmTAP: Smooth Transformer Attention Pooling. This is represented Fig. 3c The only difference with SmAP is that the neural network acting independently on the instance embeddings is replaced by transformer encoder account for global dependencies. Based the idea that smoothness be imposed at previous locations, recall Eq. 4, we propose to also apply Sm to the transformer output:
H Sm (oottmax (a (x())
(X)).
(13)
where k, and e are the standard queries, keys, and values in the dot product self-attention [4. Notice that SmTAP uses Sm two places: the first after the transformer encoder and the second the aggregator. Naturally, one could think of other variants that use Sm in only one place or the other. In Sec. 5.3 ablate these different configurations, leading similar results.
Experiments
We validate the proposed Sm in three medical MII. datasets: RSNA 13, PANDA (5, and CAME- LYON16 (2). We evaluate the effectiveness of our approach by a quantitative and qualitative analysis. All experiments have been conducted under fair and reproducible conditions. Details on the datasets and experimental setup can be found in Appendix B The code is available at httts:://githut com//rrabbleeee///mtee
We compare our approaches with state-of-the-art deep MIL methods. We consider two groups of methods, depending on the presence/absence of a transformer block to model global dependencies. In first group, we include those models that do not use this block: the proposed SmAP, ABMIL [17],
CLAM 255 DSMIL 21), and DITD-MIL 39 The second group consists of models that do use the transformer encoder: the proposed Sm SmTAP, TransMIL. (32] SETMIL. (40,, GTP 41, and
CAMIL [4]. These groups ensure fair comparison in terms of model capabilities and complexity.

---

Table 1: Localization results (mean and standard deviation from five independent runs). The best is in bold and the second-best is underlined. (4)((1) means lower/higher is better. The proposed operator improves the localization results in all three datasets and both with and without global interactions.
ranks first in eight out of twelve dataset-score pairs.
RSNA
PANDA
CAMELYONI6
AUROC (t)
FI (1)
AUROC (1)
RI (1)
AUROC (1)
FI (f)
Rank (J)
SMAP
0.798
0.477 014
0.799g.00s
0.035 cos
0.960a.007
0.840a
1.500o.548
Without
ABMIL
0,8066-912
0. 0.48
0.76%).002
0.6022.co4
0.819
0.7666.000
2.500,
global
CLAM
0.5230.069
0.076o. 154
0.7276.046
0.568p.cas cas
0.8499.044
0.821n.046
4.167 1.929
interactions
DSMIL
0.554p.004
0. 0.1800.000
0.765o.mm
0.598u.coc coc
D. 0.7800.070
0.654p.18a
4.3333..51
DFTD-MIL
0.747a.070
0.453
0. 0.79%,.ml .044
0.637o.006
0.884 .002
0.742p.040
2.500
Pm
0.7670.04
0.4774.020
0.7900..o7
0.6222..11
0.7899.0om
0.6006.com
1,500,
With global
TransMLL
0.732
0.4
0.751..011
0. 0.636..00s oos
0.781 021
0.1277.078
3.083
SETMIL
0.726g.025
0.4388..22
0.774k.u07
0.631u.o10
0.615 a31
0.1344.267
3.667u.816
interactions
GTP
0.736p.017
0.425
0.708
0.030 081
0.4422.0
0.037
3.9171.499
CAMIL
0.760
0.4566.015
0.785p.a11
0.621n.013
0. 0.7-22..2z 022
0.479p.17a
2.833
In Sec. B.3 we report the results of three more methods: DeepGraphSurv [24], PathGCN 11] and IIMMIL [29]. Note that the performance obtained by these methods does not affect the conclusions we will obtain in this section.
In See. I.1 we consider the localization task, In Sec. 5.2
we turn to the classification task. See. 3.3 shows an ablation study on how different uses of the smooth operator affect the proposed model.
5.1 Localization: instance level results
In this subsection, we analyze the ability of each model to predict the label of the instances inside bag. As explained in [nc.. 3] deep MIL models assign a scalar value fn to each instance Xa, see Fig. lal Although these can be obtained in different ways, for simplicity we will refer to them as atlention values. Thus, compare the attention values with the ground truth instance labels, which are available for the test set only for evaluation purposes.
Quantitative analysis. We analyze the performance of each method using the area under the ROC curve (AUROC) and the Fl score. Note that a critical hyperparameter for the latter is the threshold used on fn to determine the label of each instance. To ensure fair comparison, optimal threshold for each method using the validation set. As a general summary, we also report the average rank achieved by each model across metries and datasets.
compute the
The results are shown in Table We find that using Sm provides the best performance overall, placing as the best second-best within each group. Only in RSNA the proposed SmAP is outperformed by ABMIL. We attribute this to the fact that the bag graphs in CT seans are not as complex as in WSIs, and therefore the local interactions are not meaningful. Note that the performance gain is particularly significant on CAMELYONI6, where the bags have a larger number of instances, the graphs are much denser and the imbalance between positive and negative instances is more severe. Notably, SmTAP significantly outperforms SETMIL, GTP, and CAMIL, which also model local dependencies. Contrary to our method, their design is focused bag-level performance and it does not translate into meaningful instance-level properties.
Attention histograms. We examine the attention histograms produced by each model on the CAME- LYON16 dataset. The corresponding figures for RSNA and PANDA can be found in Appendix C In Fig. 4 we represent the frequency with which attention values are assigned to positive and negative instances, separately. An ideal instance classifier would place all the positive instances on the right and all the negative instances on the left. This illustrates why Sm TAP and SmAP achieve such good performance: they concentrate the negative instances to the left of the histogram while succeeding in grouping large part of the positive instances to the right. TransMIL and GTP assign low attention values to both positive and negative instances. CAMIL is able to identify positive instances, but negative instances are assigned from intermediate to high attention values. CLAM and DSMIL assign low attention values to negative instances, but the distribution of the positive instances resembles a uniform and normal distribution, respectively:
Attention maps. To visualize the localization differences, we show the attention maps generated by four of the transformer- based methods in a WSI from CAMELYONI6, see Fig. S Sm TmAP attention
7

---

Podiie furisnn
Now
Antraiinn callr
Abodion
Aoretia ohe
SmAP
ABMIL
CLAM
DSMIL
DFTD-MIL
Sm
TransMIL
SETMIL
GTP
CAMIL
Figure 4: Attention histograms on CAMELYON16. First/second rows show models without/with global interactions. SmAP and SmTAP stand out separating positive and negative instances.
Patch labels
SmTAP
TransMIL
GTP
CAMIL
Figure Attention maps on CAMELYON16. The novel SmTAP produces the most accurate one.
map resembles the most to the ground truth. As noted in Fig. 4 CAMIL assigns high attention values to both positive and negative instances. TransMIL and GTP pinpoint the regions of interest, the attention is relatively low in those areas, which produces unclear boundaries, especially in the case of TransMIL. The attention maps for the rest of the methods and datasets are in Apppendix
The results shown in this subsection validate the utility of the smooth operator at the instance level. They suggest that having smooth attention maps is powerful inductive bias that improves the instance-level performance. In the following, we analyze its impact at the bag level.
5.2 Classification: bag level results.
In this subsection, we show that the use of the smooth operator does not deteriorate the bag classifica- tion results. the contrary, some cases, improves them. Again, we focus on the AUROC and Fl scores, measured by comparing the true bag labels with the methods' bag label predictions. The threshold for the Fl score 0.5. We also report the mean rank achieved by each model.
Table 2 shows the results. The proposed models achieve the best performance overall. As in the localization task, ABMIL performs better than SmAP in RSNA. Again, we believe it be consequence of the CT scan's 's low-complexity structure. DFTD-MIL obtains the best result in CAMELYONI6. but ranks second or third in the other two datasets. GTP and SETMIL outperform the proposed Sm in PANDA, but their performance significantly decreases in CAMELYON16, obtaining the worst results. Overall, our methods provide the most consistent performance, achieving an aggregated mean rank of 1.833.
5.3 Ablation study
The proposed Sm comes with different design choices and hyperparameters: the placement of Sm, the trade-off parameter a, the number of approximation steps T. and the use of spectral normalization. We analyze them in the following, showing that Sm leads to enhanced results almost under any choice. This supports that our hypothesis - that neighboring instances are likely to have the same label - powerful inductive bias worth exploring.

---

Table Classification results (mean and standard deviation from five independent runs). The best is in bold and the second-best is underlined. (1)(()) means lower/higher is better. The models with the proposed operator achieve the best performance overall, ranking first or second in nine out of twelve
dataset-score pairs.
RSNA
PANDA
CAMELYON16
AUROC (t)
FI (1)
AUROC (1)
RI (1)
AUROC (1)
FI (t)
Rank (J)
SmAP
0.8880.005
0.787, oas
0.943p.001
0.915n.002
0.976, o07
0.916
1.833
Without
ABMIL
0.889
0. 0.79
0.9333,.002
0.900 not
0.9566.01
0.91
2.500
global
CLAM
1.6740.157
0.161e.201
0.8933.026
0.868p.ca4 ca4
0. 0.9600.029
0.897o.012
4.5000.s87
interactions
DSIIL
0.6899.oo
0:2400.012
0.921 0.00s
0.904g.coo oos
0.9477.076
0.866p.129
4.167
DFTD-MIL
0.890s.045
0.775io.22
0.940 0011
0.903
0.983
0.937
2,000
SmTA
0.9066.007
O. 0.8220.020
0.946
0.917
0.976 0114
0.948%.oz
1 1.8333
With
TransMIL
0.883y.00
0.71G
0.9333.010
0.89G cus
0.9730..s
0.911u.um
4.083
global
SETMIL
8.86..011
0.7160.006
4.9744.003
1.9466.03
0. 0.715
0.471ga341
3.5832.010
interactions
GTP
0.901L.00m
0.805 017
0.949
0.920 oos
0. 1.74
0.727
2.750p.se7
CAMIL
0.8800.019
0. 0.805p.025
0.938o.008
0.911n.c04
0.9844..07
0.918p.o1s
2.750,.17a
Table 3: Ablation study on different configurations of our models. AUROC (at both instance and bag levels), and normalized Dirichlet energy of attention values are reported. Almost all configurations improve the results in both tasks against the baseline (not using Sm).
RSNA
PANDA
CAMELYONI6
(UROC
AUROCug(
Eu (f)
(UROC
AUROCa (f)
Ep (f) AUROCa.((")
AUROCaa((1 (t)
Ep (T)
SnAP-carly
0.798
0,888
0.009
0.799
0.943
0.106
0,960
0.976
0.395
SnAP-mid
0.792
SnAP-late
0.811
0.891
0.011
0,802
0.911
0.082
0.819
0.961
0.321
ABMIL
0.806
0,889
0.023
0.768
0.933
0.141
0.819
0.956
0.419
SaT-SnAP
0.791
0.910
0.010
0,813
0.911
0.306
0.841
0.986
0.313
SrT+AP
0.791
0.910
0.010
0.754
0.940
0.356
0.734
0.984
0.320
T4SMAP
0.792
0.787
0:944
0.332
T+AP
0.792
0.910
0.020
0.760
0.912
0.391
0.781
0.981
0.133
5.3.1 Placement of
Recall that SmAP leverages by default the early variation, but we also described SmAP-mid and SmAP--lae. Likewise, we discussed different variants for SmTAP. Table 3 Jsummarizes the impact of these choices on the final performance.
Sm without the transformer encoder (SmAP). These variants differ in the place where Sm is located inside the attention pool, recall Eg. 10 12] We include ABMIL since we build our model on top of it. We see that using SmAP improves the performance at both instance and bag levels. This improvement is more noticeable in PANDA and CAMELYON. We attribute to the bag graph structure being more complex in WSIs than in CT scans. Also, the Dirichlet energy is lower when the smooth operator is used, as theoretically expected. We observe that the proposed method is robust to different placement configurations, which is consistent with the theoretical guarantees presented in Sec. 4.1 However, none of the variants consistently outperforms the others.
Sm with the transformer encoder (SmTAP), Recall that SmTAP leverages Sm both after the trans- former encoder and inside the attention pooling. Here we will refer to as SmT+SmAP, and will compare it against T+SmAP and SmT+AP (using Sm only in one of the components) and against T+AP (not using Sm). We observe that Sm has no sigmificant effect on bag-level performance. At instance-level we do observe differences: the bascline T+AP is outperformed as long as Sm is used within the attention pooling.
5.3.2 Sm hyperparameters
In the following study the influence of the trade-off parameter and of the spectral normalization. Due to space limitations, the analysis for the number of approximation steps is in Sec. B.3
The trade-off parameter From Eq. 6 we see that a E [O, 1) controls the amount of smoothness enforced by Sm. Note that a = in Eq. 7 produces no smoothness, turning Sm into the identity operator. In Fig. 6a we show the performance obtained for different values of a in CAMELYON16 Each choice of this hyperparameter improves upon the baseline ABMIL. (a = 0). We see that better localization results are obtained when a is lower, while better classification results are obtained when

---

nade
Inst. AUROC
Bog AUROC
Inst. AUROC
Bog AUROC
1.0
1.00
0.3
0.25
0.5-
0.97
0.7
0.00
3.25
0.30
0.900
0.00
0.25
3.50
0.70
0.7
No
Yo
0.90
No
Yes
1ltt spectral iccuslzztion
Usci specini cocmsdaiiion
(a) Trade-off parameter a.
(b) Spectral normalization.
Figure 6: Influence of the trade-off parameter (left) and of spectral normalization (right) in CAMELYON16. Setting a > O improves upon the baseline ABMIL (a 0) and is a trade-off between better localization results (lower a) better classification results (higher a).. Likewise, Sm without spectral normalization already improves the results upon the baseline (ABMIL), but the best performance obtained when they are used together.
is higher. Fixing a 0.5 is compromise between the two, and produces very similar results as setting it as trainable parameter initialized at = 0.5. Fig. 12 provides a visual comparison of the effect that a has on the attention maps.
The effect of spectral normalization. Spectral normalization forces the norm of the multi-layer perceptron weights remain constant. In this work, this is key design choice that helps Sm obtain attention maps with lower Dirichlet energy. In our experiments, we have used spectral normalization in the layers immediately after Sm. Note that the late variant does not require spectral normalization, since it applies Sm directly to the attention values. In Fig. 6b we show the results obtained with and without spectral normalization in CAMELYONI6. We observe that, even without spectral normalization, Sm improves upon the baseline. The improvement is more significant when Sm is paired with spectral normalization, especially at the instance level.
Discussion and conclusion
The main goal of this paper to draw attention to the study of MIL methods the instance level. To that end, we revised current deep MIL methods and provided a unified perspective on them. We proposed the smooth operator Sm to introduce local interactions in principled way. By design, produces smooth attention maps that resemble the ground truth instance labels. We conducted an exhaustive experimental validation with three real- woold MIL datasets and up eight state-of-the-an methods in both classification and localization tasks. This study showed that our method provides the best performance in localization while being highly competitive (best or second best) at classification.
Despite its advantages, our method has some limitations. The first is that, as with every other operator in GNNs, the computational costs of the smooth operator scale with the size of the bag. Fortunately, it can be paired with existing subgraph sampling techniques to mitigate this problem. The second limitation is that we do not have definite answer for where it is better to the proposed operator. We have shown that it leads to improvements in almost every place, but the optimal location may be problem-dependent and has to be tailored by the practitioner.
Finally, we hope that our work will draw more attention to the localization problem, which is very important for the deployment of computer-aided systems in the real world. In this sense, safely deploying the proposed methods clinical practice requires evaluating them a wider range of medical problems and quantifying their uncertainty. For the latter, we believe that the smooth operator could also benefit from a probabilistic formulation.
Acknowledgements
This work was supported by project PID2022-14018900--O0 funded by MCIN / AEI/ 10. 3309 / 501 10001 1033. Francisco M. Castro- acknowledges FPU contract IPU21/01877 funded by Ministerio de Universidades. Pablo Morales-álvarez. acknowledges grant C-EXP-153-UGRR funded by Consejería de Universidad, Investigación e Innovación and by the European Union (EU) ERDF Andalusia Program 2021-2027.
10

---

References
[1] Ravi Aggarwal, Viknesh Sounderajah, Guy Martin, Daniel SW Ting, Alan Karthikesalingam, Dominic King, Hutan Ashrafian, and Ara Darzi. Diagnostic accuracy of deep learning in medical imaging: a systematic review and meta-analysis. NPJ digital medicine, 4(1):65, 2021.
[2] Babak Ehteshami Bejnordi, Mitko Veta, Paul Johannes Van Diest, Bram Van Ginneken, Nico Karssemeijer, Geert Litjens, Jeroen AWM Van Der Laak, Meyke Hermsen, Quirine F Manson, Maschenka Balkenhol, et al. Diagnostic assessment of deep learning algorithms for detection of lymph node metastases in women with breast cancer. Jama, 31 (18822)22199222) 2017.
[3] Mikhail Belkin and Partha Niyogi. Semi-supervised learning on manifolds. Machine Learning Journal, 1, 2002.
[4] Christopher Michael Bishop and Hugh Bishop. Deep Learning Foundations and Concepts. 2023.
[5] Wouter Bulten, Kimmo Kartasalo, Po-Hsuan Cameron Chen, Peter Ström, Hans Pinckaers. Kunal Nagpal, Yuannan Cai, David F Steiner, Hester Van Boven, Robert Vink, et al. Artificial intelligence for diagnosis and gleason grading of prostate cancer: the panda challenge. Nature medicine, 28(1):154-163 2022.
[6] Chen Cai and Yusu Wang. A note over-smoothing for graph neural networks. arXiv preprint arXiv:2006. 13318, 2020.
[7] Gabriele Campanella, Matthew G Hanna, Luke Geneslaw, Allen Miraflor, Vitor Werneck Krauss Silva, Klaus Busam, Edi Brogi, Victor Reuter, David S Klimstra, and Thomas Fuchs. Clinical-grade computational pathology using weakly supervised deep learning on whole slide images. Nature medicine, 2018):130 2019.
[8] Marc-André Carbonneau, Veronika Cheplygina, Eric Granger, and Ghyslain Gagnon. Multiple instance learning: A survey problem characteristics and applications. Pattern Recognition. 77:329-353, 2018.
[9] Josef Cersovsky, Sadegh Mohammadi, Dagmar Kainmueller, and Johannes Hochne. Towards hicrarchical regional transformer- multiple instance learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3952-3960, 2023.
[10] Olivier Chapelle, Jason Weston, and Bernhard Schölkopf. Cluster kernels for semi- sumi-super learning. Advances in neural information processing systems, 15, 2002.
[11] Richard J Chen, Ming Y Lu, Muhammad Shaban, Chengkuan Chen, Tiffany Y Chen, Drew FK Williamson, and Faisal Mahmood. Whole slide images are 2d point clouds: Context-aware sur- vival prediction using patch-based graph convolutional networks. In Medical Image Computing and Computer Assisted Intervention-MICCAA 2021: 24th International Conference, Strasbourg, France, September 27-October 1, 2021, Proceedings, Part VIII 24, pages 339-349, Springer, 2021.
[12] Thomas G Dietterich, Richard H Lathrop, and Tomás Lozano-Pérez Solving the multiple instance problem with axis-parallel rectangles. Artificial intelligence, 89(1-2)331-1,, 1997.
[13] Adam E Flanders, Luciano M Prevedello, George Shih, Safwan S Halabi, Jayashree Kalpathy- Cramer, Robyn Ball, John Mongan, Anouk Stein, Felipe C Kitamura, Matthew P Lungren, et al. Construction of machine learning dataset through collaboration: the rsna 2019 brain ct hemorrhage challenge. Radiology: Artificial Intelligence, 2(3) 192111 2020.
[14] Olga Fourkioti, Matt De Vries, and Chris Bakal. CAMIL: Context-aware multiple instance learning for cancer detection and subtyping in whole slide images. In The Twelfth International Conference on Learning Representations, 2024. URL httphtttss//oprrviviieeeee hatpeerrrviee..ne//orrmmm net/forum?
id=rzBskAEmoc
[15] Guoji Fu, Peilin Zhao, and Yatao Bian. p-laplacian based graph neural networks. In International Conference Machine Learning, pages 6878-6917. PMLR, 2022.
11

---

[16] Johannes Gasteiger, Aleksandar Bojchevski, and Stephan Günnemann. Predict then propagate: Graph neural networks meet personalized pagerank. arXiv preprint arXiv:1810.05997, 2018.
[17] Maximilian Ilse, Jakub Tomczak, and Max Welling. Attention-based deep multiple instance learning. In International conference on machine learning, pages 2127-2136 PMLR, 2018.
[18] Mingu Kang, Heon Song, Seonwook Park, Donggeun Yoo, and Sérgio Pereira. Benchmarking self-supervised learning on diverse pathology datasets. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3344-3354, 2023.
[19] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1rXiv:1412.6688, 2014.
[20] Thomas N Kipf and Max Welling. Supi-super classification with graph convolutional 2016.
networks. arXiv preprint
[21] Bin Li, Yin Li, and Kevin W Eliceiri. Dual-stream multiple instance learning network for whole slide image classification with self-supervised contrastive learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 14318-14328, 2021.
[22] Hang Li, Fan Yang, Yu Zhao, Xiaohan Xing. Jun Zhang, Mingxuan Gao, Junzhou Huang, Liansheng Wang, and Jianhua Yao. Dt-mil: deformable transformer for multi-instann learning on histopathological image. In Medical Image Computing and Computer Assisted Intervention- MICCAI 2021: 24th International Conference, Strasbourg, France, September 27-October 1, 2021, Proceedings, Part VIII 24, pages 206-216. Springer. 2021.
[23] Qimai Li, Zhichao Han, and Xiao-Ming Wu. Deeper insights into graph convolutional networks for semi-supervisee learning. In Proceedings of the AAAI conference artificial intelligence, volume 32, 2018.
[24] Ruoyu Li, Jiawen Yao, Xinliang Zhu, Yeqing Li, and Junzhou Huang. Graph enn for survival analysis whole slide pathological images. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 174-182. Springer, 2018
[25] Ming Y Lu, Drew FK Williamson, Tiffany Y Chen, Richard J Chen, Matteo Barbieri, and Faisal Mahmood. Data-efficient and weakly supervised computational pathology on whole-slide images. Nature biomedical engineering, 5(6):555-570, 2021.
[26] Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. arXiv preprint arXiv: 000205597 2018.
[27] Andrew Ng, Michael Jordan, and Yair Weiss. On spectral clustering: Analysis and an algorithm.
Advances neural information processing systems, 14, 2001.
[28] Gwenolé Quellec, Guy Cazuguel, Béatrice Cochener, and Mathieu Lamard. Multiple-instance learning for medical image and video analysis. IEEE reviews in biomedical engineering, 10: 213-234, 2017.
[29] Qin Ren, Yu Zhao, Bing He, Bingzhe Wu, Sijie Mai, Fan Xu, Yueshan Huang, Yonghong He, Junzhou Huang, and Jianhua Yao. lib-mil: Integrated instance-level and bag-level multiple instances learning with label disambiguation for pathological image analysis. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 560-569, Springer, 2023.
[30] Brian D Ripley. Spatial statistics. Wiley Series in Probability and Statistics, 1981.
[31] Matthias Seeger. Learning with labeled and unlabeled data. 2000.
[32] Zhuchen Shao, Hao Bian, Yang Chen, Yifeng Wang. Jian Zhang, Xiangyang Ji, et al. Transmil: Transformer based correlated multiple instance learning for whole slide image classification. Advances in neural information processing systems, 34:2136-2147, 2021.
[33] Julio Sodva-Rodriguez, Adrián Colomer, Jose Dolz, and Valery Naranjo. Self-learning for weakly supervised gleason grading of local patterns. IEEE journal of biomedical and health informatics, 3594)34094 2021. 202..
12

---

[34] Andrew H Song, Guillaume Jaume, Drew FK Williamson, Ming Y Lu, Anurag Vaidya, Tiffany R Miller, and Faisal Mahmood. Artificial intelligence for digital and computational pathology. Nature Reviews Bioengineering, 1(12):30-995 9499 2023.
[35]
Andrew H. Song, Mane Williams, Drew F. K. Williamson, Sarah S. L. Chow, Guillaume Jaume, Gan Gao, Andrew Zhang, Bowen Chen, Alexander S. Baras, Robert Serafin, Richard Colling, Michelle R. Downes, Xavier Farré, Peter Humphrey. Clare Verrill, Lawrence D. True, Anil V. Parwani, Jonathan T. C. Liu, and Faisal Mahmood. Analysis of 3D pathology samples using weakly supervised AL. Cell, (101100)::5022220.eehh May 2024. ISSN 1097-4172. doi: 10.1016fj.eel12224..3..m
[36] Yunan Wu, Arne Schmidt, Enrique Hernández-Sáncchez, Rafael Molina, and Aggelos K Kat- saggelos. Combining attention-based multiple instance learning and gaussian processes for ct hemorrhage detection. In International Conference on Medical Image Computing and Computer-Asiisted Intervention, pages 582-591. Springer, 2021.
[37] Conghao Xiong, Hao Chen, Joseph JY Sung, and Irwin King. Diagnose like pathologist: Transformer-enabbled hierarchical attention-guided multiple instance learning for whole slide image classification. arXiv preprint arXiv:2301.08 2023.
[38] Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self- supervised learning via redundancy reduction. In International conference machine learning, pages 123200. PMLR, 2021.
[39] Hongrun Zhang, Yanda Meng, Yitian Zhao, Yihong Qiao, Xiaoyun Yang, Sarah E Coupland, and Yalin Zheng. Difd-mil: Double-tier feature distillation multiple instance learning for histopathology whole slide image classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18802-18812 2022.
[40] Yu Zhao, Zhenyu Lin, Kai Sun, Yidan Zhang, Junzhou Huang. Liansheng Wang, and Jianhua Yao. Setmil: spatial encoding transformer based multiple instance learning for pathological image analysis. In International Conference on Medical Image Computing and Computer- Assisted Intervention, pages 66-76. Springer, 2022.
[41] Yi Zheng, Rushin H Gindra, Emily Green, Eric J Burks, Margrit Betke, Jennifer E Beane, and Vijaya Kolachalama. A graph- trapfforrrmff for whole slide image classification. IEEE transactions medical imaging, (1(11):3003-300 2022.
[42] Dengyong Zhou and Bernhard Schölkopf. Regularization discrete spaces. In Joint Pattern Recognition Symposium, pages 361-368. Springer, 2005.
[43] Dengyong Zhou, Olivier Bousquet, Thomas Lal, Jason Weston, and Bernhard Schölkopf. Learning with local and global consistency. Advances in neural information processing systems, 16, 2003.
[44] S Kevin Zhou, Hayit Greenspan, Christos Davatzikos, James S Duncan, Bram Van Ginneken, Anant Madabhushi, Jerry I. Prince, Daniel Rueckert, and Ronald M Summers. A review of deep learning in medical imaging: Imaging traits, technology trends, case studies with progress highlights, and future promises. Proceedings of the IEEE, 82885) 2021.
13

---

A Proofs
A.1 Proof of Eq. 4
We present a general result for arbitrary multilayer perceptron with Lipschitz activation functions.
Note that assuming Lipschitzness is not a restriction, since most of the currently used activation functions meet this property [6]. The desired Eg. 4 a partieular case of this result. Let L N.
Consider a L-layer perceptron that, given E R Do, outputs Y E NxDx defined by the following rule
Y Y, Y = se (Y'W, + Be) LE E 10-
(14)
(15)
where Wr E DDixDr-1, and B, [b.....,, bu] NNDD++1 where by E RP are trainable weights, and yp: R R are activation functions applied element-wise. We suppose that each activation function ype Ke-Lipschitz. Then, we obtain the following inequality,
ED (o(Y+))) S K2 /ell ED (Y").
(16)
Before verifying it, we note that by applying this inequality to every layer, we arrive at
ED (Y4) S ..... S 12-1- W W- ED (Y') S ................................ S KL-10 [ L- ED (Y), (17)
where W woll = II W 2 and K = -i Kr. Taking - 2, Do = D, D1 = L, D2=1. =1
bo b 0, 40 = tanh, and yo Id, we recover Eq. 4
To verify Eq. 16 we write Y = [yst]
4+1
and Y = [yi, yN]] We have,
N
N
Ed (Y(+1) = 2
Aij |1y e+1 - 0
(18)
im j=1
Aij ll (WTyf + Bi) - or (WTy) + B
(19)
1=1
$ K?
Aij WT (yf x0)1
(20)
N
< K
W
Aug |y! - x11²²² K2 [ell ED (Y"),
(21)
2
where from Eq. to to Eq. 20 we have used the definition of Lipschitz function and from Eqq 20 to Eq. 21 we have used the consistency between the spectral and Euclidean norms.
A.2
Proof of Eq. 9
In this section, we adapt the proof presented in 6 for a similar result. Let U E RxD, Our goal is to show that
ED (1++70) (I+ U) & XEED (U),
(22)
where 7 and X; max {a1 (1+ An EA {0)). being A (A AN the eigen-
values of the symmetric graph Laplacian matrix L. First, we reduce the proof to univariate graph functions by looking the rows of U as univariate graph functions. Denoting them as fui...., up).
where each Uy N we have &p (I++L) U) Σd En (a 7L) us). Therefore,
will be sufficient to show that, for any u E N.N.
ED (I+TL)- u) S XEED (u)
(23)
14

---

Next, it is useful to note that if An is an eigenvalue of L with associated eigenvector Vn. then (1 + +-7))= is an eigenvalue of (I yL) with the same associated eigenvector. Finally, let (vi,..., vN be a orthonormal eigenbasis of L, being each vn associated to the eigenvalue An This basis always exists since L is symmetric matrix. Writing u = En= Cn Var with Cn E R, we have
([+
Gn 71n Vn. u
(24)
Using that the eigenvectors are orthogonal to each other, we arrive
N
N An XEED (u).
ED (a++-))'
Σ (1+
X
(25)
B Experiments: details and further results
In this section, we provide the details of the datasets, architectures, and configurations used for each experiment. The code is available at http:////ggtttbb..oo///aaeeeeeee SMMIL
B.1 Datasets
We provide insights into the datasets we have used: a description of the problem, the train/test splits, and preprocessing (instance selection and feature extraction). For all datasets, we obtain initial train/test partition. Then, we split the initial train partition into five different train/validation splits. Every model is trained on each of these splits and then evaluated on the test set. We report the average performance on this test set.
RSNA. It was published by the Radiological Society of North America (RSNA) to detect acute intracranial hemorrhage and its subtypes [3]. It available in Kaggle We use the official train-test split. It includes total of 1 1150 scans. There are total amount of 39750 slices and the number in each scan varies from 24 to 57. Each slice is preprocessed following [36].
PANDA. It is a public dataset for the classification of the severity of prostate cancer from microscopy available in Kaggle Since the official test set is not use the train/test split proposed in [33). To extract the patches from each WSI, we follow the procedure described in [33), obtaining patches of size 512 x 512 at 10x magnification.
seans of prostate biopsy samples [5] publicly available,
This results in a total amount of 10503 WSIs and 1107931 1 patches.
CAMELYON16. It a public dataset for the detection of breast cancer metastasis 2 Is available at the Registry of Open Data of AWS The official repository contains 400 WSIs in total, including 270 for training and 130 for testing. From each WSI, we extract patches of size 512 x 512 20x magnification using the method proposed by Lu et al. (25
B.2 Model and training configuration
We provide details about how we have implemented the proposed methods and how we have conducted the experiments.
Feature extractor. Due to the limited memory of the GPU, it is necessary to extract features from each instance. Otherwise, the bags will not fit in memory. In this work, we consider three options for the feature extractor, all of which are pre-trained in Imagenet: ResNet18 (P 512), ResNet50 (P = 2048), and ViT-B-32 (P = 768). In addition, for CAMELYONI6 we also consider ResNet50- BP 2048), which ResNet50 model pre-trained using the Barlow Twins Self Supe-Supere Learning method on huge dataset of WSIs patches 38 [8] The results reported in the main text correspond ResNet18 for RSNA and PANDA, and ResNet50-BT for CAMELYON16. We study how the choice of the feature extractor affects the results in Sec. B.3
httege///wwwnggggeeeccooooorrrrs cancer www.acrreg.a.ddeessseseeeeeeeeee
Weights available at htttttpss///gitti http://gituuu.ccommmemmm wwmittuiiitttttt wwwm///un-----////eennnaaa.. www.e//chmmrrssslllssttteeee
15

---

Inst. AUROC
Bag AUROC
Last. AUROC
Bac AUBOC
01.50
0.00
0.50
0.9
0.90
0.50
0.:57
0.90
0.78
Iste
01 5
10
20
0.55
01
01
10
0.00
01
(a) RSNA.
(b) CAMELYONI6.
Figure 7: Influence of the number of steps T used to approximate Sm in RSNA and CAMELYON16. ABMIL corresponds to T 0. Using T = 10 is enough to closely match the performance of the exact form (T' = o0).
Model architecture. We describe the architecture we have used for the proposed methods (SmAP and SmTAP). For the rest of the methods considered, we adopt their original implementations and default configurations, publicly available on their GitHuu repositories. For the independent instance encoding part (see Fig. 3a and Fig. 3b), the instance embeddings h, are obtained using one fully connected layer with 512 units (7) = 512). For the attention baasd pooling described by [5q. 1 and Eq. 2] we fix D 512 and L 100. The transformer encoder in Fig.. 3e implemented using two transformer layers. These layers use the standard multi-head attention mechanism equipped with skip connections and layer normalization [4]. We fix the key, query, and value dimensions 128 and the number of heads to 8. We used the Pytorch's 's implementation of dot product attention Finally, the bag-embeddiig classifier was implemented using one fully connected layer.
Training setup and hyperparameters. To ensure fair and reproducible conditions, we trained every method under the same setup. The number of epochs was set to 50. We adopt the Adam optimizer 19 with the default Pytorch configuration. For the base learning rate, we considered two different values, 10-1 and 10-5, since noticed that models that do not transformers obtained better results when the learning rate was higher. We report the best results for each model. We adopted slow start using Pytorch's LinearLR scheduler with start_ faatorrfa and cotal_iterss During training, we monitored the bag AUROC and cross-entropy loss in the validation set and kept the weights that obtained the best results. The batch size was set to 32 in RSNA and PANDA. In methods. CAMELYON16, it was set to 4 for no- no-transformer methods, and to for transformer-be However, for SETMIL, we had to set memory requirements. In RSNA we weighted the loss function to account for the imbalance between positive and negative bags since we observed it to improve the results. All the experiments were performed on one NVIDIA GeForce RTX 3090.
in PANDA and CAMELYON 16 due to its high GPU
B.3 Further ablation studies.
We complete the ablation study presented in the main paper, Sec. 5.3 by looking at the rest of the design choices or hyperparameters associated with our Sm.
Smooth operator approximation. The exact form of Sm given by Eq. becomes computationally infeasible for large bag sizes. The quality of the approximation, given by Eq. 8 is controlled by the number of steps T. Fig. 7 shows the results for different values of this hyperparameter in RSNA and CAMELYONT6, In RSNA, since the bags are smaller, we can compute the closed-form solution, which we represent as T = cc. Almost any choice of T > O improves upon ABMIL. This improvement is particularly noticeable in CAMELYON16. Moreover, in most cases, the performance stabilizes at = 10, which the value we used in our experiments.
Sm on top of other models. We have proposed two new models (SmAP and SmTAP) by applying Sm on top of two baselines (ABMIL and Transformer+ABMILL respectively). Instead, the Sm can be applied on top of other existing approaches. In Table 4 we explore how other approaches behave when combined with the proposed Sm in the CAMELYON 16 dataset. Instance-level performance is enhanced (greatly in some cases, c.g. an increase from 0.76 to 0.96 in AUROC for DSMIL), whereas beg-level results are competitive. The decrease in leg-l results for DFTD-MIL. is explained by
htttp::///yooorr..oo////c/ss/sss http:////weeroooogoogsssss///ecc functiion scaled_dot_
product_attentioon
16

---

Table 4: Using Sm on top of other models (CAMELYONI6 with ResNer50-BT features). Improve-
ments are highlighted green. Using the proposed Sm increases the instance-level performance, while the bag-level performance remains competitive.
Instance
Bag
AUROC (t)
Fl (t)
AUROC (1)
F1 (1)
CLAM
0.849..044 044
0.821
0.96p.029
0.897a.012
SmCLAM
0.928o.022
0.8730.018
0.966g.om
0.889o.017
DSMIL
0.76g.ore 078
0.654
0.947 os5
0.8666.136
SmDSMIL
0.960g.013
0.77 0.776p.08s
0.967a.011
0.919g.018
DFTD-MIL
0.984..00 002
0.7420.040
0.983p.010
0.987o.013
SmDFFD-MIL
0.984..188
0.8366.222
0.978p.158
0.903o.183
Table 5:
Instance and bag AUROC (higher is better) in CAMELYON16 using ResNet50-BB -- features for the proposed methods and the penalty-based approach. The best in each column is highlighted in bold. Sm obtains superior performance, although the differences are not large.
RSNA
PANDA
CAMELYON16
Inst.
Bag
Inst.
Bag
Inst.
Rag
W/o global int.
SmAP
0. 79.988..03
0,8888 005
0. 0.799g.005
0.9 0.943
0.961 007
0. 0.965
ABMIL+PENALTY
0. 0.7820.050
O. 8890.043
0. 0.780n.003
0.935p.000
0.979u.01a
0.963p.012
W/
S=TAP
0. 76.66
O. 0.900.0007
0.790g.oom 007
0.946p.000 0003
O. 0.789.000
0.976
global int.
T+PENALTY
0.7376.046
0.9055.00 005
0. 0.772p.01
0. 0.9470.001
0.769p. noo
0. 0.988p.00 0044
the fact that this method randomly splits each bag into different chunks. This may lead to the loss of local interactions exploited by Sm (e.g. if two adjacent instances end in different chunks).
An alternative smoothing strategy. Introducing penalty term in the loss function to favor smooth- ness is a natural alternative to the proposed operator. However, there is an important difference: the use of penalty term does modify the model architecture. The penalty term favors that the learned weights encode such property, but it is not explicitly encoded in the model. For instance, note that the penalty term is not used inference time. We compare the penalty- -based approach and the proposed Sm in Table 5 Although differences are not large, Sm obtains superior performance.
Feature extractor. We investigate whether the choice of the feature extractor influences the results and conclusions presented in the main text. We have evaluated each of the considered methods in each dataset using the feature extractors mentioned above (ResNet18, ResNet50, ViT-B-32, and ResNet50-BB -T)). The results are shown in Tables I0 We summarize them in Table 6 where we collect the average instance and bag rank of each method for each feature extractor. We observe that the proposed smooth operator Sm obtains in almost all cases the highest rank. This supports the idea that the improvement introduced by Sm does not depend on the used features.
Table 6: Instance and bag average ranks (lower is better) obtained by each method for different choices of the feature extractor. The best result within each group bolded, and the second-best is underlined. SmAP and SmTAP obtain in almost all cases the highest rank.
ResNet18
ResNer50
VIT-B-32
ResNet5O-BT
Inst.
Rag
Inst.
Bag
Inst.
Bag
Inst.
SmAP
2.000,.632
2,000,
1.6255 744
1. 1.750p.00
1.667
1. 1,500 220
1.000,.000
2.000
ABMIL
2.667,
1.66To.a10
3.750)
3.250l
4,333, 100
3.0000
4.500l 70r
3.500h
Without
DespGraphSure
1.0002.000
5.500) 201
2,500,
5.6
3,333, 211
5.0000, .o0
2,500,
6.000huco
global
CLAM
6.167u.sss
6.500,.225
4.7502053
4 1.50002.70
6.333
5.000
3,000;
3.500u
interactions
DSMIT
5.1677.38
5.333 .000
6.3755..1a
6.000p.o26
5.667
6.6677..10
6.000p..00
5.000p.00
PathGCN
5,838.472
5. 6.167
5. 5.625
1 1.6252.134
1,500,a4s
1,000,
7. 7.000
7.000uc00
DFTD-MIL
2.167a3385
2,00
3.376
2,250 458
2. 2.167
2,833 .88
4.000,
1.000..000
SaTAP
2. 2.167
1.667:
2, 2.375,
1. 1.875
1.833
2. 2.500...7
1.500,.urr
1.500,
TransMLL
3.167,.s2p
3. 3,833
3,500, omo
3. 3.625, ons
4.167,
4.500,.rr
4,000,
4.000mccoo
global
SETMIL.
3.0GTo.s1s
3.500
3.625
4.125g.051
3.60T1.008
3:333, 862
4 4.500
6.000
interactions
BBMIL
GTP
4.167u.ss
6.1072.041
3.333
6.0077.
4.500,
4.5002.158
3.375;
6.125,,042
5.000
4.167
4,831
3.1672.041
6.000
2.000,
5.000ummo
2.500p
CAMIL
2.66T,
3,000 504
2,500, aps
2.875, 356
2,167,
2,667, .011
3.000
2,000,
17

---

ResNet50
CAMELYONIC
0.961 0.007
01.816 030
0.849 0..90004
0.760 .07
0.4430. 1.0
0.88 0002
0.78% 008
0.781p..22
0.61 55
0.4422..00 00.009
0. 0.873 108
0.7422 22122.02
ResNet50=B11
CAMELYONI6
0. 0.839
0.767o..80
0.77
0.821 046
0.65 200
0.077o.114
0.742o.000
0.600 067
0.127 078
0.1344p 269
0.037o.084
0.352
0.479 1770
0.958
CAMELYONI6
CAMELYONI6
0.7730.00 .02
0.7356..1 113
0.736p 10s
0.463p.mo 004
0.061p.115
0.8
0. 0.9522.01 011
0.819p.162
0. 0.779 062
0.569p.00 n00
0.4776.005
( 1.863, oos
0. 0.939
0. 0.580p..mm
0.419
0. 0.1655.088 0s9
0.3733.00 08
0. 0.255p.184
0.4146.11
0.552 055
0. 0.552 108
0. 0.1944 000
0. 0.150p..00 000
0.08-40.045
0. 0.2966.00
0,426 855
ViT-B-32
ViT-B-32
PANDA
0.8100.007
0. 0.7733g.00
0 0,809, o0s
0. 7.777m.001
0.7 6.779g.002
0. 0.709h.0m2
0..8 0.785p.008
0. 0.822..01
0. 0.749
0. 0.789
0. 0.720p.029
0.740g.00z
0 0,8066 015
PANDA
0.6 0.645..om
0 0.605p.006
0.612
0.610g.008
0.6 0.610%.001
0.610p.02a
0.616
0.655 013
0.630p 011
0. 0.822..00 012
0.641n001
0.6555p.006
0.641p014
Table 7: Instance AUROC (higher is better) for different choices of the feature extractor.
Table 8: Instance Fll (higher better) for different choices of the feature extractor.
RSNA
0.804...17
0.7977.022
0.755 009
0. 0.500p.0mm .00
0.702o.02
0.749
0.807o.330
0. 0.795p.029
0. 0.749p.01
0. 0.755a.om.
0..600o.013
0.6990..01
0.783
RSNA
0. 0.494
0.4 0.498
1). 0.41790.0g
1) 11,0000
0. 0.3000.03)
0. 0.481o.000
0. 0.489n.033
0.4755 001
0.480, oze
0. 0.4670.008
0. 0.4470.02
0. 0.443
0.5 0.504..025
CAMELYONI6
CAMELYON16
0 1L8544,
0. 1,8066
0. 0.8144.027
0. 1.359h.056
0 0.6770..11
0. 0-85 10.29
0. 0.9522.013 2010
0. 0.716
0. 1.844 023
0. 0.78
0. 1.528%
0. 1,000.0000
0. 0.9644..014 0114
0. 0.675%.077
0. 0.6544.0m
0. 1.6633, uss
O. 0.5844 087
0. 1.2000 174
0. I.
0. 1..00 Lo.00s
0.6300.oo 090
0. 19696.1. 118
0. 0.0366.00 021
0. 1.1500. 122
0. 0.000..00
D. 1.563,
2003
ResNet50 PANDA
ResNer50
0 1.786, o00
11.77 47740.000
0 1.806..000 002
0. 0.7855.004
0. 0.7470.000
0. 17722.011
1.7884..00
0. 0.756p.02
0. 0.745p.021
0.7 0.774
11.7540.011
0. 0.7299.03
0 1.7866 027
PANDA
0. 0.630p.om
0.611oo.07
0. 0.6411.0022
0. 0.621o.oo 007
0. 0.502 005
0. 6086..0
0.61 0.6177mm
0. 0.6066 015
0. 0.622. 023
0 0.821 18221.022
0. 0.621..044
0. 0.644.p.om
0.61.5..0
RSNA
0. 1,7833.00 020
0. 0.7966.027
0.768p.013 013
0.4977.ncm nos
0.568 0015
0 0.0922.n47 .47
0.796
0. 802 802
0.70 0.7077..0z
0.67 0.678 n04
0. 0.730p..00 024
0.67 0.672..02 024
0.796 013
RSNA
0. 0.473
0.470m m11
0. 0.16 022
0.0000 000
0. 0.2711.018
0.4311.cm
0.4476.00
0. 0.517
0.442, 024
0.405p.021 02
1.431..01
0. 0.403
0.433, n24
CAMELYONIE
01.3980.00 .333
11,6799 0.82 082
@ 1.868, 091
0. 6p. 102
0. 6288..181 181
0.6 88..14
0 0.920..072 074
0. 7.750 134
0. 1880 ass
0. 0.7922 n922
0 0..5944.222 222
0 1,5000g 800
0.917..0m
CAMELYONIS
0. 1.428p.a4
0. a06 238
0. 155555 180
0. 0.563p.1.5 1.32
0. 0.581g 001
0. 1. 080
0. 0.2376..55 nss
0. 0.168p.135 132
0. 0.000n..om 200
D. 1.303. 107
0. 0.5955.10
0. 0.1500...
0.59
ResNet18
RcsNer18
PANDA
0. 0.799 000
0.768 002
0. 0.7200.011
0. 0.727n.om
0.7 0.705 008
0. 0.604n.mm m11
0. 0.795 .044
0. 1.790p.com
0. 0.751p-011
0.774
0.7688.002 2.02
0.740g.020
0.785 011
PANDA
0 0.635 .n0
0.602a..05
0.581 1.581..026 0.026
0. 0.526
0.6 0.637o.006
0. 0.6222..0 0100
0.636 o0s
0.631 0.11
0.636p.011
0.6 0.6445..007
0.621o.013
0).5 0.5688
0. 0.508
RSNA
0.798, udi
0 0.80660012
0.6810.054
0.5233.000
0.5549.004
0.71 lo.045
0.7470.070
0. 06670.046
0.7 0.7320.033
0.7266.025
0.733600117
0.6750.017
0.7600 006
RSNA
0.477
0. 0.486..n83
0 1.29%%).0
0,0766
0.180p.oom
0. 0.447o.014
0. 1.453..109
0.4740.023
0.471,
014 0.4
0. 0.438%.00 027
0 0.425%.013 018
0.4200.mm
0. 0.45660.03
013
DeepGraphsSur
SmAP
ABMIL
DeepGraphSure
CLAM
DSMIL
PabGCN
DFTD-MIL
SmTAA
TransMIL
SETMIL
GTP
IBMIIL
CAMIL
SmAP
ABMIL
CLAM
DSMIL
PathGON
DFTD-MI MIL
SmTAP
TransMIL
SETMIL
GTP
IBMI.
CAMIL
Without global interactions
With global interactions
interactions
With global interactions
Without global
18

---

ResNet50+BB
CAMELYONIS
(.976 0007
0.9566 011
0.870 0n0
0.960 02000.00
0.9477.088 .00
0.575p 200
0. 0.983g.010
0.976p 014
0.973 00.018
0.71
0.748%..12
0.97 .000
0. 0.9844.007
ResNet50=BB1
CAMELYONIA
0.916
0.912
0.772
0.8977..022
0. 0.866p 130
0.3455 5525
0. 0.987
0.948 020
0.911p 028
0.471o.341
0.727o.145
0.9222,.00 p.000
0.918p.018
CAMELYONI6
CAMELYONI6
0.775p.02 0225
0.7922..00 022
0.7477.000 000
0. 0.832n.no
0.028p.nes 063
0.74
0.801
0. 0.7544.008 002
0.6 0.6555..m 08g
0. 0.469p.10r
0. 0.4566.111
O. 7711..00
0.772 084
0. 0.7011.023
0.073 mas
0. 0.6066.066 00.00r
0.671p 003
0. 0.308p os0
0. 0.6066.088 .22
0. 0.068p.017
0.67U 044
0. 0.453p 00s
0. 0.451p..08 .65
0. 0.3844..00 40.000005
0. 0.686
0. 0.6399.000 n99
ViT-B-32
ViT-B-32
PANDA
0.947
0.9433.0002
0.9388.002
0.92Tm.001
0.925p.004
0.945p.oms
0.945p.001
0.946a.005
0. 0.939g.ous
0. 0.970n.005
0.945m..0s
b.939p.002
0.947 004
PANDA
0.5 1
0.91 22.000
0. 0.908p.004
0. 0.9004g.002
0. 0.902
0.91 1.9144.006
0.91 00140.002
1.91do.000 004
0. 0.802. 024
0. 0.953p.00.
0000000
0.912p.008
0. 0.803p.000
0.917mom2
Table 9: Bag AUROC (higher is hetter) for different choices of the feature extractor.
Table 10: Bag Fl (higher better) for different choices of the feature extractor.
RSNA
0.897 0.005
1| 118883..o
0.870p.010
0.7 0.735p.047
0.792o.04
0.880n.ozz
0.870lg.020
0. 0.8066.00m .00
0. 0.900o.913
0. 0.805
l) 1189900.005
D 11899
l) 0.8822..0m .000
RSNA
0.8 0.805
0. 0.788...2
0.7766.01
1,00000.oo
0. 0.683p.c00
0. 0.7766.012
0. 0.798
0. 0.807%.008
0.7196.027
0. 0.730p.014
0. 0.773
0. 0.723
0.792..011
CAMELYONI6
CAMELYON16
0. 7..770.046
0.7522.023
0 1.6955..00
0.775po0u
0 1099..00
0. 0.708a.064
0. 0.7200.03
0. 8.855...00 057
0. 0.791n 027
0. 6.657oomo
0) 1.4599.000
0. 1.45500.c2
0) 1.7338p.0o 000
0. 0.713
0. 0.60lamm
0. 0.5900.04
U. 0.676 041
0. 0.2122
0. 1,507o.17
0. 0.509..03
0.7 0.707o 020
0. 0.635, 075
0. 0.0133
0. 0.382,.076
0. 0.000..000
0 0.6119..009
ResNet50
ResNer50
PANDA
1199-22.000
0.925p.com 002
0.930p.002
0 0.926o.00z
0.943n.0om
0 0.945g.om 002
0. 0.9444.002
0. 0.9420.002
0 0.977 70770.
0.952
0.9399.004
0.941o.000
PANDA
0 0.918..03
0.91 29.120007
0. 0.905p.000
0. 0.904o..00
0. 0.907o.00
0.91 0.915n.004
0.917n.000
0.91 003
0. 0.905 013
0.9 0.951a.a11
0.923 003
0. 0.8890.011
0.913%.000 000
0.914
0.097
0.8866.01. 013
0.877 008
0.802n.0r 064
0. 1.761p, 1761p.09s 026
1.8800p.o n09
0. 1,893, 009
0. 1.885p 009
0.8 0.870p.nm nos
0. 1.896..06
0.56 0.861p.000 000
0 0.8922.00 0000
0. 0.788 m01
D. 0.800m n24
0.7 0.770%.005 013
0.0166.0021 024
0. 0.374g.000 001
0. 0.7576.ne0 neo
0. 8.06
1,500, 016
0.758 015
0. 0.7344..08 027
0. 0.8077.010
0.667
0. 1.011
RSNA
1.890
RSNA
CAMELYON
CAMELYONIS
(L.72gp.ol aar
0. 3.7311..00 1311..000
00117 0017 16673gg00
0. 110833.082
0. 26.72.111 130
1.585p.. 180
1.7066. .222
0. 0.783 ons
0. 1.771n..80 ls.
0. 0.028..a5)
1.5770.075
0.66419.012
0.7416p.04)
0.66 p.asa
0. 0.6677.022
0 1.560h..02 .21
0. 11.8856 272
0. 1.2522.230 239
0. 0.2877.522
0.5 0.5766.105
0. 0.67% 002
0.6366 010
0.540n.024 024
0. 1.458...22
0. 0.0000..0m 200
D. 0.6:40g 0554
.554
ResNet18
RcsNer18
PANDA
0. 0.9433..00 0001
00.9333 002
0. 0.837 .200
0-000
0. 1.8933..0m 026
0.921 008
0. 0.8488.. ans
0.9.00..00 000
0. 0.946
0. 0.933
0.9 0.9740.000
0..49% 0.004
0. 0.9331 13310.000
0) 0..938p.oo 003
PANDA
1.915 15.002
0,909 oon
() 1.823
0.8688 001
0. 0.9004..000
0.8570.005
0. 7,857 0.003
0. 0.903
0.917 002
0. 0.895..0r 020
0.9 0.946..co0 n08
0.920n. ons
0.881 0.881 n.012
0.911 0.004
RSNA
0. 0.888%.00
11 1L883 1.888)..00
0. 0.8488.017
0.674b. 157
0.6890.003
0.888..007
0 0.8900.045 0.04
0. 0.906o.007
0. 0.883..00
0 0.869..011
n..0
0.868..013
0.889).019
RSNA
0.787
0. 0.7966..111
0.71%%..o
0.161. 291
0.2 0.2400012
0.78 0.782n.064
0.775
0.8256.020 028
0.7166 031
0.7166.000 006
0.805 017
0.62lo.000
0 0.8055..00 028
DeepGraphsSury
SmAP
ABMIL
DeepGraphSure
CLAM
DSMIL
PabGCN
DFTD-MIL
SmTAA
TransMIL
SETMIL
GTP
IBMIIL
CAMIL
SmAP
ABMIL
CLAM
DSMIL
PathGON
DFTD-MI MIL
SmTAP
TransMIL
SETMIL
GTP
IBMI.
CAMIL
Without global interactions
With global interactions
interactions
With global interactions
Without global
19

---

C Additional figures
II
h:
tunh(W
hn
tanh (W
Lunh
both
Tn
boih
H Softmar
Softmar (P)
(a) ABMIL.
(b) SmtP-late.
11
h:
Eanch W.
Lach:W.)
A
Lark:WW
A
3a
Tuy
Ix
Linh W
fx
Ix
(c) SmAP-mi mid.
(d) SmAP-early.
Figure 8: Graphical representation of the different variants SmAP-late, SmAP-mid, SmAP-early. The well-known ABMIL, which we build upon, is shown in (a).
20

---

CT scan
Slice labels
SmAP
ABMIL
CLAM
DSMIL
100000000000000
DFTD-MIL
TmPAP
TransMIL
SETMIL
GTP
CAMIL
Figure 9: RSNA attention maps.
21

---

Atrantion score
WSI
Patch labels
Sm TAP
Trans
SETMIL
GTP
CAMIL
SmAP
ABMIL
CLAM
DSMIL
DFTD-MIL
Figure 10: PANDA attention maps.
22

---

WSI
Patch labels
SmTAP
TramsMIL
SETMIL
GTP
CAMIL
SmAP
ABMIL
CLAM
DSMIL
DFTD-MIL
Figure 11: CAMELYON16 attention maps.
Groung truth
ABMIL
SmAP(a 0.1)
SmAP(a 0.5)
SmAP(a 0.9)
Figure 12: Ground truth and SmAP attention maps of three different WSIs from CAMELYON16. As expected theoretically, larger produces smoother attention maps.
23

---

Arteaiio cala
Athelio Va.
Abodie raki
Abeeien ToLR
Sm
TransMIL
SETMIL
GTP
CAMIL
Neyate fartoes
SmAP
ABMIL
CLAM
DSMIL
DFTD-MIL
Figure 13: RSNA attention histograms.
Ddla
Atteza
TmP
TransMIL
SETMIL
GTP
CAMIL
Amvation vdlan
Atsuttos valan
Atsetion
SmAP
ABMIL
CLAM
DSMIL
DFTD-MIL
Figure 14: PANDA attention histograms.
24

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our claims are supported by Sec. (unified view of deep MIL methods), by Sec. 4 (derivation of the proposed smooth operator Sm), and by Sec. 5 (experimenall results).
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of the proposed method in Sec.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
25

---

Justification:
The full set of assumptions is included along with the proof of each result, as well in the main text, see Appendix A and Sec.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
We provide the details about the datasets and experimental configuration we used in Appenddix B The code associated with this paper has been uploaded supplementary material to OpenReview, and will be made public on GitHub upon the acceptance of the paper. Also, the datasets are available publicly on Kaggle (RSNA and PANDA) and the Registry of Open Data on AWS (CAMELYONI)).
Justification: We describe the proposed method in Sec. 4.3
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should be possible for other researchers to have some path to reproducing verifying the results.
26

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
The code associated with this paper has been uploaded as supplementary material to OpenReview, and will be made public GitHub upon the acceptance of the paper. Also, the datasets are available publicly on Kaggle (RSNA and PANDA) and the Registry of Open Data AWS (CAMELYON16).
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
www.c//ggieeeeeeee htt////oodiiissseeeeeeeee SusSSuumissss iilammsssion sssiissssiccciyiii for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
We provide the details about the training configuration have used in Appendix
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yss]
Justification: Our results presented as the mean and standard deviation obtained in five independent runs, see Sec. 5 and Sec. B.3] Due to space reasons, for Table we only provide the mean values.
Guidelines:
The answer NA means that the paper does not include experiments.
27

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (c.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: This information can be found along with the rest of the experimental configu- ration information, see Appendix B
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
Our research completely conforms with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
28

---

Justification: As pointed out in Sec. I and in Sec. 6 one of the main goals of this work is draw attention the localization task in MIL, which can contribute favorably to the deployment of computer-aided systems in the real world. However, as we also indicate in Seec. 6] for this to happen safely (e.g., avoiding misdiagnosis), further investigation needed about equipping them with uncertainty estimation mechanisms.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (c.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We cite the original works where the methods were proposed. We used the code provided by the corresponding authors. Each paper includes the URL to its GitHub repository, and each implementation released under (possibly) a different license, specified in the mentioned repository.
29

---

Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Our code has been uploaded supplementary material, along with instructions to preprocess each dataset and replicate the results of the experiments.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
30

---

Answer:
[NA]
Justification:
Our paper does not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
31


Reviews:
[{"review_id": "578l6LCkCo", "reviewer": "Reviewer_jUey", "summary": "The authors propose a technique to improve the localization capabilities of the current MIL, especially for CAD models that perform CT and WSI analysis. The method is based on seeing the attention attributed to each patch as a graph and minimizing its Dirichlet energy, promoting smooth transitions on the attention values of neighbor patches, which is in accordance with the assumption that neighboring patches share the same labels.", "questions": "The formulation seems to assume bags of equal size, which is not the case in MIL (in general). \n\nThe proposed method (loss) may have the drawback of over-smoothing the transition between tiles, making the localization regions blurry. Is this true? Could this be overcome?", "limitations": "yes", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The work is well-developed and well written, with a methodology explained through concise equations and theorems that guide the reader through the development of the idea. The results validate what is proposed, but minor adjustments and additional information will make the work even more valuable.", "weaknesses": "The method introduces the smooth operator on top of the existing ABMIL model. The proposed approach improves the performance of ABMIL on the CAMELYON dataset, both on localization and classification. Nevertheless, when analyzing the attention maps provided in the appendix, the differences between ABMIL and smAP are minimal. Although we have a numerical evaluation of the Dirichlet energy, we cannot tell if the drop in energy is significant only by these numbers, so this visual empiric evaluation is important for a better insight regarding the developed work. \n\nObserving the appendix figures, ABMIL already performs well in matching its attention to the tiles annotations. I suggest the authors apply the smooth operator on top of other models that did not perform well on CAMELYON for this task, such as CLAM, CAMIL, and SETMIL, to evaluate how minimizing the Dirichlet energy of the attention improves the localization task quantitatively and qualitatively.\n\nI suggest the authors perform an ablation study to evaluate how minimizing the Dirichlet energy compares to other smoothing strategies. For instance, summing the variance of the attentions to the model’s loss may provide similar results as minimizing the Dirichlet energy. \n\nThe analysis of the alpha parameter is great, but again, a qualitative approach is valuable for this case. Seeing how the attention map changes as the parameter alpha increases is highly valuable for the reader.\n\nLines 127 to 131: trying to debate equations not present in the paper is not good for the reader. I suggest writing the debated equations in the article or explaining the arguments without mentioning them.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Rev. jUey for their positive and valuable feedback. Next we address their concerns.\n\n**Visual comparison between ABMIL and SmAP.**\nAs pointed out by Rev. jUey, the fact that the Dirichlet energy decreases when using the proposed operator (Table 3 in the paper) does not imply that the instance-level performance (localization task) or the bag-level one (classification task) are enhanced. \nSuch enhancement can only be assessed by computing performance metrics both at instance and bag level, which is done in Tables 1 and 2 in the paper, respectively. Those tables show that SmAP outperforms ABMIL, especially at localization (see e.g. CAMELYON16, where they obtain 0.96 vs 0.82 in AUROC and 0.84 vs 0.77 in F1-score). Although the attention maps shown for SmAP and ABMIL in CAMELYON16 are very similar, please note that this is the visualization of *just one WSI*, which was selected at random (no cherry-picking). Instead, numerical results in Tables 1-2 in the paper consider all the 130 WSIs in the test set, not only one. Yet, we agree it can be misleading to use only one test WSI for visualization. Thus, in the rebuttal PDF we include three different test WSIs of CAMELYON16, where the overall enhancement of SmAP over ABMIL can be appreciated. It is interesting to see how SmAP better defines the positive (red) region thanks to the smooth operator.\n\n**Applying Sm on top of other baselines.**\nAs explained above, if we look at aggregated results of Table 1 in the paper instead of a particular WSI, ABMIL is not the best-performing among the baselines. Yet, it is worth exploring how other approaches behave when being combined with Sm. We have done it for the rest of the baselines w/o global interactions, i.e. CLAM, DSMIL, and DFTD-MIL, using CAMELYON16. Notice that adding Sm on top of the baselines w/ global interactions is not as natural, as those methods carry their own mechanisms to account for interactions. The results are in the table below. The improvements when adding Sm are in bold. \n\n|Model|Sm|Instance||Bag||\n|-|-|-|-|-|-|\n|||AUROC|F1-score|AUROC|F1-score|\n|CLAM|No|$0.849_{0.044}$|$0.821_{0.046}$|$0.960_{0.029}$|$0.897_{0.012}$|\n||Yes|$\\mathbf{0.928}_{0.028}$|$\\mathbf{0.873}_{0.018}$|$\\mathbf{0.966}_{0.007}$|$0.889_{0.017}$|\n|DSMIL|No|$0.76_{0.078}$|$0.654_{0.203}$|$0.947_{0.085}$|$0.866_{0.136}$|\n||Yes|$\\mathbf{0.960}_{0.013}$|$\\mathbf{0.776}_{0.088}$|$\\mathbf{0.967}_{0.011}$|$\\mathbf{0.919}_{0.018}$|\n|DFTD-MIL|No|$0.884_{0.002}$|$0.742_{0.040}$|$0.983_{0.010}$|$0.937_{0.013}$|\n| |Yes|$0.884_{0.183}$|$\\mathbf{0.836}_{0.222}$|$0.978_{0.158}$|$0.903_{0.183}$|\n\nThe conclusion is similar to the main manuscript: instance-level performance is enhanced (greatly in some cases, e.g. an increase from 0.76 to 0.96 in AUROC for DSMIL), whereas bag-level results are competitive. The decrease in bag-level for DFTD-MIL is explained because this method randomly splits each bag into different *chunks*, which may lead to the loss of local interactions exploited by Sm (e.g., if two adjacent instances end in different chunks).\n\n**Loss-based strategy for smoothing.**\nIntroducing a penalty term in the loss to favor smoothness is a natural alternative to the proposed operator. However, there is a key difference, since the use of a penalty term does not modify the model architecture. The penalty term favors that the learned weights encode such property, but it is not encoded explicitly in the model. For instance, at test stage, the penalty term is not used. Still, we find interesting to include this ablation study. The results are in Table 1 in the global response above, as this was also asked by Rev. YAE1. We conclude that, although differences are not large, Sm obtains superior performance.\n\n**Qualitative assessment of $\\alpha$.**\nIn the rebuttal PDF, we include the attention map by SmAP with three values of $\\alpha$ for the same WSIs used before. As theoretically expected, higher $\\alpha$ introduces more smoothness. Yet, as reported in the paper, the performance difference among $\\alpha$'s is small, and the largest difference is against ABMIL (which treats every instance independently).\n\n**On lines 127-131.**\nThanks for the suggestion, which helps to clarify the paper. In the revised version we have included the equations explicitly, as this is an important insight into why previous approaches that account for local interactions obtain low performance at instance level.\n\n**Bags seem to be of equal size.**\nThanks for pointing this out, the formulation in lines 94-95 can be misleading. No, bags do not need to be equal in size. Indeed, in the description of the RSNA dataset (Appendix B.1), we mention that there are 492 scans (bags) whose size varies from 24 to 57. The same applies to the rest of the datasets, where different WSIs have different number of patches. We have clarified this explicitly in lines 94-95.\n\n**Over-smoothing.**\nThank you, this is a common problem when introducing spatial correlations, not only in MIL but also in very related image processing techniques (e.g. deblurring). To address it, in our paper we use a “soft” adjacency matrix, following previous approaches, e.g. [13]:  if two instances $(i,j)$ are not adjacent, then $A_{ij}=0$. If they are adjacent, the value is $A_{ij} = \\exp\\left(-P^{-1}\\sqrt{|| \\mathbf{h}\\_i  -\\mathbf{h}\\_j ||}\\right),$\nwhere $\\mathbf{h}\\_k \\in \\mathbb{R}^P$ is the embedding of each patch through the feature extractor. \nThis $A_{ij}$ is in the interval $[0,1]$, and increases as the embeddings become closer.\nNote that the degree of smoothness depends on the *visual similarity of the instances*, avoiding unintended over-smoothing when positive and negative instances are not close. \nWe have noticed that this was not specified in the paper (it could only be seen in the code). We have included it in the revised version.\n\n---\nWe hope these responses and additional materials make the work more valuable to the reviewer.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "l7dqzI93S7", "reviewer": "Reviewer_YAE1", "summary": "The paper proposed a multi-instance learning approach. The basic idea is to make use of the spatial dependency between training samples. A smoothing operator was proposed to regularize the attention matrix with respect to inter-sample similarity (to my understanding), which the authors claimed to improved both quality of localization and classification tasks.", "questions": "1. The overall principle of the method need rephasing, i.e., how introduction the SM transformation (instead of penalty) improves classification? \n2. The smoothness assumption seems to be a property connected with localization. However, in 4.1-4.3 and Fig.3 the authors mainly present the SM operator for  classification. Yet in the experiment part, it seem that adding SM had more advantages in localization (but not sure how is it achieved?). The authors need to clarify the relationship between the two tasks.\n3. In Tab. 1, why the performance of the 'with global interactions' variants worse than the 'without' one in localization, but it is opposite in the classification (Tab. 2)?", "limitations": "The method required strict ordering (or spatial information) or 3d-to-2d samples, which is quite a strong demand but it is not discussed in the limitations.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The idea seems unique and based, but more justifications are merited.", "weaknesses": "1. Although the smoothness property do exist in multi-instance learning as the authors claimed and may introduce useful information. It is still unclear why imposing it in the neural network helps with classification. It doesn't seem to serve as an induction bias from its looks.\n2. Actually only 3 dataset are tested (though each with different variants), and the results are not so persuasive. Especially for the classification tasks that the authors want to credit, they failed to get 1st place in 2/3 of the AUC or F1 scores.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your feedback. First we provide a general clarification. Then we address the rest of points.\n\n---\n**GENERAL CLARIFICATION**\n\nIt is unfortunate that the concepts of classification and localization used in the manuscript are not found clear. These concepts are taken from the MIL literature, especially from the medical-related one. Next, we try to clarify them.\n\nGiven a test medical image (e.g., a WSI image), our goal is twofold: 1) make a global prediction of whether the WSI contains tumor or not, and 2) predict which patches are the tumorous ones. The former task is called **classification**, and the latter **localization**. Notice that, from the ML viewpoint, both are classification tasks. \n\n**Labels available during training**. The training dataset only contains labels at the WSI level. We know whether each training WSI is cancerous or not, but not which patches contain the tumor. This is exactly the MIL paradigm: WSIs are bags and their patches are instances. Under this MIL setting (bags $\\equiv$ WSI, instances $\\equiv$ patches), the classification/localization results are usually called bag-/instance-level results, respectively. \n\n**Making predictions at instance-level (localization) although only bag-level labels are available during training**.\nThis is precisely what MIL methods address. Generally, the idea is to process the instances inside a bag and then aggregate them through some instance-wise attention values ($f_n$ in Figure 1), which reflect the relevance of each instance. Then, these values are used for instance-level prediction.\n\n**We focus on localization task**.\nCurrent approaches are good at the classification task, but their localization results are comparatively limited. Thus, our goal is to enhance localization while staying competitive in classification. This focus is reflected in the paper title “enhanced localization…”. Since both tasks are related, we find that our approach is also slightly superior in classification. We hope this clarifies this sentence by the reviewer: “Especially for the classification tasks that the authors want to credit”.\n\n**The rationale behind our proposal to enhance localization**.\nCurrent MIL methods are designed to target the classification task. In terms of localization, they pinpoint some regions of interest, but a systematic instance-level evaluation reveals low performance metrics. We find these methods do not account for spatial correlations among instances (patches) or, if they do (as CAMIL or GTP), they consider them only to obtain the bag-level aggregation, but not for instance-level predictions. Thus, our idea is to encode these local dependencies into the MIL model in a principled way (through Dirichlet energy minimization with theoretical guarantees). Intuitively, these local correlations tend to uniformize instance-level predictions, avoiding isolated false positives or false negatives.\n\n---\n**REMAINING POINTS**\n\n**Why imposing smoothness helps**.\nThe basic idea is that in real-world medical images the instance labels tend to have a smooth distribution. It is not realistic to have isolated negative patches inside positive regions and the other way around. This uniformity is the inductive bias introduced by our operator. Since we are acting at instance-level, this has a clear effect in localization, see Table 1 in the paper. As a byproduct, since the WSI prediction is aggregated from instance-level info, it also improves the classification task (Table 2), although the margin is lower. \n\n**Persuasiveness of results and number of datasets.**\nRecall we expect the main improvement to occur at localization (Table 1). Regarding classification results (Table 2) the differences are lower as expected, but the proposed methods are still the best-performing ones. \nRegarding the number of datasets, the comprehensive evaluation of localization requires that the test set is completely labeled (e.g., every patch in the WSIs). This restricts the number of datasets available. Most recent MIL papers in top venues use three or fewer datasets. The papers [13], [37], [38], [29] (ICLR 24, MICCAI 22, IEEE TMI 22, NeurIPS 21) contain, respectively, 2, 2, 2 and 3 datasets.\n\n**Penalty as an alternative to the proposed operator.**\nIntroducing a penalty term in the loss to favor smoothness is a natural alternative to the proposed operator. This is also commented by Rev. jUey. Please see the response to them (paragraph headed \"Loss-based strategy for smoothing\"), which includes an ablation.\n\n**W/ and w/o global interactions behave differently in Tables 1-2**. Yes, this is an interesting observation that points out a limitation of transformers in MIL. Transformers excel at aggregating/encoding a complex sequence (e.g. a WSI) to make a global prediction, which is the goal of the classification task. That’s why methods with global interactions (mainly transformers) outstand in Table 2. However, they yield lower performance at the localization task (instance-level predictions, Table 1). This is because the self-attention layers of transformers lead to a loss of signal in the individual tokens of the sequence (e.g., the WSI patches). After such layers we obtain a valuable aggregation, but instance-level information has been degraded. Indeed, the proliferation of transformers in MIL is yet another sign that the literature has so far overlooked the localization task, which is key for real-world deployment.\n\n**Limitations on the spatial information**. We acknowledge that this method is not applicable to MIL problems where there is not a spatial structure of the bag. However, medical images (e.g. WSIs and CT scans) inherently possess this structure. To handle it efficiently, we use the torch.sparse API (a common approach in GNNs). This way, the memory overhead is minimal compared to the bag size.\n\n---\n\nWe hope the reviewer reconsiders the evaluation of our contribution in the light of this clarification. Please let us know if something remains unclear.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AD5l5fg5Ge", "reviewer": "Reviewer_uFQt", "summary": "The paper is well-written and proposes a new modular mechanism that can (selectively) combine local and global interactions in a model. The authors discuss the theory behind their proposed smoothness operator and pooling mechanism and provide detailed proof for their claims. They have also evaluated their model against other methods on three different datasets (1 CT and 2 H&E) with reporting AUC and F1 score as their evaluation metrics. Their method is superior to other methods and the proper ablation study supports their claims.", "questions": "I am curious if the author expects the same performance boost obtained on top of resent, to be obtained on top of a histopathology-specific feature extractor.", "limitations": "It has been justified :)", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The writing is concise and the contributions are clear. The author proposes a new smoothness mechanism, generally an interesting idea, to integrate local and global information. Besides the quantitative experiments, there are some qualitative experiments such as attention maps that show the model's relative importance on different regions in a slide.", "weaknesses": "There are three major concerns with this work:\n1. The choice of encoder. The authors have used Resnet18 for RSNA and Camelyon16 and Resnet50 for PANDA. I could not find any justification for why different encoders have been used! Also, with the trend toward foundation models, transformer-based backbones are now of interest to the community. Therefore, for the sake of consistency, the author should use the same encoder for different datasets or report both resent50 and resnet18. And, for the sake of the generality of their work, they should add one encoder such as ViT or Swin as well (w/ ImageNet weights) to support the generality of their claims.\n\n2. The body of research has been founded on local-to-global interactions. There are quite a few standard graph-based methodologies in the literature that the authors need to compare their work against. Currently, there are no representatives from those families of the MIL method in the benchmark. Two examples of such methods are (1) and (2).\n\n3. There is a family of MIL methods in the literature that try to pseudo-label the instances during training, which is essentially equivalent to localization in this work. For instance, two of the most recent such methods are (3) and (4). The author should compare their method against these as they are essentially from the same family.\n\n(1) Chen, Richard J., et al. \"Whole slide images are 2d point clouds: Context-aware survival prediction using patch-based graph convolutional networks.\" Medical Image Computing and Computer Assisted Intervention–MICCAI 2021: 24th International Conference, Strasbourg, France, September 27–October 1, 2021, Proceedings, Part VIII 24. Springer International Publishing, 2021.\n\n(2) Li, R., Yao, J., Zhu, X., Li, Y., Huang, J. (2018). Graph CNN for Survival Analysis on Whole Slide Pathological Images. In: Frangi, A., Schnabel, J., Davatzikos, C., Alberola-López, C., Fichtinger, G. (eds) Medical Image Computing and Computer Assisted Intervention – MICCAI 2018. MICCAI 2018. Lecture Notes in Computer Science(), vol 11071. Springer, Cham. https://doi.org/10.1007/978-3-030-00934-2_20\n\n(3) Z. Shao et al., \"LNPL-MIL: Learning from Noisy Pseudo Labels for Promoting Multiple Instance Learning in Whole Slide Image,\" 2023 IEEE/CVF International Conference on Computer Vision (ICCV), Paris, France, 2023, pp. 21438-21438, doi: 10.1109/ICCV51070.2023.01965.\n\n(4) Ren, Q. et al. (2023). IIB-MIL: Integrated Instance-Level and Bag-Level Multiple Instances Learning with Label Disambiguation for Pathological Image Analysis. In: Greenspan, H., et al. Medical Image Computing and Computer Assisted Intervention – MICCAI 2023. MICCAI 2023. Lecture Notes in Computer Science, vol 14225. Springer, Cham. https://doi.org/10.1007/978-3-031-43987-2_54", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Rev. uFQt for their valuable insights. Next we address their concerns.\n\n**The choice of encoder.**\nWe apologize for not providing a rationale for the chosen encoder in each dataset. When we started our experiments, we fixed ResNet18 w/ Imagenet weights as the default one. This is the one used for RSNA and PANDA, since baselines obtained results comparable to those reported in previous works. However, when using this encoder in CAMELYON16, we observed that all methods were performing worse than reported in previous papers. We switched to ResNet50 with Imagenet weights, but observed a similar behavior. \nWe found that some papers were using a feature extractor pre-trained with SSL on very large datasets [13, 20], so we decided to use one of these. Specifically, we used the weights provided by [17], which correspond to a ResNet50 model trained using Barlow Twins on a huge dataset of WSI patches (around 32.5 millions of patches).\nThis led to the baselines obtaining comparable results to previous papers, so we decided to fix this encoder for CAMELYON16.\n \nBy using two different encoders (ResNet18 and ResNet50 with Barlow Twins) we intended to show that the proposed method obtained improvements in localization/classification regardless of the type of features used. Yet, for the sake of generality, we agree that it’s interesting to report all the datasets with a wide range of encoders. Following the suggestion, we have run all the experiments using three different encoders (ResNet18, ResNet50 and ViT-B-32; all w/ ImageNet weights). Results at the instance and bag levels are included in Tables 1 and 2 of the rebuttal PDF. As a summary, the following table shows the average rank obtained by each method using the suggested encoders. The best result within each group is bolded, and the second-best is underlined. SmAP and SmTAP obtain in almost all cases the highest rank. This supports that the improvement by Sm does not depend on the used features.\n\n|||ResNet18||ResNet50||ViT-B-32||\n|-|-|-|-|-|-|-|-|\n|||Inst. rank|Bag rank|Inst. rank|Bag rank|Inst. rank|Bag rank|\n|W/o global int.|SmAP|$\\mathbf{1.667}_{0.516}$|$\\underline{1.917}_{0.917}$|$\\mathbf{1.750}_{0.707}$|$\\mathbf{1.938}_{1.084}$|$\\mathbf{1.667}_{0.816}$|$\\mathbf{1.500}_{1.225}$|\n||ABMIL|$2.667_{1.366}$|$\\mathbf{1.667}_{0.816}$|$3.750_{1.581}$|$3.188_{0.843}$|$4.333_{2.160}$|$3.000_{0.894}$|\n||CLAM|$6.167_{0.983}$|$5.500_{1.225}$|$4.750_{2.053}$|$4.375_{2.134}$|$6.167_{0.983}$|$5.000_{2.449}$|\n||DSMIL|$5.167_{0.983}$|$5.333_{1.506}$|$6.375_{0.518}$|$6.000_{0.926}$|$5.667_{1.033}$|$6.667_{0.516}$|\n||PathGCN|$5.833_{1.472}$|$5.250_{1.605}$|$5.625_{1.302}$|$4.688_{2.017}$|$4.667_{1.633}$|$4.000_{1.549}$|\n||DFTD-MIL|$\\underline{2.000}_{0.894}$|$2.833_{1.169}$|$3.125_{1.553}$|$\\underline{2.188}_{1.413}$|$\\underline{2.167}_{0.983}$|$\\underline{2.917}_{1.068}$|\n||DeepGraphSurv|$4.500_{1.225}$|$5.500_{1.225}$|$\\underline{2.625}_{1.188}$|$5.625_{0.916}$|$3.333_{1.211}$|$4.917_{0.376}$|\n|W/ global int.|SmTAP|$\\mathbf{2.167}_{1.835}$|$\\mathbf{1.667}_{1.033}$|$\\mathbf{2.125}_{1.553}$|$\\mathbf{1.625}_{0.518}$|$\\mathbf{1.833}_{0.983}$|$\\mathbf{2.500}_{0.837}$|\n||TransMIL|$3.250_{1.255}$|$3.917_{1.201}$|$3.250_{1.035}$|$3.375_{0.916}$|$4.167_{1.329}$|$4.417_{1.908}$|\n||SETMIL|$3.667_{0.816}$|$3.417_{1.908}$|$4.500_{0.837}$|$5.000_{0.632}$|$3.667_{1.506}$|$3.333_{1.862}$|\n||GTP|$4.083_{1.114}$|$3.250_{1.782}$|$4.250_{1.282}$|$3.250_{2.053}$|$5.083_{1.201}$|$4.833_{1.329}$|\n||IIBMIL|$5.167_{2.041}$|$5.667_{0.816}$|$4.250_{2.252}$|$4.875_{1.553}$|$4.167_{1.722}$|$3.250_{2.139}$|\n||CAMIL|$\\underline{2.667}_{1.751}$|$\\underline{3.083}_{0.801}$|$\\underline{2.250}_{1.165}$|$\\underline{2.625}_{1.061}$|$\\underline{2.083}_{1.281}$|$\\underline{2.667}_{1.211}$|\n\n**Graph-based baselines.** \nWe intended to cover these baselines by using GTP (which uses a graph convolutional network before the transformer encoder) and CAMIL (which represents the bag as a graph to mask the attention matrix). Yet, we agree that including more baselines can contextualize better. Note that the suggested methods (PathGCN and DeepGraphSurv) were proposed for survival analysis, which is a regression-related task. We have minimally adapted them for classification and run them in the three datasets, with the existing feature extractors and the new ones mentioned above. Results are included in the aforementioned tables (rebuttal PDF and summary table above).  \n\n**Pseudo-labels baselines.**\nAs mentioned by the reviewer, methods that assign pseudo-labels to instances can naturally address localization.\nThese methods fit the general framework described in our paper (take $f_n$'s in Fig. 1a to be the pseudo-labels). We tried to cover this family in the baselines with DSMIL (which supervises pseudo-labels using the max operator and CE loss) and CLAM (which supervises them using a clustering loss). Yet, we agree it's richer to include other alternatives. Of the two methods suggested, we have included IIBMIL in the complete experimentation (see the rebuttal PDF and the summary table above). The reason for not including (3) is that it requires access to (a low percentage of) training instance labels. These labels are used to train a weak classifier to generate a set of pseudo-labels. Instance labels are unavailable in our setting during training, and the rest of methods don't use this information. Therefore, it would be a misleading/unfair comparison.\n\n**On the last question/curiosity.** \nAs explained above, the results for CAMELYON16 in the paper are obtained using a histopathology-specific encoder (ResNet50 with Barlow Twins). The performance boost occurs there just as when using generic encoders.\nOur intuition is that Sm should work as long as extracted features carry valuable signal. If a poor encoder is used, then Sm may not lead to any improvement. \n\n---\nWe believe we are addressing the three raised concerns, making a more valuable work. Please let us know if something remains unclear.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
## **GLOBAL RESPONSE**

We **thank the three reviewers** for taking their time to read our work, providing constructive and valuable feedback. We appreciate it.
We are happy that **the feedback is in overall positive**: "well-developed and well written", "minor adjustments and additional information will make the work even more valuable", "the idea seems unique and based", "the writing is concise and the contributions are clear", "generally an interesting idea" etc.

**Several concerns** have been raised too. **Most of them are specific points** which we believe we have been able to address and/or clarify in the responses. We hope the reviewers find it valuable and they consider raising their scores. Please do not hesitate to request any additional clarification during the discussion phase. 

The response is organized as follows:

* This **global response** contains a general overview as well as a table requested by more than one reviewer. Each reviewer will be pointed to these tables from their individual response. 
* We also attach a **one-page rebuttal PDF file** to this global response. It contains only figures and tables. Each reviewer will be pointed to these items from their individual response. 
* We provide an **individual response** for each reviewer. 

After reading this general response, we encourage each reviewer to read their individual one, which will include or point to supporting tables and figures.

---

### **GENERAL OVERVIEW**

The following list provides a brief summary of our response to the main concerns expressed by the reviewers. We have
* Added three new baselines. [Reviewer uFQt]
* Extended all the experimentation using the three same encoders for all the datasets. [Reviewer uFQt]
* Provided a thorough clarification for the proposed approach as well as for key concepts in medical-related MIL (classification and localization tasks). [Reviewer YAE1]
* Included an ablation against an alternative way of favoring smoothness. [Reviewers YAE1 and jUey]
* Included additional attention maps to provide a more representative visualization. [Reviewer jUey]
* Evaluated how the proposed operator performs when combined with other baselines. [Reviewer jUey]

---

TABLE 1: Comparing the proposed smoothing operator with a penalty-based mechanism to favor smoothness. Requested by Revs. jUey and YAE1.

|||RSNA||PANDA||CAMELYON16||
|-|-|-|-|-|-|-|-|
|||Inst. AUROC $(\uparrow)$|Bag AUROC $(\uparrow)$|Inst. AUROC $(\uparrow)$|Bag AUROC $(\uparrow)$|Inst. AUROC $(\uparrow)$|Bag AUROC $(\uparrow)$|
|W/o global int.|SmAP|$\mathbf{0.798}_{0.033}$|$0.888_{0.005}$|$\mathbf{0.799}_{0.005}$|$\mathbf{0.943}_{0.001}$|$0.961_{0.007}$|$\mathbf{0.965}_{0.007}$|
||ABMIL+PENALTY|$0.782_{0.050}$|$\mathbf{0.889}_{0.043}$|$0.780_{0.003}$|$0.935_{0.001}$|$\mathbf{0.979}_{0.013}$|$0.963_{0.012}$|
|W/ global int.|SmTAP|$\mathbf{0.767}_{0.046}$|$\mathbf{0.906}_{0.007}$|$\mathbf{0.790}_{0.007}$|$0.946_{0.003}$|$\mathbf{0.789}_{0.008}$|$0.976_{0.014}$|
||TAP+PENALTY|$0.737_{0.045}$|$0.905_{0.005}$|$0.772_{0.011}$|$\textbf{0.947}_{0.001}$|$0.769_{0.099}$|$\mathbf{0.988}_{0.004}$|

Meta-review: This paper introduces an approach to improve the localization task of multiple instance learning (MIL) methodologies, particularly the ones applied to CT and WSI. Relying on a spatial dependency between training samples, the paper presents a smoothing operator to regularize the attention matrix with respect to inter-sample similarity. The paper also presents a theoretical analysis of the method. The proposed method shows better results than competing approaches and the ablation study supports its claims. The paper received an average score of 5.33 (5,5,6), and the main strengths found are:  1) paper is well-written and concise, 2) claims are well supported by experimental results, 3) idea explored in the paper is interesting, and 4) qualitative experiments show the model’s advantages.  The reviewers also identified the following weaknesses: 1) negligible differences between ABMIL and the proposed method; 2) missing experiment on the application of the proposed smooth operator on top of other methods (e.g., CLAM, CAMIL, SETMIL); 3) missing ablation study on a comparison between the minimisation of Dirichlet energy and other smoothing strategies; 4) missing qualitative experiment on the role of alpha parameter; 5) unclear role of the smoothness property for the classification from a neural network; 6) unconvincing results using only 3 datasets, where the proposed method failed to get first place in 2/3 of the AUC or F1 scores; 7) unclear justification for different encoders for different datasets; 8) missing comparisons with graph-based methodologies; and 9) missing comparison with MIL methods that rely on pseudo labels. There was a fair bit of discussion during the rebuttal, and the only remaining issue of the paper is that the performance improvement compared to competing approaches is marginal. I support the publication of this paper, but encourage the authors to address the issues raised during this rebuttal phase.
Decision: Accept (poster)