Title: Sm: enhanced localization in Multiple Instance Learning for medical imaging classification
Venue: NeurIPS 2024
Accepted: yes
Keywords: Multiple Instance Learning

Transformers

Graph Neural Networks

Medical Imaging

Weakly Supervised Learning
Abstract: Multiple Instance Learning (MIL) is widely used in medical imaging classification to reduce the labeling effort. 
While only bag labels are available for training, one typically seeks predictions at both bag and instance levels (classification and localization tasks, respectively). Early MIL methods treated the instances in a bag independently. Recent methods account for global and local dependencies among instances. Although they have yielded excellent results in classification, their performance in terms of localization is comparatively limited. We argue that these models have been designed to target the classification task, while implications at the instance level have not been deeply investigated. Motivated by a simple observation -- that neighboring instances are likely to have the same label -- we propose a novel, principled, and flexible mechanism to model local dependencies. It can be used alone or combined with any mechanism to model global dependencies (e.g., transformers). A thorough empirical validation shows that our module leads to state-of-the-art performance in localization while being competitive or superior in classification. Our code is at https://github.com/Franblueee/SmMIL.

Sm : enhanced localization in Multiple Instance Learning for medical imaging classification

Abstract

Multiple Instance Learning (MIL) is widely used in medical imaging classification to reduce the labeling effort. While only bag labels are available for training, one typically seeks predictions at both bag and instance levels (classification and localization tasks, respectively). Early MIL methods treated the instances in a bag independently. Recent methods account for global and local dependencies among instances. Although they have yielded excellent results in classification, their performance in terms of localization is comparatively limited. We argue that these models have been designed to target the classification task, while implications at the instance level have not been deeply investigated. Motivated by a simple observation - that neighboring instances are likely to have the same label - we propose a novel, principled, and flexible mechanism to model local dependencies. It can be used alone or combined with any mechanism to model global dependencies (e.g., transformers). A thorough empirical validation shows that our module leads to state-of-the-art performance in localization while being competitive or superior in classification. Our code is at https://github.com/Franblueee/SmMIL .

1 Introduction

Over the last decades, medical imaging classification has benefited from advances in deep learning [35, 44]. However, the performance of these methods drops when the number of labeled samples is low, which is common in real-world medical scenarios [1]. To overcome this, Multiple Instance Learning (MIL) has emerged as a popular weakly supervised approach [14, 8, 12].

In MIL, instances are arranged in bags. At train time, a label is available for the entire bag, while the instance labels remain unknown. The goal is to train a method that, given a test bag, can predict both at bag and instance levels (classification and localization tasks, respectively). This paradigm is well suited to the medical imaging domain [28]. In cancer detection from Whole Slide Images (WSIs), the WSI represents the bag, and the patches are the instances. In intracranial hemorrhage detection from Computerized Tomographic (CT) scans, the full scan represents the bag, and the slices at different heights are the instances. In these scenarios, making accurate predictions of instance

labels is extremely important from a clinical viewpoint, as it translates into pinpointing the location of the lesion [7].

Most successful approaches in MIL build on the attention-based pooling [17], a permutation-invariant operator that assigns an attention value to each instance independently. This method has been extended in different ways while maintaining the permutation-invariant property [21, 25, 39]. The aforementioned works pose a problem: the dependencies between the instances, which are important when making a diagnosis, are ignored. To account for this, TransMIL [32] proposed to model global dependencies using a transformer encoder. The idea is to use the self-attention mechanism to introduce interactions between each pair of instances. Based on it, other transformer-based approaches have emerged, also focusing on global dependencies [9, 22, 37]. More recently, several works have also incorporated natural local interactions, which are those between neighboring instances [14, 40, 41].

Although these methods accounting for dependencies have resulted in excellent performance at the bag level, the evaluation at the instance level has received less attention and the results are not comparatively good so far, see the very recent [14]. In this work, we argue that recent MIL methods have been designed with the classification task in mind, and we propose a new model that focuses on both the classification and localization tasks. Specifically, we propose a novel and theoretically grounded mechanism to introduce local dependencies, hereafter referred to as the smooth operator Sm . This is a flexible module that can be used alone on top of classical MIL approaches, or in combination with transformers to also account for global dependencies. In both cases, we show that the proposed operator achieves state-of-the-art localization results while being competitive in classification. We compare against a total amount of eight methods, including very recent ones [14, 40]. We utilize three different datasets of different nature and size, covering two different medical imaging problems (cancer detection in WSI images and hemorrhage detection in CT scans).

Our main contributions are: (i) we provide a unified view of current deep MIL approaches; (ii) we propose a principled mechanism to introduce local interactions, which is a modular component that can be combined or not with global interactions; and (iii) we evaluate up to eight state-of-the-art MIL methods on three real-world MIL datasets in both classification and localization tasks, showing that the proposed method stands out in localization while being competitive or superior in classification.

2 Related work

In this work, we tackle the localization task in deep MIL methods using existing concepts and techniques from deep MIL and Graph Neural Networks (GNNs) theory.

Deep Multiple Instance Learning. As explained by Song et al. [34], deep MIL methods can be divided into two broad categories, namely instance-based or embedding-based, depending on the level at which a specific aggregation operator is applied. In this paper, we focus on the embedding-based category, and in particular attention-based ones.

Ilse et al. [17] proposed attention-based pooling to weigh each instance in the bag. To improve it, different modifications were proposed, including the use of clustering layers [25], grouping the instances in pseudo-bags [39], and using similarity measures to critical instances to compute the attention values [21]. However, these methods ignore the existing dependencies between the instances in a bag. To address this, Shao et al. [32] proposed to use a transformer-based architecture and the PPEG position encoding module. This has been extended with different transformer variations, including the deformable transformer architecture [22], hierarchical attention [37], and regional sampling [9]. Recently, these methods have been improved to include spatial information in different ways, including the use of a Graph Convolutional Network (GCN) before the transformer [41], a neighbor-constrained attention module [14], and a spatial-encoding transformer architecture [40].

In the studies mentioned above, the objective is to obtain increasingly better bag-level results, while the evaluation at the instance level is usually performed qualitatively. In contrast, our work addresses both the instance localization task and the bag classification task, as both are of great importance for making a diagnosis. Moreover, our work is not limited to WSI classification; it is also valid for other medical imaging modalities.

Graph Neural Networks. Our motivation - that neighboring instances are likely to have the same label - is a well-established assumption within the machine learning community, often referred to as the cluster assumption [10, 31]. Since leveraged in 1984 by Ripley [30] in the context of spatial

X

f

x

1

x

2

.

.

.

x

N

f

1

f

2

.

.

.

f

N

ˆ

Y

Deep MIL

method

(a)

General deep

MIL

model.

X

f

x

1

x

2

.

.

.

x

N

f

1

f

2

.

.

.

f

N

ˆ

Y

Agg.

NeuralNet

NeuralNet

NeuralNet

X

f

x

1

x

2

.

.

.

x

N

f

1

f

2

.

.

.

f

N

ˆ

Y

Global in-

teractions

(Transformer)

(c)

Only global

interac- tions.

X

f

x

1

x

2

.

.

.

x

N

f

1

f

2

.

.

.

f

N

ˆ

Y

Global in-

teractions

(Transformer)

+

Local in-

teractions

(GNN, ...)

Figure 1: (a) Unified view of deep MIL models. Depending on how instances interact with each other in (a), we devise three different families of methods: (b), (c), (d).

statistics, it has been extensively used in spectral clustering [27], semi-supervised learning on graphs [3], and recently in GNNs [20]. Our work builds upon seminal works in these areas.

The proposed smooth operator is derived considering a Dirichlet energy minimization problem, similar to the work by Zhou and Schölkopf [42] and Zhou et al. [43]. This approach has been employed in recent years to obtain new GNN models, including the p -Laplacian layer [15], and PPNP layer [16]. Moreover, the Dirichlet energy has been studied in the context of GNNs to analyze the over-smoothing phenomenon [6, 23]. In this regard, our bound on the decrease of the Dirichlet energy is analogous to the result derived by Li et al. [23] to study over-smoothing for GCNs. Our result, however, holds for the proposed mechanism, of which the graph convolutional layer is a special type.

3 Background: A unified view of deep MIL approaches

We first describe the binary MIL problem tackled in this paper. Then, we provide a unified view of the most popular deep MIL methods. As explained in Sec. 2, we focus on embedding-based approaches.

In MIL, the training set consists of pairs of the form ( X , Y ) , where X ∈ R N × P is a bag of instances and Y ∈ { 0 , 1 } is the bag label. We write X = [ x 1 , . . . , x N ] T ∈ R N × P , where x n ∈ R P are the instances. Each instance x n is associated to a label y n ∈ { 0 , 1 } , not available during training . It is assumed that Y = max { y 1 , . . . , y N } , i.e., a bag X is considered positive if and only if there is at least one positive instance in the bag.

Given a previously unseen bag (e.g., a medical image), the goal at test time is to: i) predict the bag label (classification task) and ii) obtain predictions or estimates for the instance labels (localization task). In general, deep MIL models output a bag-level prediction ˆ Y , as well as instance-level scalars f n that are used for instance-level prediction. This general process is depicted in Fig. 1a. In many approaches, these f n are the so-called attention values (e.g., ABMIL [17], TransMIL [32], CAMIL [14]), but they can be obtained in different ways (e.g., through GraphCAM in GTP [41]). Within the general process in Fig. 1a, deep MIL models can be categorized into three families, depending on how instances interact with each other, see Fig. 1b, Fig. 1c, and Fig. 1d.

In the first family, shown in Fig. 1b, the instances are encoded independently and then aggregated. The well-known ABMIL [17] fits in this paradigm. Subsequent works introduce slight modifications to ABMIL, while still encoding each instance independently [21, 25, 39]. ABMIL, on which we will rely to introduce our model, is depicted in Fig. 3a. First, a bag of embeddings H = [ h 1 , . . . , h N ] ∈ R N × D is obtained by applying a neural network independently to each instance. Then, the attention-based pooling computes the attention values f and the bag embedding z according to

where W ∈ R L × D , w ∈ R L are trainable weights. Last, ˆ Y is obtained by applying a linear classifier on z .

The second family accounts for global interactions between instances, possibly long-range ones, see Fig. 1c. These works treat instances as tokens that interact through the self-attention mechanism. This way, global interactions between instances are learned. One of the most popular approaches in this family is TransMIL [32], which was later extended in different directions [9, 22]. The third family complements the previous one with local interactions defined by a fixed neighborhood, see

(a) Labelled patches in a WSI.

00000000006

(b) Labelled slices in a CT scan.

Fig. 1d. They differ in how local interactions are represented, e.g., as a graph in CAMIL [14] and GTP [41], or using a position-encoded feature map in SETMIL [40].

In most of these works, the localization task is assessed qualitatively, e.g., by visually comparing the attention maps. This contrasts with the classification task, which is always evaluated quantitatively. As evidenced by Fourkioti et al. [14], this has translated into comparatively poor performance in terms of localization. We notice that current models have been designed to target the classification task, and they excel at that. However, their model design is not as careful about the instance-level implications. For example, CAMIL [14] does not leverage any local information to obtain the instance-level attention values. Indeed, from their Eq. (8) one deduces that the a i values are obtained from the tile representations t i , which have not undergone any local interaction. Observe that local interactions take place in Eq. (4) and Eq. (5) in their paper, but these only affect the bag-level predictions, not the instance-level ones. Similarly, GTP [41] introduces local interactions through an off-the-shelf graph convolutional layer, the effect of which is not investigated at the instance level. In the following section, we propose a principled approach to account for meaningful local interactions based on the Dirichlet energy. The idea is motivated by a natural property often observed in the instance-level labels of medical images: the similarity between neighboring instances.

4 Method: Introducing smoothness in the attention values

In medical imaging, instance labels are a priori expected to exhibit local dependencies with their neighboring instances: an instance is likely to be surrounded by instances with the same label, see Fig. 2. Recall that attention values are commonly used as a proxy to estimate these labels, so they should inherit this property. Based on these observations, our intuitive idea is to favor a smoothness property on the attention values. To this end, Sec. 4.1 formalizes the notion of smoothness through the Dirichlet energy. Sec. 4.2 presents the proposed smoothing operator Sm , which encourages smoothness as well as fidelity to the original signal. Sec. 4.3 proposes how to leverage Sm in the context of MIL, both in combination with global interactions (via transformers), and without them. We will build on top of the well-known and simple ABMIL to isolate the effect of Sm and avoid over-sophisticated models.

4.1 Modelling the smoothness

We represent each bag as a graph, where the nodes are the instances and the edges represent the spatial connectivity between instances. Formally, we suppose that each bag X ∈ R N × D has been assigned an adjacency matrix A = [ A ij ] ∈ R N × N , defined by A ij > 0 if instances x i and x j are neighbors, and A ij = 0 otherwise. We assume that the adjacency matrix is symmetric, i.e. A ij = A ji .

The Dirichlet energy is a well-known functional that measures the variability of a function defined on a graph [42, 43]. In our case, we think of this function as the attention values f ∈ R N , recall Fig. 1a. As we shall see below, it will be necessary to define the Dirichlet energy for multivariate graph functions. Given a multivariate graph function U = [ u 1 , . . . , u N ] ⊤ ∈ R N × D defined on the bag graph, the Dirichlet energy of U is given by

where ∥·∥ 2 denotes the Euclidean norm, L is the graph Laplacian matrix L = D -A , D ∈ R N × N is the degree matrix, D = Diag ( D 1 , . . . , D N ) , D n = ∑ i A ni . When D = 1 we obtain the definition for univariate graph functions, such as the attention values f .

Bounding E D on the attention values. In most deep MIL approaches, the attention values f are obtained by applying a neural network to instance-level features. One example is ABMIL [17], which uses a two-layer perceptron defined by Eq. 1. Noting that tanh is a Lipschitz function with Lipschitz constant equal to 1 , we arrive at the following chain of inequalities

where ∥·∥ 2 denotes the spectral norm. A more general result holds in the general case of an arbitrary multi-layer perceptron, see Appendix A for a proof. The above chain of inequalities tells us that if we want E D ( f ) to be low, we can act on f itself or on previous layers (e.g., on F or on H ) , constraining the norm of the trainable weights to remain constant. This constraint can be achieved using spectral normalization [26], and we study its influence in Sec. B.3. In the next subsection, we propose an operator that can be used on any of these levels ( f , F , H ) to reduce the Dirichlet energy of its output.

4.2 The smooth operator

Our goal now turns into finding an operator Sm : R N × D → R N × D that, given a bag graph multivariate function U ∈ R N × D , returns another bag graph multivariate function Sm ( U ) ∈ R N × D such that its Dirichlet energy is lower without losing the information present in the original U . Following seminal works [42, 43], we cast this as an optimization problem,

where α ∈ [0 , 1) accounts for the trade off between both terms, and ∥·∥ F denotes the Frobenius norm. The first term in the above equation penalizes functions with too much variability, while the second term penalizes functions that differ too much from the original U . Note that this can be interpreted as a maximum a posteriori formulation, where the first term corresponds to the prior distribution and the second to the observation model, see [30]. The objective function E is strictly convex, and therefore admits a unique solution, given by

where γ = α/ (1 -α ) . Unfortunately, the expression in Eq. (7), although elegant, incurs prohibitive computational and memory costs, especially when the number of instances in the bag is large (which is the case of WSIs). Instead, we can take an iterative approach, defining Sm ( U ) = G ( T ) , with

As demonstrated by Zhou et al. [43], the sequence { G ( t ) } converges to the optimal solution in Eq. 7. As studied by Gasteiger et al. [16], it is enough to use a small number of iterations T to approximate the exact solution. Therefore, in this work, we will adopt the iterative approach described by Eq. 8. Based on previous work [16], we will use T = 10 , and α will be set as a trainable parameter initialized at α = 0 . 5 . See Sec. B.3 for a study on the effects of these hyperparameters and Fig. 12 for a visual comparison of the effect that α has on the attention maps.

Theoretical guarantees via the normalized Laplacian. We present a result that informs us about the rate at which the Dirichlet energy decreases when applying Sm . Let us define λ ∗ γ = max { (1 + γλ n ) -2 : λ n ∈ Λ \ { 0 } } , where Λ = { λ 1 , . . . , λ N } are the eigenvalues of the bag graph Laplacian matrix. Then, we have the following inequality,

The proof is inspired by Cai and Wang [6], see Appendix A. If λ ∗ γ < 1 , then the smooth operator effectively decreases the Dirichlet energy. If we replace the Laplacian matrix by the normalized Laplacian matrix, ˜ L = D -1 / 2 LD -1 / 2 , it is known that its eigenvalues lie in the interval [0 , 2) , and then λ ∗ γ < 1 holds. This motivates the use of the normalized Laplacian in our experiments.

The smooth operator Sm only introduces one parameter to be estimated, α . Also, it is differentiable with respect to its input. Therefore, it can be integrated into simple attention-based MIL models, such as ABMIL, to account for local dependencies.

X

H

f

x

1

x

2

.

.

.

x

N

h

1

h

2

.

.

.

h

N

f

1

f

2

.

.

.

f

N

z

ˆ

Y

Att.

Pooling

Classifier

NeuralNet

NeuralNet

NeuralNet

(a) ABMIL.

X

H

f

x

1

x

2

.

.

.

x

N

h

1

h

2

.

.

.

h

N

f

1

f

2

.

.

.

f

N

z

ˆ

Y

Att.

Pooling

+

Sm

Classifier

NeuralNet

NeuralNet

NeuralNet

(b)

Sm

AP.

X

H

f

x

1

x

2

.

.

.

x

N

h

1

h

2

.

.

.

h

N

f

1

f

2

.

.

.

f

N

z

ˆ

Y

Trans-

former

+

Sm

Att.

Pooling

+

Sm

Classifier

(c)

Sm

TAP.

4.3 The proposed model

Here we propose how to leverage the operator Sm in the context of MIL. We build on top of the well-known ABMIL. First, we introduce Sm AP, which integrates ABMIL with Sm and only accounts for local interactions. Second, we introduce Sm TAP, which equips Sm AP with a transformer encoder to account for global dependencies. The proposed models are depicted in Fig. 3b and Fig. 3c. The details about the architecture we have used can be found in Sec. B.2.

Sm AP: Smooth Attention Pooling. This is represented in Fig. 3b. First, the bag of embeddings H is obtained as in ABMIL [17], i.e. treating the instances independently. Then, the operator Sm is integrated within the attention pooling. Based on Eq. 4, this can be done on the attention values themselves or on previous representations. We consider three different variants: Sm AP-late, Sm AP-mid, Sm AP-early. They act, respectively, on f (the attention values themselves), on F (i.e. before entering the last layer), and on H (i.e. before entering the attention-based pooling). Formally,

While Sm AP-late and Sm AP-mid act on the computation of the attention values, Sm AP-early acts on the embedding that is passed to the attention-based pooling, see Fig. 8 in Appendix C. We use Sm AP-early by default. Sec. 5.3 shows that results do not differ much among configurations.

Sm TAP: Smooth Transformer Attention Pooling. This is represented in Fig. 3c. The only difference with Sm AP is that the neural network acting independently on the instance embeddings is replaced by a transformer encoder to account for global dependencies. Based on the idea that smoothness can be imposed at previous locations, recall Eq. 4, we propose to also apply Sm to the transformer output:

where q , k , and v are the standard queries, keys, and values in the dot product self-attention [4]. Notice that Sm TAP uses Sm in two places: the first after the transformer encoder and the second in the aggregator. Naturally, one could think of other variants that use Sm in only one place or the other. In Sec. 5.3 we ablate these different configurations, leading to similar results.

5 Experiments

We validate the proposed Sm in three medical MIL datasets: RSNA [13], PANDA [5], and CAMELYON16 [2]. We evaluate the effectiveness of our approach by a quantitative and qualitative analysis. All experiments have been conducted under fair and reproducible conditions. Details on the datasets and experimental setup can be found in Appendix B. The code is available at https://github.com/Franblueee/SmMIL .

We compare our approaches with state-of-the-art deep MIL methods. We consider two groups of methods, depending on the presence/absence of a transformer block to model global dependencies. In the first group, we include those models that do not use this block: the proposed Sm AP, ABMIL [17], CLAM [25], DSMIL [21], and DFTD-MIL [39]. The second group consists of models that do use the transformer encoder: the proposed Sm TAP, TransMIL [32], SETMIL [40], GTP [41], and CAMIL [14]. These groups ensure a fair comparison in terms of model capabilities and complexity.

In Sec. B.3 we report the results of three more methods: DeepGraphSurv [24], PathGCN [11], and IIBMIL [29]. Note that the performance obtained by these methods does not affect the conclusions we will obtain in this section.

In Sec. 5.1 we consider the localization task. In Sec. 5.2 we turn to the classification task. Sec. 5.3 shows an ablation study on how different uses of the smooth operator affect the proposed model.

5.1 Localization: instance level results

In this subsection, we analyze the ability of each model to predict the label of the instances inside a bag. As explained in Sec. 3, deep MIL models assign a scalar value f n to each instance x n , see Fig. 1a. Although these can be obtained in different ways, for simplicity we will refer to them as attention values . Thus, we compare the attention values with the ground truth instance labels, which are available for the test set only for evaluation purposes.

Quantitative analysis. We analyze the performance of each method using the area under the ROC curve (AUROC) and the F1 score. Note that a critical hyperparameter for the latter is the threshold used on f n to determine the label of each instance. To ensure a fair comparison, we compute the optimal threshold for each method using the validation set. As a general summary, we also report the average rank achieved by each model across metrics and datasets.

The results are shown in Table 1. We find that using Sm provides the best performance overall, placing as the best or second-best within each group. Only in RSNA the proposed Sm AP is outperformed by ABMIL. We attribute this to the fact that the bag graphs in CT scans are not as complex as in WSIs, and therefore the local interactions are not as meaningful. Note that the performance gain is particularly significant on CAMELYON16, where the bags have a larger number of instances, the graphs are much denser and the imbalance between positive and negative instances is more severe. Notably, Sm TAP significantly outperforms SETMIL, GTP, and CAMIL, which also model local dependencies. Contrary to our method, their design is focused on bag-level performance and it does not translate into meaningful instance-level properties.

Attention histograms. We examine the attention histograms produced by each model on the CAMELYON16 dataset. The corresponding figures for RSNA and PANDA can be found in Appendix C. In Fig. 4, we represent the frequency with which attention values are assigned to positive and negative instances, separately. An ideal instance classifier would place all the positive instances on the right and all the negative instances on the left. This illustrates why Sm TAP and Sm AP achieve such a good performance: they concentrate the negative instances to the left of the histogram while succeeding in grouping a large part of the positive instances to the right. TransMIL and GTP assign low attention values to both positive and negative instances. CAMIL is able to identify positive instances, but negative instances are assigned from intermediate to high attention values. CLAM and DSMIL assign low attention values to negative instances, but the distribution of the positive instances resembles a uniform and a normal distribution, respectively.

Attention maps. To visualize the localization differences, we show the attention maps generated by four of the transformer-based methods in a WSI from CAMELYON16, see Fig. 5. Sm TAP attention

-

5

0

5

10

Attention value

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

Positive instances

Negative instances

-

5

.

0

-

2

.

5

0

.

0

2

.

5

5

.

0

Attention value

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

-

10

-

5

0

5

10

Attention value

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

-

10

-

5

0

5

10

Attention value

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

-

5

0

5

10

Attention value

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

Sm

AP

ABMIL

CLAM

DSMIL

DFTD-MIL

-

5

.

0

-

2

.

5

0

.

0

2

.

5

5

.

0

Attention value

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

0

.

0000

0

.

0002

0

.

0004

0

.

0006

Attention value

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

-

4

-

2

0

2

Attention value

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

0

.

000

0

.

005

0

.

010

0

.

015

0

.

020

0

.

025

Attention value

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

-

4

-

2

0

2

4

Attention value

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

Sm

TAP

TransMIL

SETMIL

GTP

CAMIL

0

1

Attention score

Patch labels

Sm

TAP

TransMIL

GTP

CAMIL

map resembles the most to the ground truth. As noted in Fig. 4, CAMIL assigns high attention values to both positive and negative instances. TransMIL and GTP pinpoint the regions of interest, but the attention is relatively low in those areas, which produces unclear boundaries, especially in the case of TransMIL. The attention maps for the rest of the methods and datasets are in Appendix C.

The results shown in this subsection validate the utility of the smooth operator at the instance level. They suggest that having smooth attention maps is a powerful inductive bias that improves the instance-level performance. In the following, we analyze its impact at the bag level.

5.2 Classification: bag level results.

In this subsection, we show that the use of the smooth operator does not deteriorate the bag classification results. On the contrary, in some cases, it improves them. Again, we focus on the AUROC and F1 scores, measured by comparing the true bag labels with the methods' bag label predictions. The threshold for the F1 score is 0 . 5 . We also report the mean rank achieved by each model.

Table 2 shows the results. The proposed models achieve the best performance overall. As in the localization task, ABMIL performs better than Sm AP in RSNA. Again, we believe it to be a consequence of the CT scan's low-complexity structure. DFTD-MIL obtains the best result in CAMELYON16, but ranks second or third in the other two datasets. GTP and SETMIL outperform the proposed Sm TAP in PANDA, but their performance significantly decreases in CAMELYON16, obtaining the worst results. Overall, our methods provide the most consistent performance, achieving an aggregated mean rank of 1.833.

5.3 Ablation study

The proposed Sm comes with different design choices and hyperparameters: the placement of Sm , the trade-off parameter α , the number of approximation steps T , and the use of spectral normalization. We analyze them in the following, showing that Sm leads to enhanced results almost under any choice. This supports that our hypothesis - that neighboring instances are likely to have the same label - is a powerful inductive bias worth exploring.

5.3.1 Placement of Sm

Recall that Sm AP leverages by default the early variation, but we also described Sm AP-mid and Sm AP-late. Likewise, we discussed different variants for Sm TAP. Table 3 summarizes the impact of these choices on the final performance.

Sm without the transformer encoder ( Sm AP). These variants differ in the place where Sm is located inside the attention pool, recall Eq. 10-Eq. 12. We include ABMIL since we build our model on top of it. We see that using Sm AP improves the performance at both instance and bag levels. This improvement is more noticeable in PANDA and CAMELYON. We attribute it to the bag graph structure being more complex in WSIs than in CT scans. Also, the Dirichlet energy is lower when the smooth operator is used, as theoretically expected. We observe that the proposed method is robust to different placement configurations, which is consistent with the theoretical guarantees presented in Sec. 4.1. However, none of the variants consistently outperforms the others.

Sm with the transformer encoder ( Sm TAP). Recall that Sm TAP leverages Sm both after the transformer encoder and inside the attention pooling. Here we will refer to it as Sm T+ Sm AP, and will compare it against T+ Sm AP and Sm T+AP (using Sm only in one of the components) and against T+AP (not using Sm ). We observe that Sm has no significant effect on bag-level performance. At instance-level we do observe differences: the baseline T+AP is outperformed as long as Sm is used within the attention pooling.

5.3.2 Sm hyperparameters

In the following we study the influence of the trade-off parameter α and of the spectral normalization. Due to space limitations, the analysis for the number of approximation steps T is in Sec. B.3.

The trade-off parameter α . From Eq. 6 we see that α ∈ [0 , 1) controls the amount of smoothness enforced by Sm . Note that α = 0 in Eq. 7 produces no smoothness, turning Sm into the identity operator. In Fig. 6a we show the performance obtained for different values of α in CAMELYON16. Each choice of this hyperparameter improves upon the baseline ABMIL ( α = 0 ). We see that better localization results are obtained when α is lower, while better classification results are obtained when

0

.

00

0

.

25

0

.

50

0

.

75

α

0

.

7

0

.

8

0

.

9

1

.

0

Inst. AUROC

0

.

00

0

.

25

0

.

50

0

.

75

α

0

.

90

0

.

92

0

.

94

0

.

96

0

.

98

1

.

00

Bag AUROC

early

mid

late

α

= trainable

(a) Trade-off parameter

α

.

No

Yes

Uses spectral normalization

0

.

7

0

.

8

0

.

9

1

.

0

Inst. AUROC

No

Yes

Uses spectral normalization

0

.

90

0

.

92

0

.

94

0

.

96

0

.

98

1

.

00

Bag AUROC

early

mid

ABMIL

(b) Spectral normalization.

α is higher. Fixing α = 0 . 5 is a compromise between the two, and produces very similar results as setting it as a trainable parameter initialized at α = 0 . 5 . Fig. 12 provides a visual comparison of the effect that α has on the attention maps.

The effect of spectral normalization. Spectral normalization forces the norm of the multi-layer perceptron weights to remain constant. In this work, this is a key design choice that helps Sm to obtain attention maps with lower Dirichlet energy. In our experiments, we have used spectral normalization in the layers immediately after Sm . Note that the late variant does not require spectral normalization, since it applies Sm directly to the attention values. In Fig. 6b we show the results obtained with and without spectral normalization in CAMELYON16. We observe that, even without spectral normalization, Sm improves upon the baseline. The improvement is more significant when Sm is paired with spectral normalization, especially at the instance level.

6 Discussion and conclusion

The main goal of this paper is to draw attention to the study of MIL methods at the instance level. To that end, we revised current deep MIL methods and provided a unified perspective on them. We proposed the smooth operator Sm to introduce local interactions in a principled way. By design, it produces smooth attention maps that resemble the ground truth instance labels. We conducted an exhaustive experimental validation with three real-world MIL datasets and up to eight state-of-the-art methods in both classification and localization tasks. This study showed that our method provides the best performance in localization while being highly competitive (best or second best) at classification.

Despite its advantages, our method has some limitations. The first is that, as with every other operator in GNNs, the computational costs of the smooth operator scale with the size of the bag. Fortunately, it can be paired with existing subgraph sampling techniques to mitigate this problem. The second limitation is that we do not have a definite answer for where it is better to use the proposed operator. We have shown that it leads to improvements in almost every place, but the optimal location may be problem-dependent and has to be tailored by the practitioner.

Finally, we hope that our work will draw more attention to the localization problem, which is very important for the deployment of computer-aided systems in the real world. In this sense, safely deploying the proposed methods in clinical practice requires evaluating them in a wider range of medical problems and quantifying their uncertainty. For the latter, we believe that the smooth operator could also benefit from a probabilistic formulation.

References

Ravi Aggarwal, Viknesh Sounderajah, Guy Martin, Daniel SW Ting, Alan Karthikesalingam, Dominic King, Hutan Ashrafian, and Ara Darzi. Diagnostic accuracy of deep learning in medical imaging: a systematic review and meta-analysis. NPJ digital medicine , 4(1):65, 2021.

Babak Ehteshami Bejnordi, Mitko Veta, Paul Johannes Van Diest, Bram Van Ginneken, Nico Karssemeijer, Geert Litjens, Jeroen AWM Van Der Laak, Meyke Hermsen, Quirine F Manson, Maschenka Balkenhol, et al. Diagnostic assessment of deep learning algorithms for detection of lymph node metastases in women with breast cancer. Jama , 318(22):2199-2210, 2017.

Mikhail Belkin and Partha Niyogi. Semi-supervised learning on manifolds. Machine Learning Journal , 1, 2002.

Christopher Michael Bishop and Hugh Bishop. Deep Learning - Foundations and Concepts . 2023.

Wouter Bulten, Kimmo Kartasalo, Po-Hsuan Cameron Chen, Peter Ström, Hans Pinckaers, Kunal Nagpal, Yuannan Cai, David F Steiner, Hester Van Boven, Robert Vink, et al. Artificial intelligence for diagnosis and gleason grading of prostate cancer: the panda challenge. Nature medicine , 28(1):154-163, 2022.

Chen Cai and Yusu Wang. A note on over-smoothing for graph neural networks. arXiv preprint arXiv:2006.13318 , 2020.

Gabriele Campanella, Matthew G Hanna, Luke Geneslaw, Allen Miraflor, Vitor Werneck Krauss Silva, Klaus J Busam, Edi Brogi, Victor E Reuter, David S Klimstra, and Thomas J Fuchs. Clinical-grade computational pathology using weakly supervised deep learning on whole slide images. Nature medicine , 25(8):1301-1309, 2019.

Marc-André Carbonneau, Veronika Cheplygina, Eric Granger, and Ghyslain Gagnon. Multiple instance learning: A survey of problem characteristics and applications. Pattern Recognition , 77:329-353, 2018.

Josef Cersovsky, Sadegh Mohammadi, Dagmar Kainmueller, and Johannes Hoehne. Towards hierarchical regional transformer-based multiple instance learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 3952-3960, 2023.

Olivier Chapelle, Jason Weston, and Bernhard Schölkopf. Cluster kernels for semi-supervised learning. Advances in neural information processing systems , 15, 2002.

Richard J Chen, Ming Y Lu, Muhammad Shaban, Chengkuan Chen, Tiffany Y Chen, Drew FK Williamson, and Faisal Mahmood. Whole slide images are 2d point clouds: Context-aware survival prediction using patch-based graph convolutional networks. In Medical Image Computing and Computer Assisted Intervention-MICCAI 2021: 24th International Conference, Strasbourg, France, September 27-October 1, 2021, Proceedings, Part VIII 24 , pages 339-349. Springer, 2021.

Thomas G Dietterich, Richard H Lathrop, and Tomás Lozano-Pérez. Solving the multiple instance problem with axis-parallel rectangles. Artificial intelligence , 89(1-2):31-71, 1997.

Adam E Flanders, Luciano M Prevedello, George Shih, Safwan S Halabi, Jayashree KalpathyCramer, Robyn Ball, John T Mongan, Anouk Stein, Felipe C Kitamura, Matthew P Lungren, et al. Construction of a machine learning dataset through collaboration: the rsna 2019 brain ct hemorrhage challenge. Radiology: Artificial Intelligence , 2(3):e190211, 2020.

Olga Fourkioti, Matt De Vries, and Chris Bakal. CAMIL: Context-aware multiple instance learning for cancer detection and subtyping in whole slide images. In The Twelfth International Conference on Learning Representations , 2024. URL https://openreview.net/forum? id=rzBskAEmoc .

Guoji Fu, Peilin Zhao, and Yatao Bian. p -laplacian based graph neural networks. In International Conference on Machine Learning , pages 6878-6917. PMLR, 2022.

Johannes Gasteiger, Aleksandar Bojchevski, and Stephan Günnemann. Predict then propagate: Graph neural networks meet personalized pagerank. arXiv preprint arXiv:1810.05997 , 2018.

Maximilian Ilse, Jakub Tomczak, and Max Welling. Attention-based deep multiple instance learning. In International conference on machine learning , pages 2127-2136. PMLR, 2018.

Mingu Kang, Heon Song, Seonwook Park, Donggeun Yoo, and Sérgio Pereira. Benchmarking self-supervised learning on diverse pathology datasets. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 3344-3354, 2023.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907 , 2016.

Bin Li, Yin Li, and Kevin W Eliceiri. Dual-stream multiple instance learning network for whole slide image classification with self-supervised contrastive learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 14318-14328, 2021.

Hang Li, Fan Yang, Yu Zhao, Xiaohan Xing, Jun Zhang, Mingxuan Gao, Junzhou Huang, Liansheng Wang, and Jianhua Yao. Dt-mil: deformable transformer for multi-instance learning on histopathological image. In Medical Image Computing and Computer Assisted InterventionMICCAI 2021: 24th International Conference, Strasbourg, France, September 27-October 1, 2021, Proceedings, Part VIII 24 , pages 206-216. Springer, 2021.

Qimai Li, Zhichao Han, and Xiao-Ming Wu. Deeper insights into graph convolutional networks for semi-supervised learning. In Proceedings of the AAAI conference on artificial intelligence , volume 32, 2018.

Ruoyu Li, Jiawen Yao, Xinliang Zhu, Yeqing Li, and Junzhou Huang. Graph cnn for survival analysis on whole slide pathological images. In International Conference on Medical Image Computing and Computer-Assisted Intervention , pages 174-182. Springer, 2018.

Ming Y Lu, Drew FK Williamson, Tiffany Y Chen, Richard J Chen, Matteo Barbieri, and Faisal Mahmood. Data-efficient and weakly supervised computational pathology on whole-slide images. Nature biomedical engineering , 5(6):555-570, 2021.

Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. arXiv preprint arXiv:1802.05957 , 2018.

Andrew Ng, Michael Jordan, and Yair Weiss. On spectral clustering: Analysis and an algorithm. Advances in neural information processing systems , 14, 2001.

Gwenolé Quellec, Guy Cazuguel, Béatrice Cochener, and Mathieu Lamard. Multiple-instance learning for medical image and video analysis. IEEE reviews in biomedical engineering , 10: 213-234, 2017.

Qin Ren, Yu Zhao, Bing He, Bingzhe Wu, Sijie Mai, Fan Xu, Yueshan Huang, Yonghong He, Junzhou Huang, and Jianhua Yao. Iib-mil: Integrated instance-level and bag-level multiple instances learning with label disambiguation for pathological image analysis. In International Conference on Medical Image Computing and Computer-Assisted Intervention , pages 560-569. Springer, 2023.

Brian D Ripley. Spatial statistics. Wiley Series in Probability and Statistics , 1981.

Matthias Seeger. Learning with labeled and unlabeled data. 2000.

Zhuchen Shao, Hao Bian, Yang Chen, Yifeng Wang, Jian Zhang, Xiangyang Ji, et al. Transmil: Transformer based correlated multiple instance learning for whole slide image classification. Advances in neural information processing systems , 34:2136-2147, 2021.

Julio Silva-Rodriguez, Adrián Colomer, Jose Dolz, and Valery Naranjo. Self-learning for weakly supervised gleason grading of local patterns. IEEE journal of biomedical and health informatics , 25(8):3094-3104, 2021.

Andrew H Song, Guillaume Jaume, Drew FK Williamson, Ming Y Lu, Anurag Vaidya, Tiffany R Miller, and Faisal Mahmood. Artificial intelligence for digital and computational pathology. Nature Reviews Bioengineering , 1(12):930-949, 2023.

Andrew H. Song, Mane Williams, Drew F. K. Williamson, Sarah S. L. Chow, Guillaume Jaume, Gan Gao, Andrew Zhang, Bowen Chen, Alexander S. Baras, Robert Serafin, Richard Colling, Michelle R. Downes, Xavier Farré, Peter Humphrey, Clare Verrill, Lawrence D. True, Anil V. Parwani, Jonathan T. C. Liu, and Faisal Mahmood. Analysis of 3D pathology samples using weakly supervised AI. Cell , 187(10):2502-2520.e17, May 2024. ISSN 1097-4172. doi: 10.1016/j.cell.2024.03.035.

Yunan Wu, Arne Schmidt, Enrique Hernández-Sánchez, Rafael Molina, and Aggelos K Katsaggelos. Combining attention-based multiple instance learning and gaussian processes for ct hemorrhage detection. In International Conference on Medical Image Computing and Computer-Assisted Intervention , pages 582-591. Springer, 2021.

Conghao Xiong, Hao Chen, Joseph JY Sung, and Irwin King. Diagnose like a pathologist: Transformer-enabled hierarchical attention-guided multiple instance learning for whole slide image classification. arXiv preprint arXiv:2301.08125 , 2023.

Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Selfsupervised learning via redundancy reduction. In International conference on machine learning , pages 12310-12320. PMLR, 2021.

Hongrun Zhang, Yanda Meng, Yitian Zhao, Yihong Qiao, Xiaoyun Yang, Sarah E Coupland, and Yalin Zheng. Dtfd-mil: Double-tier feature distillation multiple instance learning for histopathology whole slide image classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 18802-18812, 2022.

Yu Zhao, Zhenyu Lin, Kai Sun, Yidan Zhang, Junzhou Huang, Liansheng Wang, and Jianhua Yao. Setmil: spatial encoding transformer-based multiple instance learning for pathological image analysis. In International Conference on Medical Image Computing and ComputerAssisted Intervention , pages 66-76. Springer, 2022.

Yi Zheng, Rushin H Gindra, Emily J Green, Eric J Burks, Margrit Betke, Jennifer E Beane, and Vijaya B Kolachalama. A graph-transformer for whole slide image classification. IEEE transactions on medical imaging , 41(11):3003-3015, 2022.

Dengyong Zhou and Bernhard Schölkopf. Regularization on discrete spaces. In Joint Pattern Recognition Symposium , pages 361-368. Springer, 2005.

Dengyong Zhou, Olivier Bousquet, Thomas Lal, Jason Weston, and Bernhard Schölkopf. Learning with local and global consistency. Advances in neural information processing systems , 16, 2003.

S Kevin Zhou, Hayit Greenspan, Christos Davatzikos, James S Duncan, Bram Van Ginneken, Anant Madabhushi, Jerry L Prince, Daniel Rueckert, and Ronald M Summers. A review of deep learning in medical imaging: Imaging traits, technology trends, case studies with progress highlights, and future promises. Proceedings of the IEEE , 109(5):820-838, 2021.

A Proofs

A.1 Proof of Eq. 4

We present a general result for an arbitrary multilayer perceptron with Lipschitz activation functions. Note that assuming Lipschitzness is not a restriction, since most of the currently used activation functions meet this property [6]. The desired Eq. 4 is a particular case of this result. Let L ∈ N . Consider a L -layer perceptron that, given Y ∈ R N × D 0 , outputs Y L ∈ R N × D L defined by the following rule

where W ℓ ∈ R D ℓ × D ℓ +1 , and B ℓ = [ b ℓ , . . . , b ℓ ] ⊤ ∈ R N × D ℓ +1 where b ℓ ∈ R D ℓ +1 are trainable weights, and φ ℓ : R → R are activation functions applied element-wise. We suppose that each activation function φ ℓ is K ℓ -Lipschitz. Then, we obtain the following inequality,

Before verifying it, we note that by applying this inequality to every layer, we arrive at

To verify Eq. 16, we write Y ℓ +1 = [ y ℓ +1 1 , . . . , y ℓ +1 N ] ⊤ and Y ℓ = [ y ℓ 1 , . . . , y ℓ N ] ⊤ . We have, where ∥ W ℓ :0 ∥ 2 = ∏ ℓ j =0 ∥ W j ∥ 2 and K ℓ :0 = ∏ ℓ j =0 K ℓ . Taking L = 2 , D 0 = D , D 1 = L , D 2 = 1 , b 0 = b 1 = 0 , φ 0 = tanh , and φ 1 = Id , we recover Eq. 4.

where from Eq. 19 to Eq. 20 we have used the definition of Lipschitz function and from Eq. 20 to Eq. 21 we have used the consistency between the spectral and Euclidean norms.

A.2 Proof of Eq. 9

In this section, we adapt the proof presented in [6] for a similar result. Let U ∈ R N × D . Our goal is to show that where γ > 0 and λ ∗ γ = max { (1 + γλ n ) -2 : λ n ∈ Λ \ { 0 } } , being Λ = { λ 1 , . . . , λ N } the eigenvalues of the symmetric graph Laplacian matrix L . First, we reduce the proof to univariate graph functions by looking at the rows of U as univariate graph functions. Denoting them as { u 1 , . . . , u D } , where each u d ∈ R N , we have E D ( ( I + γ L ) -1 U ) = ∑ D d =1 E D ( ( I + γ L ) -1 u d ) . Therefore, it will be sufficient to show that, for any u ∈ R N ,

Next, it is useful to note that if λ n is an eigenvalue of L with associated eigenvector v n , then (1 + γλ n ) -1 is an eigenvalue of ( I + γ L ) -1 with the same associated eigenvector. Finally, let { v 1 , . . . , v N } be a orthonormal eigenbasis of L , being each v n associated to the eigenvalue λ n . This basis always exists since L is a symmetric matrix. Writing u = ∑ N n =1 c n v n , with c n ∈ R , we have

Using that the eigenvectors are orthogonal to each other, we arrive at

B Experiments: details and further results

In this section, we provide the details of the datasets, architectures, and configurations used for each experiment. The code is available at https://github.com/Franblueee/SmMIL .

B.1 Datasets

We provide insights into the datasets we have used: a description of the problem, the train/test splits, and preprocessing (instance selection and feature extraction). For all datasets, we obtain an initial train/test partition. Then, we split the initial train partition into five different train/validation splits. Every model is trained on each of these splits and then evaluated on the test set. We report the average performance on this test set.

RSNA. It was published by the Radiological Society of North America (RSNA) to detect acute intracranial hemorrhage and its subtypes [13]. It is available in Kaggle 1 . We use the official train-test split. It includes a total of 1150 scans. There are a total amount of 39750 slices and the number in each scan varies from 24 to 57. Each slice is preprocessed following [36].

PANDA. It is a public dataset for the classification of the severity of prostate cancer from microscopy scans of prostate biopsy samples [5]. It is available in Kaggle 2 . Since the official test set is not publicly available, we use the train/test split proposed in [33]. To extract the patches from each WSI, we follow the procedure described in [33], obtaining patches of size 512 × 512 at 10 × magnification. This results in a total amount of 10503 WSIs and 1107931 patches.

CAMELYON16. It is a public dataset for the detection of breast cancer metastasis [2]. It is available at the Registry of Open Data of AWS 3 . The official repository contains 400 WSIs in total, including 270 for training and 130 for testing. From each WSI, we extract patches of size 512 × 512 at 20 × magnification using the method proposed by Lu et al. [25].

B.2 Model and training configuration

Weprovide details about how we have implemented the proposed methods and how we have conducted the experiments.

Feature extractor. Due to the limited memory of the GPU, it is necessary to extract features from each instance. Otherwise, the bags will not fit in memory. In this work, we consider three options for the feature extractor, all of which are pre-trained in Imagenet: ResNet18 ( P = 512 ), ResNet50 ( P = 2048 ), and ViT-B-32 ( P = 768 ). In addition, for CAMELYON16 we also consider ResNet50BT 4 ( P = 2048 ), which is a ResNet50 model pre-trained using the Barlow Twins Self-Supervised Learning method on a huge dataset of WSIs patches [38, 18]. The results reported in the main text correspond to ResNet18 for RSNA and PANDA, and to ResNet50-BT for CAMELYON16. We study how the choice of the feature extractor affects the results in Sec. B.3.

0 1

5

10

20

∞

T

0

.

76

0

.

78

0

.

80

0

.

82

0

.

84

Inst. AUROC

early

mid

late

0 1

5

10

20

∞

T

0

.

85

0

.

86

0

.

87

0

.

88

0

.

89

0

.

90

Bag AUROC

(a) RSNA.

0 1

5

10

20

T

0

.

7

0

.

8

0

.

9

1

.

0

Inst. AUROC

0 1

5

10

20

T

0

.

90

0

.

92

0

.

94

0

.

96

0

.

98

1

.

00

Bag AUROC

(b) CAMELYON16.

Model architecture. We describe the architecture we have used for the proposed methods ( Sm AP and Sm TAP). For the rest of the methods considered, we adopt their original implementations and default configurations, publicly available on their GitHub repositories. For the independent instance encoding part (see Fig. 3a and Fig. 3b), the instance embeddings h n are obtained using one fully connected layer with 512 units ( D = 512 ). For the attention-based pooling described by Eq. 1 and Eq. 2, we fix D = 512 and L = 100 . The transformer encoder in Fig. 3c is implemented using two transformer layers. These layers use the standard multi-head attention mechanism equipped with skip connections and layer normalization [4]. We fix the key, query, and value dimensions to 128 and the number of heads to 8. We used the Pytorch's implementation of dot product attention 5 . Finally, the bag-embedding classifier was implemented using one fully connected layer.

Training setup and hyperparameters. To ensure fair and reproducible conditions, we trained every method under the same setup. The number of epochs was set to 50 . We adopt the Adam optimizer [19] with the default Pytorch configuration. For the base learning rate, we considered two different values, 10 -4 and 10 -5 , since we noticed that models that do not use transformers obtained better results when the learning rate was higher. We report the best results for each model. We adopted a slow start using Pytorch's LinearLR scheduler with start_factor=0.1 and total_iters=5 . During training, we monitored the bag AUROC and cross-entropy loss in the validation set and kept the weights that obtained the best results. The batch size was set to 32 in RSNA and PANDA. In CAMELYON16, it was set to 4 for no-transformer methods, and to 1 for transformer-based methods. However, for SETMIL, we had to set it to 1 in PANDA and CAMELYON16 due to its high GPU memory requirements. In RSNA we weighted the loss function to account for the imbalance between positive and negative bags since we observed it to improve the results. All the experiments were performed on one NVIDIA GeForce RTX 3090.

B.3 Further ablation studies.

We complete the ablation study presented in the main paper, Sec. 5.3, by looking at the rest of the design choices or hyperparameters associated with our Sm .

Smooth operator approximation. The exact form of Sm given by Eq. 7 becomes computationally infeasible for large bag sizes. The quality of the approximation, given by Eq. 8, is controlled by the number of steps T . Fig. 7 shows the results for different values of this hyperparameter in RSNA and CAMELYON16. In RSNA, since the bags are smaller, we can compute the closed-form solution, which we represent as T = ∞ . Almost any choice of T > 0 improves upon ABMIL. This improvement is particularly noticeable in CAMELYON16. Moreover, in most cases, the performance stabilizes at T = 10 , which is the value we used in our experiments.

Sm on top of other models. We have proposed two new models ( Sm AP and Sm TAP) by applying Sm on top of two baselines (ABMIL and Transformer+ABMIL, respectively). Instead, the Sm can be applied on top of other existing approaches. In Table 4 we explore how other approaches behave when combined with the proposed Sm in the CAMELYON16 dataset. Instance-level performance is enhanced (greatly in some cases, e.g. an increase from 0.76 to 0.96 in AUROC for DSMIL), whereas bag-level results are competitive. The decrease in bag-level results for DFTD-MIL is explained by

the fact that this method randomly splits each bag into different chunks. This may lead to the loss of local interactions exploited by Sm (e.g. if two adjacent instances end in different chunks).

An alternative smoothing strategy. Introducing a penalty term in the loss function to favor smoothness is a natural alternative to the proposed operator. However, there is an important difference: the use of a penalty term does not modify the model architecture. The penalty term favors that the learned weights encode such a property, but it is not explicitly encoded in the model. For instance, note that the penalty term is not used at inference time. We compare the penalty-based approach and the proposed Sm in Table 5. Although differences are not large, Sm obtains superior performance.

Feature extractor. We investigate whether the choice of the feature extractor influences the results and conclusions presented in the main text. We have evaluated each of the considered methods in each dataset using the feature extractors mentioned above (ResNet18, ResNet50, ViT-B-32, and ResNet50-BT). The results are shown in Tables 7-10. We summarize them in Table 6, where we collect the average instance and bag rank of each method for each feature extractor. We observe that the proposed smooth operator Sm obtains in almost all cases the highest rank. This supports the idea that the improvement introduced by Sm does not depend on the used features.

C Additional figures

H

f

f

h

1

h

2

.

.

.

h

N

f

1

f

2

.

.

.

f

N

f

1

f

2

.

.

.

f

N

H

/latticetop

Softmax

(

f

)

z

w

/latticetop

W

·

)

w

/latticetop

W

·

)

.

.

.

w

/latticetop

W

·

)

(a) ABMIL.

H

f

f

h

1

h

2

.

.

.

h

N

f

1

f

2

.

.

.

f

N

f

1

f

2

.

.

.

f

N

H

/latticetop

Softmax

(

f

)

z

Sm

w

/latticetop

W

·

)

w

/latticetop

W

·

)

.

.

.

w

/latticetop

W

·

)

(b)

Sm

AP-late.

H

f

f

h

1

h

2

.

.

.

h

N

f

1

f

2

.

.

.

f

N

f

1

f

2

.

.

.

f

N

H

/latticetop

Softmax

(

f

)

z

Sm

w

/latticetop

W

·

)

w

/latticetop

W

·

)

.

.

.

w

/latticetop

W

·

)

(c)

Sm

AP-mid.

H

̂

H

=

Sm

(

H

)

f

f

h

1

h

2

.

.

.

h

N

̂

h

1

̂

h

2

.

.

.

̂

h

N

f

1

f

2

.

.

.

f

N

f

1

f

2

.

.

.

f

N

̂

H

/latticetop

Softmax

(

f

)

z

Sm

w

/latticetop

W

·

)

w

/latticetop

W

·

)

.

.

.

w

/latticetop

W

·

)

(d)

Sm

AP-early.

CT scan

Slice labels

Sm AP

ABMIL

CLAM

DSMIL

DFTD-MIL

Sm TAP

TransMIL

SETMIL

GTP

N . 000000000000000ccc8&&5$ g

0000000000000066884668窖

000000000000000668886移9窖

0000000000000066888269窖

®

CAMIL

0

1

Attention score

WSI

Patch labels

Sm

TAP

TransMIL

SETMIL

GTP

CAMIL

Sm

AP

ABMIL

CLAM

DSMIL

DFTD-MIL

0

1

Attention score

WSI

Patch labels

Sm

TAP

TransMIL

SETMIL

GTP

CAMIL

Sm

AP

ABMIL

CLAM

DSMIL

DFTD-MIL

Groung truth

ABMIL

α

= 0

.

1

)

α

= 0

.

5

)

α

= 0

.

9

)

-

3

-

2

-

1

0

1

2

Attention value

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

0

.

00

0

.

02

0

.

04

0

.

06

Attention value

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

-

2

-

1

0

1

2

Attention value

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

0

.

0000

0

.

0001

0

.

0002

0

.

0003

Attention value

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

-

4

-

2

0

2

Attention value

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

Sm

TAP

TransMIL

SETMIL

GTP

CAMIL

-

4

-

2

0

2

Attention value

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

Positive instances

Negative instances

-

4

-

2

0

2

Attention value

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

-

4

-

2

0

2

4

Attention value

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

-

2

.

5

-

2

.

0

-

1

.

5

-

1

.

0

-

0

.

5

Attention value

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

-

2

0

2

4

Attention value

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

Sm

AP

ABMIL

CLAM

DSMIL

DFTD-MIL

-

2

0

2

Attention value

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

0

.

000

0

.

025

0

.

050

0

.

075

0

.

100

0

.

125

Attention value

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

-

4

-

2

0

2

4

Attention value

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

0

.

00

0

.

01

0

.

02

0

.

03

0

.

04

0

.

05

Attention value

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

-

2

0

2

Attention value

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

Sm

TAP

TransMIL

SETMIL

GTP

CAMIL

-

4

-

2

0

2

4

Attention value

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

Positive instances

Negative instances

-

4

-

2

0

2

4

Attention value

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

0

5

10

Attention value

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

-

5

0

5

Attention value

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

-

5

0

5

10

15

Attention value

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

Sm

AP

ABMIL

CLAM

DSMIL

DFTD-MIL

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Our claims are supported by Sec. 3 (unified view of deep MIL methods), by Sec. 4 (derivation of the proposed smooth operator Sm ), and by Sec. 5 (experimental results).

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss the limitations of the proposed method in Sec. 6.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: The full set of assumptions is included along with the proof of each result, as well as in the main text, see Appendix A and Sec. 4.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We describe the proposed method in Sec. 4.3. We provide the details about the datasets and experimental configuration we used in Appendix B. The code associated with this paper has been uploaded as supplementary material to OpenReview, and will be made public on GitHub upon the acceptance of the paper. Also, the datasets are available publicly on Kaggle (RSNA and PANDA) and the Registry of Open Data on AWS (CAMELYON16).

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The code associated with this paper has been uploaded as supplementary material to OpenReview, and will be made public on GitHub upon the acceptance of the paper. Also, the datasets are available publicly on Kaggle (RSNA and PANDA) and the Registry of Open Data on AWS (CAMELYON16).

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We provide the details about the training configuration we have used in Appendix B.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: Our results are presented as the mean and standard deviation obtained in five independent runs, see Sec. 5 and Sec. B.3. Due to space reasons, for Table 3 we only provide the mean values.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: This information can be found along with the rest of the experimental configuration information, see Appendix B.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Our research completely conforms with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: As pointed out in Sec. 1 and in Sec. 6, one of the main goals of this work is to draw attention to the localization task in MIL, which can contribute favorably to the deployment of computer-aided systems in the real world. However, as we also indicate in Sec. 6, for this to happen safely (e.g., avoiding misdiagnosis), further investigation is needed about equipping them with uncertainty estimation mechanisms.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We cite the original works where the methods were proposed. We used the code provided by the corresponding authors. Each paper includes the URL to its GitHub repository, and each implementation is released under (possibly) a different license, specified in the mentioned repository.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: Our code has been uploaded as supplementary material, along with instructions to preprocess each dataset and replicate the results of the experiments.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: Our paper does not involve crowdsourcing or research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: Our paper does not involve crowdsourcing or research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper is well-written and proposes a new modular mechanism that can (selectively) combine local and global interactions in a model. The authors discuss the theory behind their proposed smoothness operator and pooling mechanism and provide detailed proof for their claims. They have also evaluated their model against other methods on three different datasets (1 CT and 2 H&E) with reporting AUC and F1 score as their evaluation metrics. Their method is superior to other methods and the proper ablation study supports their claims.\n\nSTRENGTHS:\nThe writing is concise and the contributions are clear. The author proposes a new smoothness mechanism, generally an interesting idea, to integrate local and global information. Besides the quantitative experiments, there are some qualitative experiments such as attention maps that show the model's relative importance on different regions in a slide.\n\nWEAKNESSES:\nThere are three major concerns with this work:\n1. The choice of encoder. The authors have used Resnet18 for RSNA and Camelyon16 and Resnet50 for PANDA. I could not find any justification for why different encoders have been used! Also, with the trend toward foundation models, transformer-based backbones are now of interest to the community. Therefore, for the sake of consistency, the author should use the same encoder for different datasets or report both resent50 and resnet18. And, for the sake of the generality of their work, they should add one encoder such as ViT or Swin as well (w/ ImageNet weights) to support the generality of their claims.\n\n2. The body of research has been founded on local-to-global interactions. There are quite a few standard graph-based methodologies in the literature that the authors need to compare their work against. Currently, there are no representatives from those families of the MIL method in the benchmark. Two examples of such methods are (1) and (2).\n\n3. There is a family of MIL methods in the literature that try to pseudo-label the instances during training, which is essentially equivalent to localization in this work. For instance, two of the most recent such methods are (3) and (4). The author should compare their method against these as they are essentially from the same family.\n\n(1) Chen, Richard J., et al. \"Whole slide images are 2d point clouds: Context-aware survival prediction using patch-based graph convolutional networks.\" Medical Image Computing and Computer Assisted Intervention–MICCAI 2021: 24th International Conference, Strasbourg, France, September 27–October 1, 2021, Proceedings, Part VIII 24. Springer International Publishing, 2021.\n\n(2) Li, R., Yao, J., Zhu, X., Li, Y., Huang, J. (2018). Graph CNN for Survival Analysis on Whole Slide Pathological Images. In: Frangi, A., Schnabel, J., Davatzikos, C., Alberola-López, C., Fichtinger, G. (eds) Medical Image Computing and Computer Assisted Intervention – MICCAI 2018. MICCAI 2018. Lecture Notes in Computer Science(), vol 11071. Springer, Cham. https://doi.org/10.1007/978-3-030-00934-2_20\n\n(3) Z. Shao et al., \"LNPL-MIL: Learning from Noisy Pseudo Labels for Promoting Multiple Instance Learning in Whole Slide Image,\" 2023 IEEE/CVF International Conference on Computer Vision (ICCV), Paris, France, 2023, pp. 21438-21438, doi: 10.1109/ICCV51070.2023.01965.\n\n(4) Ren, Q. et al. (2023). IIB-MIL: Integrated Instance-Level and Bag-Level Multiple Instances Learning with Label Disambiguation for Pathological Image Analysis. In: Greenspan, H., et al. Medical Image Computing and Computer Assisted Intervention – MICCAI 2023. MICCAI 2023. Lecture Notes in Computer Science, vol 14225. Springer, Cham. https://doi.org/10.1007/978-3-031-43987-2_54\n\nQUESTIONS:\nI am curious if the author expects the same performance boost obtained on top of resent, to be obtained on top of a histopathology-specific feature extractor.\n\nLIMITATIONS:\nIt has been justified :)", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposed a multi-instance learning approach. The basic idea is to make use of the spatial dependency between training samples. A smoothing operator was proposed to regularize the attention matrix with respect to inter-sample similarity (to my understanding), which the authors claimed to improved both quality of localization and classification tasks.\n\nSTRENGTHS:\nThe idea seems unique and based, but more justifications are merited.\n\nWEAKNESSES:\n1. Although the smoothness property do exist in multi-instance learning as the authors claimed and may introduce useful information. It is still unclear why imposing it in the neural network helps with classification. It doesn't seem to serve as an induction bias from its looks.\n2. Actually only 3 dataset are tested (though each with different variants), and the results are not so persuasive. Especially for the classification tasks that the authors want to credit, they failed to get 1st place in 2/3 of the AUC or F1 scores.\n\nQUESTIONS:\n1. The overall principle of the method need rephasing, i.e., how introduction the SM transformation (instead of penalty) improves classification? \n2. The smoothness assumption seems to be a property connected with localization. However, in 4.1-4.3 and Fig.3 the authors mainly present the SM operator for  classification. Yet in the experiment part, it seem that adding SM had more advantages in localization (but not sure how is it achieved?). The authors need to clarify the relationship between the two tasks.\n3. In Tab. 1, why the performance of the 'with global interactions' variants worse than the 'without' one in localization, but it is opposite in the classification (Tab. 2)?\n\nLIMITATIONS:\nThe method required strict ordering (or spatial information) or 3d-to-2d samples, which is quite a strong demand but it is not discussed in the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a technique to improve the localization capabilities of the current MIL, especially for CAD models that perform CT and WSI analysis. The method is based on seeing the attention attributed to each patch as a graph and minimizing its Dirichlet energy, promoting smooth transitions on the attention values of neighbor patches, which is in accordance with the assumption that neighboring patches share the same labels.\n\nSTRENGTHS:\nThe work is well-developed and well written, with a methodology explained through concise equations and theorems that guide the reader through the development of the idea. The results validate what is proposed, but minor adjustments and additional information will make the work even more valuable.\n\nWEAKNESSES:\nThe method introduces the smooth operator on top of the existing ABMIL model. The proposed approach improves the performance of ABMIL on the CAMELYON dataset, both on localization and classification. Nevertheless, when analyzing the attention maps provided in the appendix, the differences between ABMIL and smAP are minimal. Although we have a numerical evaluation of the Dirichlet energy, we cannot tell if the drop in energy is significant only by these numbers, so this visual empiric evaluation is important for a better insight regarding the developed work. \n\nObserving the appendix figures, ABMIL already performs well in matching its attention to the tiles annotations. I suggest the authors apply the smooth operator on top of other models that did not perform well on CAMELYON for this task, such as CLAM, CAMIL, and SETMIL, to evaluate how minimizing the Dirichlet energy of the attention improves the localization task quantitatively and qualitatively.\n\nI suggest the authors perform an ablation study to evaluate how minimizing the Dirichlet energy compares to other smoothing strategies. For instance, summing the variance of the attentions to the model’s loss may provide similar results as minimizing the Dirichlet energy. \n\nThe analysis of the alpha parameter is great, but again, a qualitative approach is valuable for this case. Seeing how the attention map changes as the parameter alpha increases is highly valuable for the reader.\n\nLines 127 to 131: trying to debate equations not present in the paper is not good for the reader. I suggest writing the debated equations in the article or explaining the arguments without mentioning them.\n\nQUESTIONS:\nThe formulation seems to assume bags of equal size, which is not the case in MIL (in general). \n\nThe proposed method (loss) may have the drawback of over-smoothing the transition between tiles, making the localization regions blurry. Is this true? Could this be overcome?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}