Sm: enhanced localization in Multiple Instance Learning for medical imaging classification

Authors: francisco m castro-macías
pablo  morales-álvarez
yunan  wu
rafael  molina
aggelos k katsaggelos
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-09 23:14:37

Abstract: Multiple Instance Learning (MIL) is widely used in medical imaging classification to reduce the labeling effort. 
While only bag labels are available for training, one typically seeks predictions at both bag and instance levels (classification and localization tasks, respectively). Early MIL methods treated the instances in a bag independently. Recent methods account for global and local dependencies among instances. Although they have yielded excellent results in classification, their performance in terms of localization is comparatively limited. We argue that these models have been designed to target the classification task, while implications at the instance level have not been deeply investigated. Motivated by a simple observation -- that neighboring instances are likely to have the same label -- we propose a novel, principled, and flexible mechanism to model local dependencies. It can be used alone or combined with any mechanism to model global dependencies (e.g., transformers). A thorough empirical validation shows that our module leads to state-of-the-art performance in localization while being competitive or superior in classification. Our code is at https://github.com/Franblueee/SmMIL.

Main paper: 
 introduction: 
 Over the last decades, medical imaging classification has benefited from advances in deep learning [35, 44] . However, the performance of these methods drops when the number of labeled samples is low, which is common in real-world medical scenarios [1] . To overcome this, Multiple Instance Learning (MIL) has emerged as a popular weakly supervised approach [14, 8, 12] .In MIL, instances are arranged in bags. At train time, a label is available for the entire bag, while the instance labels remain unknown. The goal is to train a method that, given a test bag, can predict both at bag and instance levels (classification and localization tasks, respectively). This paradigm is well suited to the medical imaging domain [28] . In cancer detection from Whole Slide Images (WSIs), the WSI represents the bag, and the patches are the instances. In intracranial hemorrhage detection from Computerized Tomographic (CT) scans, the full scan represents the bag, and the slices at different heights are the instances. In these scenarios, making accurate predictions of instance labels is extremely important from a clinical viewpoint, as it translates into pinpointing the location of the lesion [7] .Most successful approaches in MIL build on the attention-based pooling [17] , a permutation-invariant operator that assigns an attention value to each instance independently. This method has been extended in different ways while maintaining the permutation-invariant property [21, 25, 39] . The aforementioned works pose a problem: the dependencies between the instances, which are important when making a diagnosis, are ignored. To account for this, TransMIL [32] proposed to model global dependencies using a transformer encoder. The idea is to use the self-attention mechanism to introduce interactions between each pair of instances. Based on it, other transformer-based approaches have emerged, also focusing on global dependencies [9, 22, 37] . More recently, several works have also incorporated natural local interactions, which are those between neighboring instances [14, 40, 41] .Although these methods accounting for dependencies have resulted in excellent performance at the bag level, the evaluation at the instance level has received less attention and the results are not comparatively good so far, see the very recent [14] . In this work, we argue that recent MIL methods have been designed with the classification task in mind, and we propose a new model that focuses on both the classification and localization tasks. Specifically, we propose a novel and theoretically grounded mechanism to introduce local dependencies, hereafter referred to as the smooth operator Sm. This is a flexible module that can be used alone on top of classical MIL approaches, or in combination with transformers to also account for global dependencies. In both cases, we show that the proposed operator achieves state-of-the-art localization results while being competitive in classification. We compare against a total amount of eight methods, including very recent ones [14, 40] . We utilize three different datasets of different nature and size, covering two different medical imaging problems (cancer detection in WSI images and hemorrhage detection in CT scans).Our main contributions are: (i) we provide a unified view of current deep MIL approaches; (ii) we propose a principled mechanism to introduce local interactions, which is a modular component that can be combined or not with global interactions; and (iii) we evaluate up to eight state-of-the-art MIL methods on three real-world MIL datasets in both classification and localization tasks, showing that the proposed method stands out in localization while being competitive or superior in classification. 
related work: 
 In this work, we tackle the localization task in deep MIL methods using existing concepts and techniques from deep MIL and Graph Neural Networks (GNNs) theory.Deep Multiple Instance Learning. As explained by Song et al. [34] , deep MIL methods can be divided into two broad categories, namely instance-based or embedding-based, depending on the level at which a specific aggregation operator is applied. In this paper, we focus on the embedding-based category, and in particular attention-based ones.Ilse et al. [17] proposed attention-based pooling to weigh each instance in the bag. To improve it, different modifications were proposed, including the use of clustering layers [25] , grouping the instances in pseudo-bags [39] , and using similarity measures to critical instances to compute the attention values [21] . However, these methods ignore the existing dependencies between the instances in a bag. To address this, Shao et al. [32] proposed to use a transformer-based architecture and the PPEG position encoding module. This has been extended with different transformer variations, including the deformable transformer architecture [22] , hierarchical attention [37] , and regional sampling [9] . Recently, these methods have been improved to include spatial information in different ways, including the use of a Graph Convolutional Network (GCN) before the transformer [41] , a neighbor-constrained attention module [14] , and a spatial-encoding transformer architecture [40] .In the studies mentioned above, the objective is to obtain increasingly better bag-level results, while the evaluation at the instance level is usually performed qualitatively. In contrast, our work addresses both the instance localization task and the bag classification task, as both are of great importance for making a diagnosis. Moreover, our work is not limited to WSI classification; it is also valid for other medical imaging modalities.Graph Neural Networks. Our motivation -that neighboring instances are likely to have the same label -is a well-established assumption within the machine learning community, often referred to as the cluster assumption [10, 31] . Since leveraged in 1984 by Ripley [30] in the context of spatial statistics, it has been extensively used in spectral clustering [27] , semi-supervised learning on graphs [3] , and recently in GNNs [20] . Our work builds upon seminal works in these areas.The proposed smooth operator is derived considering a Dirichlet energy minimization problem, similar to the work by Zhou and Schölkopf [42] and Zhou et al. [43] . This approach has been employed in recent years to obtain new GNN models, including the p-Laplacian layer [15] , and PPNP layer [16] . Moreover, the Dirichlet energy has been studied in the context of GNNs to analyze the over-smoothing phenomenon [6, 23] . In this regard, our bound on the decrease of the Dirichlet energy is analogous to the result derived by Li et al. [23] to study over-smoothing for GCNs. Our result, however, holds for the proposed mechanism, of which the graph convolutional layer is a special type. 
background: a unified view of deep mil approaches: 
 We first describe the binary MIL problem tackled in this paper. Then, we provide a unified view of the most popular deep MIL methods. As explained in Sec. 2, we focus on embedding-based approaches.In MIL, the training set consists of pairs of the form (X, Y ), where X ∈ R N ×P is a bag of instances and Y ∈ {0, 1} is the bag label. We write X = [x 1 , . . . , x N ]T ∈ R N ×P , where x n ∈ R P are the instances. Each instance x n is associated to a label y n ∈ {0, 1}, not available during training. It is assumed that Y = max {y 1 , . . . , y N }, i.e., a bag X is considered positive if and only if there is at least one positive instance in the bag.Given a previously unseen bag (e.g., a medical image), the goal at test time is to: i) predict the bag label (classification task) and ii) obtain predictions or estimates for the instance labels (localization task). In general, deep MIL models output a bag-level prediction Ŷ , as well as instance-level scalars f n that are used for instance-level prediction. This general process is depicted in Fig. 1a . In many approaches, these f n are the so-called attention values (e.g., ABMIL [17] , TransMIL [32] , CAMIL [14] ), but they can be obtained in different ways (e.g., through GraphCAM in GTP [41] ). Within the general process in Fig. 1a , deep MIL models can be categorized into three families, depending on how instances interact with each other, see Fig. 1b , Fig. 1c , and Fig. 1d .In the first family, shown in Fig. 1b , the instances are encoded independently and then aggregated. The well-known ABMIL [17] fits in this paradigm. Subsequent works introduce slight modifications to ABMIL, while still encoding each instance independently [21, 25, 39] . ABMIL, on which we will rely to introduce our model, is depicted in Fig. 3a . First, a bag of embeddings H = [h 1 , . . . , h N ] ∈ R N ×D is obtained by applying a neural network independently to each instance. Then, the attention-based pooling computes the attention values f and the bag embedding z according toEQUATIONEQUATIONwhere W ∈ R L×D , w ∈ R L are trainable weights. Last, Ŷ is obtained by applying a linear classifier on z.The second family accounts for global interactions between instances, possibly long-range ones, see Fig. 1c . These works treat instances as tokens that interact through the self-attention mechanism. This way, global interactions between instances are learned. One of the most popular approaches in this family is TransMIL [32] , which was later extended in different directions [9, 22] . The third family complements the previous one with local interactions defined by a fixed neighborhood, see Fig. 1d . They differ in how local interactions are represented, e.g., as a graph in CAMIL [14] and GTP [41] , or using a position-encoded feature map in SETMIL [40] .In most of these works, the localization task is assessed qualitatively, e.g., by visually comparing the attention maps. This contrasts with the classification task, which is always evaluated quantitatively. As evidenced by Fourkioti et al. [14] , this has translated into comparatively poor performance in terms of localization. We notice that current models have been designed to target the classification task, and they excel at that. However, their model design is not as careful about the instance-level implications. For example, CAMIL [14] does not leverage any local information to obtain the instance-level attention values. Indeed, from their Eq. ( 8) one deduces that the a i values are obtained from the tile representations t i , which have not undergone any local interaction. Observe that local interactions take place in Eq. ( 4) and Eq. ( 5) in their paper, but these only affect the bag-level predictions, not the instance-level ones. Similarly, GTP [41] introduces local interactions through an off-the-shelf graph convolutional layer, the effect of which is not investigated at the instance level. In the following section, we propose a principled approach to account for meaningful local interactions based on the Dirichlet energy. The idea is motivated by a natural property often observed in the instance-level labels of medical images: the similarity between neighboring instances.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [Yes]• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
method: introducing smoothness in the attention values: 
 In medical imaging, instance labels are a priori expected to exhibit local dependencies with their neighboring instances: an instance is likely to be surrounded by instances with the same label, see Fig. 2 . Recall that attention values are commonly used as a proxy to estimate these labels, so they should inherit this property. Based on these observations, our intuitive idea is to favor a smoothness property on the attention values. To this end, Sec. 4.1 formalizes the notion of smoothness through the Dirichlet energy. Sec. 4.2 presents the proposed smoothing operator Sm, which encourages smoothness as well as fidelity to the original signal. Sec. 4.3 proposes how to leverage Sm in the context of MIL, both in combination with global interactions (via transformers), and without them. We will build on top of the well-known and simple ABMIL to isolate the effect of Sm and avoid over-sophisticated models.We represent each bag as a graph, where the nodes are the instances and the edges represent the spatial connectivity between instances. Formally, we suppose that each bag X ∈ R N ×D has been assigned an adjacency matrix A = [A ij ] ∈ R N ×N , defined by A ij > 0 if instances x i and x j are neighbors, and A ij = 0 otherwise. We assume that the adjacency matrix is symmetric, i.e. A ij = A ji .The Dirichlet energy is a well-known functional that measures the variability of a function defined on a graph [42, 43] . In our case, we think of this function as the attention values f ∈ R N , recall Fig. 1a . As we shall see below, it will be necessary to define the Dirichlet energy for multivariate graph functions. Given a multivariate graph function U = [u 1 , . . . , u N ] ⊤ ∈ R N ×D defined on the bag graph, the Dirichlet energy of U is given byEQUATIONwhere ∥•∥ 2 denotes the Euclidean norm, L is the graph Laplacian matrix L = D -A, D ∈ R N ×N is the degree matrix, D = Diag (D 1 , . . . , D N ), D n = i A ni . When D = 1 we obtain the definition for univariate graph functions, such as the attention values f .Bounding E D on the attention values. In most deep MIL approaches, the attention values f are obtained by applying a neural network to instance-level features. One example is ABMIL [17] , which uses a two-layer perceptron defined by Eq. 1. Noting that tanh is a Lipschitz function with Lipschitz constant equal to 1, we arrive at the following chain of inequalitiesEQUATIONwhere ∥•∥ 2 denotes the spectral norm. A more general result holds in the general case of an arbitrary multi-layer perceptron, see Appendix A for a proof. The above chain of inequalities tells us that if we want E D (f ) to be low, we can act on f itself or on previous layers (e.g., on F or on H), constraining the norm of the trainable weights to remain constant. This constraint can be achieved using spectral normalization [26] , and we study its influence in Sec. B.3. In the next subsection, we propose an operator that can be used on any of these levels (f , F, H) to reduce the Dirichlet energy of its output.Our goal now turns into finding an operator Sm : R N ×D → R N ×D that, given a bag graph multivariate function U ∈ R N ×D , returns another bag graph multivariate function Sm (U) ∈ R N ×D such that its Dirichlet energy is lower without losing the information present in the original U. Following seminal works [42, 43] , we cast this as an optimization problem,EQUATIONEQUATIONwhere α ∈ [0, 1) accounts for the trade off between both terms, and ∥•∥ F denotes the Frobenius norm. The first term in the above equation penalizes functions with too much variability, while the second term penalizes functions that differ too much from the original U. Note that this can be interpreted as a maximum a posteriori formulation, where the first term corresponds to the prior distribution and the second to the observation model, see [30] . The objective function E is strictly convex, and therefore admits a unique solution, given byEQUATIONwhere γ = α/(1α). Unfortunately, the expression in Eq. ( 7), although elegant, incurs prohibitive computational and memory costs, especially when the number of instances in the bag is large (which is the case of WSIs). Instead, we can take an iterative approach, defining Sm (U) = G(T ), withEQUATIONAs demonstrated by Zhou et al. [43] , the sequence {G(t)} converges to the optimal solution in Eq. 7. As studied by Gasteiger et al. [16] , it is enough to use a small number of iterations T to approximate the exact solution. Therefore, in this work, we will adopt the iterative approach described by Eq. 8. Based on previous work [16] , we will use T = 10, and α will be set as a trainable parameter initialized at α = 0.5. See Sec. B.3 for a study on the effects of these hyperparameters and Fig. 12 for a visual comparison of the effect that α has on the attention maps.Theoretical guarantees via the normalized Laplacian. We present a result that informs us about the rate at which the Dirichlet energy decreases when applying Sm. Let us defineλ * γ = max (1 + γλ n ) -2 : λ n ∈ Λ \ {0}, where Λ = {λ 1 , . . . , λ N } are the eigenvalues of the bag graph Laplacian matrix. Then, we have the following inequality,EQUATION)The proof is inspired by Cai and Wang [6] , see Appendix A. If λ * γ < 1, then the smooth operator effectively decreases the Dirichlet energy. If we replace the Laplacian matrix by the normalized Laplacian matrix, L = D -1/2 LD -1/2 , it is known that its eigenvalues lie in the interval [0, 2), and then λ * γ < 1 holds. This motivates the use of the normalized Laplacian in our experiments. The smooth operator Sm only introduces one parameter to be estimated, α. Also, it is differentiable with respect to its input. Therefore, it can be integrated into simple attention-based MIL models, such as ABMIL, to account for local dependencies. Here we propose how to leverage the operator Sm in the context of MIL. We build on top of the well-known ABMIL. First, we introduce SmAP, which integrates ABMIL with Sm and only accounts for local interactions. Second, we introduce SmTAP, which equips SmAP with a transformer encoder to account for global dependencies. The proposed models are depicted in Fig. 3b and Fig. 3c . The details about the architecture we have used can be found in Sec. B.2.SmAP: Smooth Attention Pooling. This is represented in Fig. 3b . First, the bag of embeddings H is obtained as in ABMIL [17] , i.e. treating the instances independently. Then, the operator Sm is integrated within the attention pooling. Based on Eq. 4, this can be done on the attention values themselves or on previous representations. We consider three different variants: SmAP-late, SmAP-mid, SmAP-early. They act, respectively, on f (the attention values themselves), on F (i.e. before entering the last layer), and on H (i.e. before entering the attention-based pooling). Formally,EQUATIONEQUATIONEQUATIONWhile SmAP-late and SmAP-mid act on the computation of the attention values, SmAP-early acts on the embedding that is passed to the attention-based pooling, see Fig. 8 in Appendix C. We use SmAP-early by default. Sec. 5.3 shows that results do not differ much among configurations.SmTAP: Smooth Transformer Attention Pooling. This is represented in Fig. 3c . The only difference with SmAP is that the neural network acting independently on the instance embeddings is replaced by a transformer encoder to account for global dependencies. Based on the idea that smoothness can be imposed at previous locations, recall Eq. 4, we propose to also apply Sm to the transformer output:EQUATIONwhere q, k, and v are the standard queries, keys, and values in the dot product self-attention [4] .Notice that SmTAP uses Sm in two places: the first after the transformer encoder and the second in the aggregator. Naturally, one could think of other variants that use Sm in only one place or the other. In Sec. 5.3 we ablate these different configurations, leading to similar results. 
experiments: 
 We validate the proposed Sm in three medical MIL datasets: RSNA [13] , PANDA [5] , and CAME-LYON16 [2] . We evaluate the effectiveness of our approach by a quantitative and qualitative analysis.All experiments have been conducted under fair and reproducible conditions. Details on the datasets and experimental setup can be found in Appendix B. The code has been uploaded as supplementary material and will be uploaded to GitHub upon the acceptance of the paper.We compare our approaches with state-of-the-art deep MIL methods. We consider two groups of methods, depending on the presence/absence of a transformer block to model global dependencies. In the first group, we include those models that do not use this block: the proposed SmAP, ABMIL [17] , CLAM [25] , DSMIL [21] , and DFTD-MIL [39] . The second group consists of models that do use the transformer encoder: the proposed SmTAP, TransMIL [32] , SETMIL [40] , GTP [41] , and CAMIL [14] . These groups ensure a fair comparison in terms of model capabilities and complexity. In Sec. B.3 we report the results of three more methods: DeepGraphSurv [24] , PathGCN [11] , and IIBMIL [29] . Note that the performance obtained by these methods does not affect the conclusions we will obtain in this section.In Sec. 5.1 we consider the localization task. In Sec. 5.2 we turn to the classification task. Sec. 5.3 shows an ablation study on how different uses of the smooth operator affect the proposed model.In this subsection, we analyze the ability of each model to predict the label of the instances inside a bag. As explained in Sec. 3, deep MIL models assign a scalar value f n to each instance x n , see Fig. 1a . Although these can be obtained in different ways, for simplicity we will refer to them as attention values. Thus, we compare the attention values with the ground truth instance labels, which are available for the test set only for evaluation purposes.Quantitative analysis. We analyze the performance of each method using the area under the ROC curve (AUROC) and the F1 score. Note that a critical hyperparameter for the latter is the threshold used on f n to determine the label of each instance. To ensure a fair comparison, we compute the optimal threshold for each method using the validation set. As a general summary, we also report the average rank achieved by each model across metrics and datasets.The results are shown in Table 1 . We find that using Sm provides the best performance overall, placing as the best or second-best within each group. Only in RSNA the proposed SmAP is outperformed by ABMIL. We attribute this to the fact that the bag graphs in CT scans are not as complex as in WSIs, and therefore the local interactions are not as meaningful. Note that the performance gain is particularly significant on CAMELYON16, where the bags have a larger number of instances, the graphs are much denser and the imbalance between positive and negative instances is more severe. Notably, SmTAP significantly outperforms SETMIL, GTP, and CAMIL, which also model local dependencies. Contrary to our method, their design is focused on bag-level performance and it does not translate into meaningful instance-level properties.Attention histograms. We examine the attention histograms produced by each model on the CAME-LYON16 dataset. The corresponding figures for RSNA and PANDA can be found in Appendix C. In Fig. 4 , we represent the frequency with which attention values are assigned to positive and negative instances, separately. An ideal instance classifier would place all the positive instances on the right and all the negative instances on the left. This illustrates why SmTAP and SmAP achieve such a good performance: they concentrate the negative instances to the left of the histogram while succeeding in grouping a large part of the positive instances to the right. TransMIL and GTP assign low attention values to both positive and negative instances. CAMIL is able to identify positive instances, but negative instances are assigned from intermediate to high attention values. CLAM and DSMIL assign low attention values to negative instances, but the distribution of the positive instances resembles a uniform and a normal distribution, respectively.Attention maps. To visualize the localization differences, we show the attention maps generated by four of the transformer-based methods in a WSI from CAMELYON16, see Fig. 5 . SmTAP attention The results shown in this subsection validate the utility of the smooth operator at the instance level. They suggest that having smooth attention maps is a powerful inductive bias that improves the instance-level performance. In the following, we analyze its impact at the bag level.In this subsection, we show that the use of the smooth operator does not deteriorate the bag classification results. On the contrary, in some cases, it improves them. Again, we focus on the AUROC and F1 scores, measured by comparing the true bag labels with the methods' bag label predictions. The threshold for the F1 score is 0.5. We also report the mean rank achieved by each model.Table 2 shows the results. The proposed models achieve the best performance overall. As in the localization task, ABMIL performs better than SmAP in RSNA. Again, we believe it to be a consequence of the CT scan's low-complexity structure. DFTD-MIL obtains the best result in CAMELYON16, but ranks second or third in the other two datasets. GTP and SETMIL outperform the proposed SmTAP in PANDA, but their performance significantly decreases in CAMELYON16, obtaining the worst results. Overall, our methods provide the most consistent performance, achieving an aggregated mean rank of 1.833.The proposed Sm comes with different design choices and hyperparameters: the placement of Sm, the trade-off parameter α, the number of approximation steps T , and the use of spectral normalization. We analyze them in the following, showing that Sm leads to enhanced results almost under any choice. This supports that our hypothesis -that neighboring instances are likely to have the same label -is a powerful inductive bias worth exploring. Recall that SmAP leverages by default the early variation, but we also described SmAP-mid and SmAP-late. Likewise, we discussed different variants for SmTAP. Table 3 summarizes the impact of these choices on the final performance.Sm without the transformer encoder (SmAP). These variants differ in the place where Sm is located inside the attention pool, recall Eq. 10-Eq. 12. We include ABMIL since we build our model on top of it. We see that using SmAP improves the performance at both instance and bag levels. This improvement is more noticeable in PANDA and CAMELYON. We attribute it to the bag graph structure being more complex in WSIs than in CT scans. Also, the Dirichlet energy is lower when the smooth operator is used, as theoretically expected. We observe that the proposed method is robust to different placement configurations, which is consistent with the theoretical guarantees presented in Sec. 4.1. However, none of the variants consistently outperforms the others.Sm with the transformer encoder (SmTAP). Recall that SmTAP leverages Sm both after the transformer encoder and inside the attention pooling. Here we will refer to it as SmT+SmAP, and will compare it against T+SmAP and SmT+AP (using Sm only in one of the components) and against T+AP (not using Sm). We observe that Sm has no significant effect on bag-level performance. At instance-level we do observe differences: the baseline T+AP is outperformed as long as Sm is used within the attention pooling.In the following we study the influence of the trade-off parameter α and of the spectral normalization. Due to space limitations, the analysis for the number of approximation steps T is in Sec. B.3.The trade-off parameter α. From Eq. 6 we see that α ∈ [0, 1) controls the amount of smoothness enforced by Sm. Note that α = 0 in Eq. 7 produces no smoothness, turning Sm into the identity operator. In Fig. 6a we show the performance obtained for different values of α in CAMELYON16. Each choice of this hyperparameter improves upon the baseline ABMIL (α = 0). We see that better localization results are obtained when α is lower, while better classification results are obtained when Setting α > 0 improves upon the baseline ABMIL (α = 0) and is a trade-off between better localization results (lower α) or better classification results (higher α). Likewise, Sm without spectral normalization already improves the results upon the baseline (ABMIL), but the best performance is obtained when they are used together.α is higher. Fixing α = 0.5 is a compromise between the two, and produces very similar results as setting it as a trainable parameter initialized at α = 0.5. Fig. 12 provides a visual comparison of the effect that α has on the attention maps.The effect of spectral normalization. Spectral normalization forces the norm of the multi-layer perceptron weights to remain constant. In this work, this is a key design choice that helps Sm to obtain attention maps with lower Dirichlet energy. In our experiments, we have used spectral normalization in the layers immediately after Sm. Note that the late variant does not require spectral normalization, since it applies Sm directly to the attention values. In Fig. 6b we show the results obtained with and without spectral normalization in CAMELYON16. We observe that, even without spectral normalization, Sm improves upon the baseline. The improvement is more significant when Sm is paired with spectral normalization, especially at the instance level. 
discussion and conclusion: 
 The main goal of this paper is to draw attention to the study of MIL methods at the instance level.To that end, we revised current deep MIL methods and provided a unified perspective on them. We proposed the smooth operator Sm to introduce local interactions in a principled way. By design, it produces smooth attention maps that resemble the ground truth instance labels. We conducted an exhaustive experimental validation with three real-world MIL datasets and up to eight state-of-the-art methods in both classification and localization tasks. This study showed that our method provides the best performance in localization while being highly competitive (best or second best) at classification.Despite its advantages, our method has some limitations. The first is that, as with every other operator in GNNs, the computational costs of the smooth operator scale with the size of the bag. Fortunately, it can be paired with existing subgraph sampling techniques to mitigate this problem. The second limitation is that we do not have a definite answer for where it is better to use the proposed operator.We have shown that it leads to improvements in almost every place, but the optimal location may be problem-dependent and has to be tailored by the practitioner.Finally, we hope that our work will draw more attention to the localization problem, which is very important for the deployment of computer-aided systems in the real world. In this sense, safely deploying the proposed methods in clinical practice requires evaluating them in a wider range of medical problems and quantifying their uncertainty. For the latter, we believe that the smooth operator could also benefit from a probabilistic formulation. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes] Justification: This information can be found along with the rest of the experimental configuration information, see Appendix B.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: As pointed out in Sec. 1 and in Sec. 6, one of the main goals of this work is to draw attention to the localization task in MIL, which can contribute favorably to the deployment of computer-aided systems in the real world. However, as we also indicate in Sec. 6, for this to happen safely (e.g., avoiding misdiagnosis), further investigation is needed about equipping them with uncertainty estimation mechanisms. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: The paper poses no such risks. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: We cite the original works where the methods were proposed. We used the code provided by the corresponding authors. Each paper includes the URL to its GitHub repository, and each implementation is released under (possibly) a different license, specified in the mentioned repository. 
 
 Appendix: A.1 Proof of Eq. 4We present a general result for an arbitrary multilayer perceptron with Lipschitz activation functions. Note that assuming Lipschitzness is not a restriction, since most of the currently used activation functions meet this property [6] . The desired Eq. 4 is a particular case of this result. Let L ∈ N. Consider a L-layer perceptron that, given Y ∈ R N ×D0 , outputs Y L ∈ R N ×D L defined by the following ruleEQUATIONEQUATIONwhere W ℓ ∈ R D ℓ ×D ℓ+1 , andB ℓ = [b ℓ , . . . , b ℓ ] ⊤ ∈ R N ×D ℓ+1where b ℓ ∈ R D ℓ+1 are trainable weights, and φ ℓ : R → R are activation functions applied element-wise. We suppose that each activation function φ ℓ is K ℓ -Lipschitz. Then, we obtain the following inequality,EQUATION)Before verifying it, we note that by applying this inequality to every layer, we arrive atEQUATIONwhere∥W ℓ:0 ∥ 2 = ℓ j=0 ∥W j ∥ 2 and K ℓ:0 = ℓ j=0 K ℓ . Taking L = 2, D 0 = D, D 1 = L, D 2 = 1, b 0 = b 1 = 0, φ 0 =tanh, and φ 1 = Id, we recover Eq. 4.To verify Eq. 16, we write Y ℓ+1 = y ℓ+1 1 , . . . , y ℓ+1 N ⊤and Y ℓ = y ℓ 1 , . . . , y ℓ N ⊤ . We have,EQUATIONwhere from Eq. 19 to Eq. 20 we have used the definition of Lipschitz function and from Eq. 20 to Eq. 21 we have used the consistency between the spectral and Euclidean norms.A.2 Proof of Eq. 9In this section, we adapt the proof presented in [6] for a similar result. Let U ∈ R N ×D . Our goal is to show thatEQUATIONwhere γ > 0 and λ * γ = max (1 + γλ n ) -2 : λ n ∈ Λ \ {0} , being Λ =EQUATIONNext, it is useful to note that if λ n is an eigenvalue of L with associated eigenvector v n , then (1 + γλ n ) -1 is an eigenvalue of (I + γL) -1 with the same associated eigenvector. Finally, let {v 1 , . . . , v N } be a orthonormal eigenbasis of L, being each v n associated to the eigenvalue λ n . This basis always exists since L is a symmetric matrix. Writing u = N n=1 c n v n , with c n ∈ R, we haveEQUATIONUsing that the eigenvectors are orthogonal to each other, we arrive atEQUATIONB Experiments: details and further resultsIn this section, we provide the details of the datasets, architectures, and configurations used for each experiment. The code has been uploaded as supplementary material and will be uploaded to GitHub upon the acceptance of the paper.We provide insights into the datasets we have used: a description of the problem, the train/test splits, and preprocessing (instance selection and feature extraction). For all datasets, we obtain an initial train/test partition. Then, we split the initial train partition into five different train/validation splits. Every model is trained on each of these splits and then evaluated on the test set. We report the average performance on this test set.RSNA. It was published by the Radiological Society of North America (RSNA) to detect acute intracranial hemorrhage and its subtypes [13] . It is available in Kaggle 1 . We use the official train-test split. It includes a total of 1150 scans. There are a total amount of 39750 slices and the number in each scan varies from 24 to 57. Each slice is preprocessed following [36] .PANDA. It is a public dataset for the classification of the severity of prostate cancer from microscopy scans of prostate biopsy samples [5] . It is available in Kaggle2 . Since the official test set is not publicly available, we use the train/test split proposed in [33] . To extract the patches from each WSI, we follow the procedure described in [33] , obtaining patches of size 512 × 512 at 10× magnification. This results in a total amount of 10503 WSIs and 1107931 patches.CAMELYON16. It is a public dataset for the detection of breast cancer metastasis [2] . It is available at the Registry of Open Data of AWS 3 . The official repository contains 400 WSIs in total, including 270 for training and 130 for testing. From each WSI, we extract patches of size 512 × 512 at 20× magnification using the method proposed by Lu et al. [25] .We provide details about how we have implemented the proposed methods and how we have conducted the experiments.Feature extractor. Due to the limited memory of the GPU, it is necessary to extract features from each instance. Otherwise, the bags will not fit in memory. In this work, we consider three options for the feature extractor, all of which are pre-trained in Imagenet: ResNet18 (P = 512), ResNet50 (P = 2048), and ViT-B-32 (P = 768). In addition, for CAMELYON16 we also consider ResNet50-BT 4 (P = 2048), which is a ResNet50 model pre-trained using the Barlow Twins Self-Supervised Learning method on a huge dataset of WSIs patches [38, 18] . The results reported in the main text Model architecture. We describe the architecture we have used for the proposed methods (SmAP and SmTAP). For the rest of the methods considered, we adopt their original implementations and default configurations, publicly available on their GitHub repositories. For the independent instance encoding part (see Fig. 3a and Fig. 3b ), the instance embeddings h n are obtained using one fully connected layer with 512 units (D = 512). For the attention-based pooling described by Eq. 1 and Eq. 2, we fix D = 512 and L = 100. The transformer encoder in Fig. 3c is implemented using two transformer layers. These layers use the standard multi-head attention mechanism equipped with skip connections and layer normalization [4] . We fix the key, query, and value dimensions to 128 and the number of heads to 8. We used the Pytorch's implementation of dot product attention 5 . Finally, the bag-embedding classifier was implemented using one fully connected layer.Training setup and hyperparameters. To ensure fair and reproducible conditions, we trained every method under the same setup. The number of epochs was set to 50. We adopt the Adam optimizer [19] with the default Pytorch configuration. For the base learning rate, we considered two different values, 10 -4 and 10 -5 , since we noticed that models that do not use transformers obtained better results when the learning rate was higher. We report the best results for each model. We adopted a slow start using Pytorch's LinearLR scheduler with start_factor=0.1 and total_iters=5.During training, we monitored the bag AUROC and cross-entropy loss in the validation set and kept the weights that obtained the best results. The batch size was set to 32 in RSNA and PANDA. In CAMELYON16, it was set to 4 for no-transformer methods, and to 1 for transformer-based methods. However, for SETMIL, we had to set it to 1 in PANDA and CAMELYON16 due to its high GPU memory requirements. In RSNA we weighted the loss function to account for the imbalance between positive and negative bags since we observed it to improve the results. All the experiments were performed on one NVIDIA GeForce RTX 3090.We complete the ablation study presented in the main paper, Sec. 5.3, by looking at the rest of the design choices or hyperparameters associated with our Sm.Smooth operator approximation. The exact form of Sm given by Eq. 7 becomes computationally infeasible for large bag sizes. The quality of the approximation, given by Eq. 8, is controlled by the number of steps T . Fig. 7 shows the results for different values of this hyperparameter in RSNA and CAMELYON16. In RSNA, since the bags are smaller, we can compute the closed-form solution, which we represent as T = ∞. Almost any choice of T > 0 improves upon ABMIL. This improvement is particularly noticeable in CAMELYON16. Moreover, in most cases, the performance stabilizes at T = 10, which is the value we used in our experiments.Sm on top of other models. We have proposed two new models (SmAP and SmTAP) by applying Sm on top of two baselines (ABMIL and Transformer+ABMIL, respectively). Instead, the Sm can be applied on top of other existing approaches. In Table 4 we explore how other approaches behave when combined with the proposed Sm in the CAMELYON16 dataset. Instance-level performance is enhanced (greatly in some cases, e.g. an increase from 0.76 to 0.96 in AUROC for DSMIL), whereas bag-level results are competitive. The decrease in bag-level results for DFTD-MIL is explained by the fact that this method randomly splits each bag into different chunks. This may lead to the loss of local interactions exploited by Sm (e.g. if two adjacent instances end in different chunks).An alternative smoothing strategy. Introducing a penalty term in the loss function to favor smoothness is a natural alternative to the proposed operator. However, there is an important difference: the use of a penalty term does not modify the model architecture. The penalty term favors that the learned weights encode such a property, but it is not explicitly encoded in the model. For instance, note that the penalty term is not used at inference time. We compare the penalty-based approach and the proposed Sm in Table 5 . Although differences are not large, Sm obtains superior performance.Feature extractor. We investigate whether the choice of the feature extractor influences the results and conclusions presented in the main text. We have evaluated each of the considered methods in each dataset using the feature extractors mentioned above (ResNet18, ResNet50, ViT-B-32, and ResNet50-BT). The results are shown in Tables 7 8 9 10 . We summarize them in Table 6 , where we collect the average instance and bag rank of each method for each feature extractor. We observe that the proposed smooth operator Sm obtains in almost all cases the highest rank. This supports the idea that the improvement introduced by Sm does not depend on the used features. NeurIPS Paper Checklist Justification: We discuss the limitations of the proposed method in Sec. 6. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.Justification: Our research completely conforms with the NeurIPS Code of Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).https://www.kaggle.com/c/rsna-intracranial-hemorrhage-detectionhttps://www.kaggle.com/c/prostate-cancer-grade-assessment/datahttps://registry.opendata.aws/camelyon/Weights available at https://github.com/lunit-io/benchmark-ssl-pathology.https://pytorch.org/docs/stable/generated/torch.nn.functional.scaled_dot_ product_attention.html 


Decision: This paper introduces an approach to improve the localization task of multiple instance learning (MIL) methodologies, particularly the ones applied to CT and WSI. Relying on a spatial dependency between training samples, the paper presents a smoothing operator to regularize the attention matrix with respect to inter-sample similarity. The paper also presents a theoretical analysis of the method. The proposed method shows better results than competing approaches and the ablation study supports its claims. The paper received an average score of 5.33 (5,5,6), and the main strengths found are:  1) paper is well-written and concise, 2) claims are well supported by experimental results, 3) idea explored in the paper is interesting, and 4) qualitative experiments show the model’s advantages.  The reviewers also identified the following weaknesses: 1) negligible differences between ABMIL and the proposed method; 2) missing experiment on the application of the proposed smooth operator on top of other methods (e.g., CLAM, CAMIL, SETMIL); 3) missing ablation study on a comparison between the minimisation of Dirichlet energy and other smoothing strategies; 4) missing qualitative experiment on the role of alpha parameter; 5) unclear role of the smoothness property for the classification from a neural network; 6) unconvincing results using only 3 datasets, where the proposed method failed to get first place in 2/3 of the AUC or F1 scores; 7) unclear justification for different encoders for different datasets; 8) missing comparisons with graph-based methodologies; and 9) missing comparison with MIL methods that rely on pseudo labels. There was a fair bit of discussion during the rebuttal, and the only remaining issue of the paper is that the performance improvement compared to competing approaches is marginal. I support the publication of this paper, but encourage the authors to address the issues raised during this rebuttal phase.

Reviews: review: paper_summary: paper_summary: The authors propose a technique to improve the localization capabilities of the current MIL, especially for CAD models that perform CT and WSI analysis. The method is based on seeing the attention attributed to each patch as a graph and minimizing its Dirichlet energy, promoting smooth transitions on the attention values of neighbor patches, which is in accordance with the assumption that neighboring patches share the same labels.
strength_weakness: strength_weakness: The method introduces the smooth operator on top of the existing ABMIL model. The proposed approach improves the performance of ABMIL on the CAMELYON dataset, both on localization and classification. Nevertheless, when analyzing the attention maps provided in the appendix, the differences between ABMIL and smAP are minimal. Although we have a numerical evaluation of the Dirichlet energy, we cannot tell if the drop in energy is significant only by these numbers, so this visual empiric evaluation is important for a better insight regarding the developed work. 

Observing the appendix figures, ABMIL already performs well in matching its attention to the tiles annotations. I suggest the authors apply the smooth operator on top of other models that did not perform well on CAMELYON for this task, such as CLAM, CAMIL, and SETMIL, to evaluate how minimizing the Dirichlet energy of the attention improves the localization task quantitatively and qualitatively.

I suggest the authors perform an ablation study to evaluate how minimizing the Dirichlet energy compares to other smoothing strategies. For instance, summing the variance of the attentions to the model’s loss may provide similar results as minimizing the Dirichlet energy. 

The analysis of the alpha parameter is great, but again, a qualitative approach is valuable for this case. Seeing how the attention map changes as the parameter alpha increases is highly valuable for the reader.

Lines 127 to 131: trying to debate equations not present in the paper is not good for the reader. I suggest writing the debated equations in the article or explaining the arguments without mentioning them.
questions: questions: The formulation seems to assume bags of equal size, which is not the case in MIL (in general). 

The proposed method (loss) may have the drawback of over-smoothing the transition between tiles, making the localization regions blurry. Is this true? Could this be overcome?
limitations: limitations: yes
paper_summary: paper_summary: The paper proposed a multi-instance learning approach. The basic idea is to make use of the spatial dependency between training samples. A smoothing operator was proposed to regularize the attention matrix with respect to inter-sample similarity (to my understanding), which the authors claimed to improved both quality of localization and classification tasks.
strength_weakness: strength_weakness: 1. Although the smoothness property do exist in multi-instance learning as the authors claimed and may introduce useful information. It is still unclear why imposing it in the neural network helps with classification. It doesn't seem to serve as an induction bias from its looks.
2. Actually only 3 dataset are tested (though each with different variants), and the results are not so persuasive. Especially for the classification tasks that the authors want to credit, they failed to get 1st place in 2/3 of the AUC or F1 scores.
questions: questions: 1. The overall principle of the method need rephasing, i.e., how introduction the SM transformation (instead of penalty) improves classification? 
2. The smoothness assumption seems to be a property connected with localization. However, in 4.1-4.3 and Fig.3 the authors mainly present the SM operator for  classification. Yet in the experiment part, it seem that adding SM had more advantages in localization (but not sure how is it achieved?). The authors need to clarify the relationship between the two tasks.
3. In Tab. 1, why the performance of the 'with global interactions' variants worse than the 'without' one in localization, but it is opposite in the classification (Tab. 2)?
limitations: limitations: The method required strict ordering (or spatial information) or 3d-to-2d samples, which is quite a strong demand but it is not discussed in the limitations.
paper_summary: paper_summary: The paper is well-written and proposes a new modular mechanism that can (selectively) combine local and global interactions in a model. The authors discuss the theory behind their proposed smoothness operator and pooling mechanism and provide detailed proof for their claims. They have also evaluated their model against other methods on three different datasets (1 CT and 2 H&E) with reporting AUC and F1 score as their evaluation metrics. Their method is superior to other methods and the proper ablation study supports their claims.
strength_weakness: strength_weakness: There are three major concerns with this work:
1. The choice of encoder. The authors have used Resnet18 for RSNA and Camelyon16 and Resnet50 for PANDA. I could not find any justification for why different encoders have been used! Also, with the trend toward foundation models, transformer-based backbones are now of interest to the community. Therefore, for the sake of consistency, the author should use the same encoder for different datasets or report both resent50 and resnet18. And, for the sake of the generality of their work, they should add one encoder such as ViT or Swin as well (w/ ImageNet weights) to support the generality of their claims.

2. The body of research has been founded on local-to-global interactions. There are quite a few standard graph-based methodologies in the literature that the authors need to compare their work against. Currently, there are no representatives from those families of the MIL method in the benchmark. Two examples of such methods are (1) and (2).

3. There is a family of MIL methods in the literature that try to pseudo-label the instances during training, which is essentially equivalent to localization in this work. For instance, two of the most recent such methods are (3) and (4). The author should compare their method against these as they are essentially from the same family.

(1) Chen, Richard J., et al. "Whole slide images are 2d point clouds: Context-aware survival prediction using patch-based graph convolutional networks." Medical Image Computing and Computer Assisted Intervention–MICCAI 2021: 24th International Conference, Strasbourg, France, September 27–October 1, 2021, Proceedings, Part VIII 24. Springer International Publishing, 2021.

(2) Li, R., Yao, J., Zhu, X., Li, Y., Huang, J. (2018). Graph CNN for Survival Analysis on Whole Slide Pathological Images. In: Frangi, A., Schnabel, J., Davatzikos, C., Alberola-López, C., Fichtinger, G. (eds) Medical Image Computing and Computer Assisted Intervention – MICCAI 2018. MICCAI 2018. Lecture Notes in Computer Science(), vol 11071. Springer, Cham. https://doi.org/10.1007/978-3-030-00934-2_20

(3) Z. Shao et al., "LNPL-MIL: Learning from Noisy Pseudo Labels for Promoting Multiple Instance Learning in Whole Slide Image," 2023 IEEE/CVF International Conference on Computer Vision (ICCV), Paris, France, 2023, pp. 21438-21438, doi: 10.1109/ICCV51070.2023.01965.

(4) Ren, Q. et al. (2023). IIB-MIL: Integrated Instance-Level and Bag-Level Multiple Instances Learning with Label Disambiguation for Pathological Image Analysis. In: Greenspan, H., et al. Medical Image Computing and Computer Assisted Intervention – MICCAI 2023. MICCAI 2023. Lecture Notes in Computer Science, vol 14225. Springer, Cham. https://doi.org/10.1007/978-3-031-43987-2_54
questions: questions: I am curious if the author expects the same performance boost obtained on top of resent, to be obtained on top of a histopathology-specific feature extractor.
limitations: limitations: It has been justified :)

Comments: title: Concern whether our reply was satisfactory
Have your concerns been addressed?
On the new questions
reviewer feedback
Please read the rebuttal to check if the authors addressed your concerns
Please read the rebuttal to check if the authors addressed your concerns
Please read the rebuttal to check if the authors addressed your concerns
Questions regarding the new experiments
comment: I appreciate the authors' effort in providing further details and explanations for my questions. The response provided addressed my concern partly, yet I agree with Reviewer YAE1 regarding the improvement being marginal. However, given the efforts in providing newer benchmarks and also considering that the content needs to be updated with consistent backbones for different datasets (as provided in the previous reply). Therefore, I would raise my rating to Borderline Accept).
In case it is necessary, we want to summarize our response to the new questions by the reviewer. 

> **Q. It has been mentioned in line 45 that "This is a flexible module that can be used alone on top of classical MIL approaches, or in combination with transformers to also account for global dependencies." If I have understood it right, the main contribution is to have sm as a local-global module that helps to improve performance by capturing both local and global. Judging only from Table 1 and Resnet50 with ImageNet weights, how do authors justify the models without global interactions that are superior to their proposed smTAP with global interactions? If I am not missing anything, this is a bit contradictory to the claims of the paper.**

Instance level results worsen with transformers. This is not a consequence of the proposed Sm, but of the oversmoothing phenomenon observed in transformers. This has been studied in the works referenced in our previous response. 

The proposed method without a transformer outperforms methods that do not a transformer; and the proposed method with a transformer outperforms methods that use a transformer. 


> **Q. Also, comparing Resnet50+BT and Resnet50 only for the Calmelyon16 dataset, it seems that their proposed model benefits heavily from pre-trained features (improvement comes mostly from the encoder rather than the aggregation proposed). I am wondering if the authors can justify this observation. Does this come from any biological underlying meaning or does the Sm method can be intuitively linked to anything?**

Note that every method has access to the same features, and every method benefits from using them (see bag level results in Table 2). That our method benefits at the instance level from using, we interpret it as a sign that local dependencies indeed play an important role in MIL localization. 

Please let us know if you have any questions so that we can respond in a timely manner.

We look forward to your response, 

Authors
Dear Reviewer jUey,

We are wondering if you need any additional clarification. Please let us know so that we can respond to you in a timely manner. 

We look forward to your response, 

Authors
We would like to thank Reviewer uFQt for their positive comments on the new experiments and the new insights we have provided. We address the new questions below. 

> **Q. It has been mentioned in line 45 that "This is a flexible module that can be used alone on top of classical MIL approaches, or in combination with transformers to also account for global dependencies." If I have understood it right, the main contribution is to have sm as a local-global module that helps to improve performance by capturing both local and global. Judging only from Table 1 and Resnet50 with ImageNet weights, how do authors justify the models without global interactions that are superior to their proposed smTAP with global interactions? If I am not missing anything, this is a bit contradictory to the claims of the paper.**

As we indicate in the paper, the proposed Sm operator introduces local interactions in a principled manner (through Dirichlet energy minimization with the presented theoretical guarantees). This operator can be naturally combined with self-attention layers (transformer layers) to account for both local and global interactions. 

As the reviewer points out, in Table 1 of the rebuttal PDF, when looking at ResNet50 with Imagenet weights, the models without global interactions (i.e., no transformer layers) do better at the instance level (localization task), with the proposed SmAP achieving the highest rank (see the table in the rebuttal to Reviewer uFQt). However, when looking at the classification task (bag level, Table 2), the models with global interactions outperform to those without global interactions, with the proposed SmTAP achieving the highest rank (see the table in the rebuttal to Reviewer uFQt). 

These results suggest that models without global interactions (SmAP) perform better at the instance level (localization task, Table 1), while models with global interactions perform better at the bag level (classification task, Table 2). This behaviour was also pointed out by Rev. YAE1. As we argue in the rebuttal to Reviewer YAE1, this is explained by the fact that self-attention layers of transformers obtain a richer bag representation (which leads to better bag level results). However, the signal that identifies each token in the sequence becomes weaker after each self-attention layer (the tokens become more similar). This leads to worse instance level results. Note that this phenomenon is known as over-smoothing and has been observed in (1), (2), (3) (see references below). 

We believe that the results in Table 1 are not contradictory with the claims in the paper, but rather reinforce them. The proposed Sm improves instance level results (localization) and remains competitive at the bag level (classification). This holds true whether it is combined with transformer layers (SmTAP) or not (SmAP).  

> **Q. Also, comparing Resnet50+BT and Resnet50 only for the Calmelyon16 dataset, it seems that their proposed model benefits heavily from pre-trained features (improvement comes mostly from the encoder rather than the aggregation proposed). I am wondering if the authors can justify this observation. Does this come from any biological underlying meaning or does the Sm method can be intuitively linked to anything?**

As the reviewer points out, the proposed Sm benefits from SSL pre-trained features. Our intuition is that if an SSL pre-trained encoder is used, instances that are similar from a biological point of view will tend to cluster together in the feature space. In particular, instances with the same label will be clustered together, and far away from instances with a different label. Since neighboring instances are naturally (a priori) expected to have similar biological properties (e.g., the same label), the smoothness introduced by the proposed Sm will be particularly beneficial. This interesting observation is related to that in CAMIL [13], where it was shown that a SSL pre-trained encoder is crucial for the optimal performance of their neighbor-constrained attention mask.

----

(1) Shi, Han, et al. "Revisiting Over-smoothing in BERT from the Perspective of Graph." International Conference on Learning Representations (ICLR). 2022.

(2) Wang, Peihao, et al. "Anti-Oversmoothing in Deep Vision Transformers via the Fourier Domain Analysis: From Theory to Practice." International Conference on Learning Representations (ICLR). 2022.

(3) Dong, Yihe, Jean-Baptiste Cordonnier, and Andreas Loukas. "Attention is not all you need: Pure attention loses rank doubly exponentially with depth." International Conference on Machine Learning (ICML). PMLR, 2021.
Thanks for the authors' effort to present additional results and make clarifications.   I raised the score despite the concern that this still seems to be a marginal improvement.
Dear Reviewer uFQt, 

Can you have a look at the rebuttal and see if your concerns have been addressed? 

Best regards
Your AC.
Dear Reviewer YAE1, 

Can you have a look at the rebuttal and see if your concerns have been addressed? 

Best regards
Your AC.
Dear Reviewer jUey, 

Can you have a look at the rebuttal and see if your concerns have been addressed? 

Best regards
Your AC.
I would like to thank the authors for extending their experiments and including new insights. I would appreciate the authors if they could provide more details regarding the following questions.

It has been mentioned in line 45 that "This is a flexible module that can be used alone on top of classical MIL approaches, or in combination with transformers to also account for global dependencies." If I have understood it right, the main contribution is to have sm as a local-global module that helps to improve performance by capturing both local and global. Judging only from Table 1 and Resnet50 with ImageNet weights, how do authors justify the models without global interactions that are superior to their proposed smTAP with global interactions?
If I am not missing anything, this is a bit contradictory to the claims of the paper. 

Also, comparing Resnet50+BT and Resnet50 only for the Calmelyon16 dataset, it seems that their proposed model benefits heavily from pre-trained features (improvement comes mostly from the encoder rather than the aggregation proposed). I am wondering if the authors can justify this observation. Does this come from any biological underlying meaning or can the Sm method be intuitively linked to anything?