xMIL: Insightful Explanations for Multiple Instance Learning in Histopathology
Authors: Julius Hense
Mina Jamshidi Idaji
Oliver Eberle
Thomas Schnake
Jonas Dippel
Laure Ciernik
Oliver Buchstab
Andreas Mock
Frederick Klauschen
Klaus Robert Muller
Venue: NeurIPS 2024 poster (2024)
Keywords: computational pathology
multiple instance learning
explainable AI
layer-wise relevance propagation
knowledge discovery
Primary area: machine_learning_for_healthcare

Abstract: Multiple instance learning (MIL) is an effective and widely used approach for weakly supervised machine learning. In histopathology, MIL models have achieved remarkable success in tasks like tumor detection, biomarker prediction, and outcome prognostication. However, MIL explanation methods are still lagging behind, as they are limited to small bag sizes or disregard instance interactions. We revisit MIL through the lens of explainable AI (XAI) and introduce xMIL, a refined framework with more general assumptions. We demonstrate how to obtain improved MIL explanations using layer-wise relevance propagation (LRP) and conduct extensive evaluation experiments on three toy settings and four real-world histopathology datasets. Our approach consistently outperforms previous explanation attempts with particularly improved faithfulness scores on challenging biomarker prediction tasks. Finally, we showcase how xMIL explanations enable pathologists to extract insights from MIL models, representing a significant advance for knowledge discovery and model debugging in digital histopathology.

xMIL: Insightful Explanations for Multiple Instance Learning in Histopathology
Julius Hense 122 Mina Jamshidi Idaji 12.
Oliver E 1.2 Thomas Schnake 1.2
Jonas Dippel Laure Ciernik 1.2
Oliver Buchstab
Andreas Mock 1,2,7,8.
Frederick Klauschen 14,5,6
Klaus-Robert Müller 1.27,.8
Berlin Institute for the Foundations of Learning and Data, Berlin, Germany Machine Learning Group, Technische Universität Berlin, Berlin, Germany
Aignostics GmbH, Berlin, Germany
"Institute of Pathology, Ludwig Maximilian University, Munich, Germany
GGerman Cancer Research Center, Heidelberg, and German Cancer Consortium, Munich, Germany 6 Institute Pathology, Charité Universitätsmedizin, Berlin, Germany
7 DDepartment of Artificial Intelligence, Korea University, Seoul, Korea MMxx-Planc Institute for Informatics, Saarbrücken, Germany "Equal contribution
{{ j. hense,,, mina. jamshidi idaji, klaus- wwwessrobert..mae. mueller J@tu-berlin..
Abstract
Multiple instance learning (MIL) an effective and widely used approach for weakly supervised machine learning. In histopathology, MIL models have achieved remarkable success tasks like tumor detection, biomarker prediction, and out- come prognostication. However, MIL explanation methods are still lagging behind, as they are limited small bag sizes or disregard instance interactions. We revisit MII. through the lens of explainable AI (XAl) and introduce xMII., a refined frame- work with more general assumptions. We demonstrate how to obtain improved MII. explanations using layer-wise relevance propagation (LRP) and conduct extensive evaluation experiments on three toy settings and four real-world histopathology datasets. Our approach consistently outperforms previous explanation attempts with particularly improved faithfulness scores on challenging biomarker prediction tasks. Finally, showcase how xMIL explanations enable pathologists to ex- tract insights from MIL models, representing a significant advance for knowledge discovery and model debugging in digital histopathology. Codes available at: http://ggttb...ommbiiiiaaaaamiis
Introduction
is learning paradigm in which single label is predicted from a bag instances. Various MIL methods have been proposed, differing how they aggregate instances into bag information 3
Multiple instance learning (MIL.) 1 2
g 9 10 12]. MII. has become particularly pop- ular in histopathology. where gigapixel microscopy slides are cut into patches representing small tissue regions. From these patches, MIL models can learn to detect tumor [3] or classify disease subtypes (6), aiming to support pathologists their routine diagnostic workflows. They have further demonstrated remarkable success at tasks that even pathologists cannot perform reliably due to lack of known histopathological patterns associated with the target, e.g., predicting clinically relevant biomarkers
4 5
6 7
[14 [5] [6] or outcomes like survival [8] directly from whole slide images.
38th Conference Neural Information Processing Systems (NeurIPS 2024).

---

Slide
XMILL LRP
Attention
Single
Figure 1: In digital pathology, heatmaps guide the identification of tissue slide areas most important for a model prediction. The figure displays heatmaps from different MIL explanation methods (columns) for head and neck tumor slide (top row) with selected zoomed-in region (bottom row). The MII. model has been trained to predict HPV status. The xMIL--RPP heatmap shows that the model identified evidence in favor of an HPV infection at the tumor border (red area) and evidence against an HPV infection inside the tumor (blue area, lower half of the tissue). The dominant blue region explains why the model mispredicted the slide HPV-negative. Investigation of the tumor border by a pathologist revealed a higher lymphocyte density, which is one of the known recurrent but not always defining visual features of HPV infection in head and neck tumors. xMIL-LRP allows pathologists to extract fine- seee-grined insights about the model strategy. In contrast, the "attention" " and "single" methods neither explain the negative prediction nor distinguish the relevant areas.
Explaining which visual features a MIL model uses for its prediction is highly relevant this context. It allows experts to sanity-check the model strategy [19], e.g., whether model focuses on the disease area for making a diagnosis. This particularly important in histopathology. where models operating in high-stake environments are prone to learning confounding factors like artifacts or staining differences instead of actual signal (20 21 22] On top of that, MIL explanations can enable pathologists discover novel connections between visual features and prediction targets. For example, the explanations could reveal a previously unknown association of a histopathological pattern with poor survival, leading to the identification of targetable disease mechanism. Previous works have shown the potential of scientific knowledge discovery from explainable AI (XAI)
22
23
24
25 26)
Most studies have used attention scores MIL explanations 3 4
27
28 29 However, it has
been shown that attention heatmaps are limited in faithfully reflecting model predictions [30 31
32 [3
Further MIL explanation methods have been proposed, including perturbation schemes passing modified bags through the model (34] and architectural changes towards fully additive MIL models [33] Nevertheless, these methods do not account for the complexities inherent to many histopathological prediction tasks, as they are limited to small bag sizes or disregard instance interactions.
We revisit MIL through the lens of XAI and introduce xMIL, a more general and realistic multiple in- stance learning framework including requirements for good explanations. We then present xMIL-LRP, MIL. xMIL-LRP distinguishes between positive and negative evidence, disentangles instance interactions, and scales to large bag sizes. applies to various MIL models without requiring architecture modifications, including Attention MIL ( and TransMIL (4) We assess the performance of multiple explanation techniques via three toy experiments, which can serve as a novel benchmarking tool for MIL explanations in complex tasks with instance interactions and context-sensitiv targets. We further perform faithfulness experiments on four real-world histopathology datasets covering tumor detection, disease subtyping, and biomarker prediction. XMIL-LRP consistently outperforms previous attempts across all tasks and model architectures, with the biggest advantages observed for Transformer based biomarker prediction.
an adaptation of layer-wise relevance propagation (LRP) 35 36
Figure showcases the importance of understanding positive and negative evidence for prediction. Only XMIL-LRP uncovers that the model found evidence for the presence of the biomarker, but stronger evidence against This explains why it predicted the biomarker to be absent and enables

---

pathologists to extract insights about the visual features that support or reject the presence of the biomarker according to the model. The example illustrates the strength of our approach. suggesting that xMII.-I.PP represents significant advance for model debugging and knowledge discovery in histopathology.
The paper is structured as follows: In Section 2 we review MIL assumptions, models, and explanation methods related to this work. In Section 3 we introduce xMIL general form of MIL, and xMIL-
LRP as solution for it. In Section 4 we experimentally show the improved explanation quality of our approach. We demonstrate how to extract insights from example heatmaps in Section 5 Our contributions are summarized as follows:
Methodical:
Despite attempts to apply XAI MIL models in histopathology (e.g. 6 27 28
29
39 40), there exists no formalism guiding the interpretation of the heatmaps and defining their desired properties. xMIL is novel framework addressing this gap. Within xMILL, heatmaps estimate the instances' impact on the bag label, which makes their interpretation straightforward and insightful.
33 34
37
38
Empirical: Our extensive empirical evaluation of XAI methods for MIL on synthetic and real- world histopathology datasets is the first of its kind. It reveals that the widely used MIL explanation methods regularly yield misleading results. In contrast, xMIL-LRP sets a new state-of-the of-tth-art for explainability in AttnMIL and TransMIL models in histopathology.
Previous studies [33 34 conducted qualitative assessments of heatmaps on casy-to-learn datasets like CAMELYON or TCGA NSCLC. The insights gained in these settings are limited to model debugging, i.c., "Does the model focus on the disease area?" To our knowledge, we are the first to present a method generating heatmaps that enable pathologists to extract fine-grained insights about the model in a difficult biomarker prediction task.
Insight generation:
2 Background
2.1 Multiple instance learning (MIL)
MIL formulations. In MIL, a sample is represented by a bag of instances X {x1,.... xx with a bag label y, where X2 E RDD is the k-th instance. The number of instances per bag K may vary across samples. In its standard formulation [ 2 33, the instances of a bag exhibit neither dependency nor ordering among each other. It is further assumed that binary instance labels yıs E {0. 1) exist but are not necessarily known. The binary bag label is 1 if and only if at least one instance label is 1, i.e. max, {ys}; }. Various extensions have been proposed [4] 42) each making different assumptions about the relationships between instances and bag labels.
MIL models. MIL. architectures typically consist of three components as illustrated in Figure a backbone extracting instance representations, an aggregation function fusing the instance repre- sentations into a bag representation, and a prediction head inferring the final bag prediction. As recent foundation models for histopathology have become powerful feature extractors suitable for wide range of tasks (29
2
45 (4). the weights of the backbone are often frozen, allowing for a more efficient training. For aggregation, earlier works used parameter-fi mean or max pooling Recently, attention mechanisms could improve performance, flexibly ex- tracting relevant instance-level information using non-lineat weighting B o 50) and self-attention 41 Attention MIL (AttnMIL) 3 computes weighted average of the instances' feature vectors via a single attention head. TransMIL. 4 uses a custom two-layer Transformer architecture, viewing instance representations as tokens. The bag representation is extracted from the class token at the final layer. TransMIL allows for computing arbitrary pairwise interactions between all instances relevant to the prediction task. While various extensions of AttnMIL and TransMIL have been proposed (c.g.. 52 53
43
44
approaches
48 49
10] IT I2)), these two methods are arguably prototypical and among the most commonly used in the digital histopathology community.
5 6 8 9
MIL explanation methods. From the few studies investigating MIL interpretability. most of them use attention heatmaps have been explored for specific architectures and applications
28 29]. Moreover, basic gradient- and propagation -losee based methods
[54 (5)) Sadafi et al. (55) applied LRP to generate pixel-level attributions for single-cell images in a blood cancer diagnosis task, but did not consider its potential for instance-level explanations. Perturbation-
methods, building on model-agnostic approaches like SHAP (56), perturb bag instances and compute importance scores
3

---

from the resulting change in the model prediction; Early et al. [4] proposed passing bags of single instances through the model ("single"), dropping single instances from bags ("one-removed"), and sampling coalitions of instances to be removed ("MILLI"), Javed et al. [33] introduced "additive MII." providing directly interpretable instance scores while constraining the model's ability to capture instance interactions.
2.2 Limitations of MIL in histopathology
Histopathological datasets and prediction tasks are diverse and come with various inherent challenges. We highlight the following three features.
Instance ambiguity. Instances are small high-resolution patches from large images. Their individ- ual information content may be limited, as they can be subject to noise or only be interpretable as part of larger structure. For example, is not always possible to distinguish benign high-grade adenoma from malignant adenocarcinoma
patch level due to their similar morphology.
Positive, negative, and class-wise evidence. A single bag may contain evidence for multiple classes that MIL model needs weigh for correct decision-making. survival prediction, for example, strong immune response may support longer survival, while an aggressive tumor pattern speaks for shorter survival.
Instance interactions. In many prediction tasks, it may be necessary consider interactions between instances. A gene mutation may generate morphological alterations in the tumor area, the tumor microenvironment, and the healthy tissue, all of which may need to be considered together to reliably predict the biomarker.
Existing MIL formulations make explicit assumptions about the relationship between instances and bag labels 44), limiting their ability to capture the full complexity of a histopathologica prediction task. The standard MIL formulation, in particular. does not consider any of the aforementioned aspects, rendering it unsuitable framework for most histopathologica. settings.
Similarly, previous MIL explanation methods suffer from various shortcomings that limit their ap- plicability in real-worid histopathology datasets. The direct interpretability of attention scores is insufficient faithfully reflect the model predictions (30 31 32). Moreover, they cannot distinguish between positive, negative, or class-wise evidence [33]. Purely gradient- explanations may suf- fer from shattered gradients, resulting in unreliable explanations [57]. Perturbation-base approaches come with high computational complexity. While the linear "single" and "one removed" methods require K forward passes per bag, MILLI scales quadratically with the number of instances (34). In histopathology. where bags typically contain more than 1,000 and frequently more than 10,000 instances, quadratic runtime practically infeasible. Additive MIL and linear perturbation-base methods do not consider higher-order instance interactions. In prediction tasks depending on interac- tions, linear perturbation- baaed explanations may fail to provide faithful explanations, while additive models may not achieve competitive performances.
3 Methods
Notation. We denote vectors with boldface lowercase letters (e.g., x), scalars with lowercase letters (e.g., a), and sets with uppercase letters (e.g., X).
3.1 XMIL: An XAI-based framework for multiple instance learning
We address the limitations discussed in Section 2.2 and introduce a more general formulation of
MIL: explainable multiple instance learning (xMIL). At its core, we propose moving away from the notion of instance labels towards context-aware evidence scores, which better reflect the intricacies of histopathology while laying the foundation for developing and evaluating MIL explanation methods.
Definition 3.1 (Fxplainable multiple instance learning). Let X (xi...., xx} } be a bag of instances with bag label y E TR.
(1) There exists an aggregation function A that maps the bag to its label, i.e., A(X) y. We make no assumptions about the relationship among the instances or between the instances and the label y. (2) There exists evidence function & assigning evidence score E(X, y, xx) 6 E R to any

---

A
Predction
rqu side
hood
Agarogatol
real vecior
Nocel
Fretura Estraction
Appregation
Attention Mosule
B
Iratence striuution SOONNN
IrAoroo RiSE copeeretoo hestrcp
0.01
Figure 2: The two steps of xMIL: estimating the aggregation function (A) and the evidence function (B). Panel A shows a block diagram of a MIL model applied to a histopathology slide. The feature extraction module is typically combination of frozen foundation model followed by shallow MLP. In most of the recent MIL models, the aggregation module uses attention mechanisms for combining the instance feature vectors into single feature representation per bag. The prediction head is linear layer or an MLP. Panel B schematically shows xMIL-LRP for explaining AttnMIL. In XMIL.-LRP, the model output is backpropagated relevance flow. Red and blue colors encode the positive and negative values. The attention module handled via the AH-rule as described in Section 3.2 discussed in Section 33 the instance explanation scores can be computed at the output of the foundation model or at the input level.
the input instances. The colored lines represent the
instance X₃ in the bag, quantifying the impact the instance has on the bag label y-
The aim of xMIL is to estimate (i) the aggregation function A and (ii) the evidence function E.
Definition 3.2 (Properties of the evidence function). Let X X be instances from a bag X. We assume that & has the following properties.
(1) Context sensitivity. The evidence score of instance X may depend on other instances from X.
(2) Positive and negative evidence. If ex >> 0, the instance X has positive impact on the bag label y.
e < O, then X has a negative impact on y. If c O, then x is irrelevant to y.
then instance X₃ has a higher positive impact on y than xv. 0 2 ex > Eg, then instance X has a higher negative impact on than X
(3) Ordering. If Ek Ew'
Similar to our definition, previous works described context sensitivity and accounting for positive and negative evidence as desirable properties of MII. explanation methods 33 34 However, xMII integrates these principles directly into the formalization of the MIL. problem.
In contrast to previous MIL formulations, XMIL addresses the potential complexities within histopathological prediction tasks by refraining from posing strict assumptions on A. Via the evidence function ε, we suggest that instances may vary in their ability to support or refute class and that their influence may depend on the context within the bag. In practice, the evidence function is often unknown, as the notion of an "impact" on the bag label is hard to quantify. For the standard MIL setting, however, the binary instance labels fulfill the criteria of the evidence function. Therefore, XMII. is more general and realistic formulation of multiple instance learning for histopathology.
We learn the aggregation function A via training MIL model. To gain deeper insights into the prediction task by estimating the evidence function E, we design an explanation method for the learned aggregation function with characteristics suitable to the properties of the evidence function.
5

---

3.2 XMIL-LRP: Estimating the evidence function
We introduce xMIL-LRP as an efficient solution to xMIL, bringing layer-wise relevance propagation (LRP) to MIL. LRP is a well-established XAI method [35] [58] with a large body of literature supporting its performance in explaining various types of architectures in different tasks 22 36 59
60 61 62 Starting from the prediction score of selected class, the LRP attribution of neuron i in layer / receives incoming messages from neurons j from subsequent layer + 1, resulting in
44 (1+1) with ęı being the contribution of neuron i of layer to relevance scores r = -E₁ E. 3
relevance rj
(++)] the contribution 4sj in specific model layers. For the attention mechanism, many MIL architectures, we employ the AH-rule introduced by Ali et al. [32]. In general attention mechanism, let 2₁ Iztald be the embedding vector of the k-th token and Pkj the attention score between tokens & and j. The output vector of the attention module is y, = Ex Pkjzz. The AH-rule of LRP treats attention scores If R(yja) is the relevance of the d-th dimension of y, Wyall the AH-rule computes the relevance of the d-th feature of 2; as:
A variety of so-called "propagation rules" have been proposed [36] to specify core component of
constant weighting matrix during the backpropagation pass of LRP.
R(=kd) Σ 3
2kdPls R(yaa).
Σi FidPij
This formulation can be directly applied to AttnMIL,, and also adapted to QKV attention block in transformer, where 2 the embedding associated with the value representation.
We illustrate the effect of this rule in AttnMIL in Figure D. The relevance flow separates the instances weighted by the attention mechanism into positive, negative, and neutral instances, resulting in more descriptive heatmaps that better show the relevant tissue regions compared to attention scores.
We further implement the LRP-c rule for linear layers followed by ReLU activation function [58), as well as the LN-rule to address the break of conservation in layer norm [32], with details presented in Appendix A.2
At the instance-level. xMIL-LRP assigns each instance X₃ [ækd]a IR. a relevance vector r; = |raala with F'dd = R(xxa) = "kd being the relevance score of the d-th feature of xą. We define the instance-wise relevance score as an estimate for the evidence score of the instance as E La rkd.
3.3 Properties of xMIL-LRP and other explanation methods
The properties of XMIL-LRP are particularly suitable for estimating the evidence function:
Context sensitivity: xMIL-LRP disentangles instance interactions and contextual information as it jointly considers the relevance flow across the whole bag. LRP and Gradient x Input (GxI) are rooted in a deep Taylor decomposition of the model prediction [63] and consequently capture dependencies between features by tracing relevance flow through the components of the MIL. model. While attention is aawtee is limited to considering dependencies of features at a specific layer. The "single" method unaware of context. "One-removed" and additive MIL can only capture the impact of individual instances on the prediction.
Positive and negative evidence: xMIL-LRP relevance scores real-valued and can identify whether an instance supports or refutes the model prediction. Features irrelevant to the prediction will receive an explanation score close to zero. Therefore, the range of explanation scores matches the range of the assumed evidence function. The same holds for additive MIL, MILLI, and "one-removed". Attention and "single" do not distinguish between positive and negative evidence.
Conservation: Following the conservation principle LRP, xMIL-LRP provides an instance-wist decomposition of the model output, i.e., Σ é Σk.d Tla y. This instance-level conservation also holds for additive MIL, but not for the other discussed methods. The local conservation principle of L.R (36 further allows us to analyze attribution scores at the instance feature vector level without requiring propagation through the foundation model the instance-wiss attribution scores are the same at any layer of the model.
In some XAI research papers, uppercase letters, e.g., R are used for denoting relevance values.

---

4 Experiments and results
Baseline methods. We compared several explanation methods our xMIL-LRP (see Appendix A.1 for details). For AttnMII. and TransMil,, we selected Gradient x Input (GxI) 64 65 and Integrated gradients (IG) [66 as gradient-baser baselines. We further included the "single" perturbation method (single) [34, which involves using predictions for individual instances as explanation scores. Single the only computationally feasible perturbation-
approach for the bag sizes considered here (up to 24,000). We evaluated raw attention scores for AttnML and attention rollout [67] for TransMIL (attn). In the random baseline (rand), instance scores were randomly sampled from standard normal distribution. For additive attention MIL (AddMIL) [33], assessed raw attention scores (attn) and the model-intrinsic instance-wise predictions (logits).
4.1 Toy experiments
We designed novel toy experiments to assess and compare the characteristies of xMIL-LRP and the baseline methods for AttnMII., TransMIl., and AddMIL in controlled settings. We focused on evaluating to what extent the explanations account for context sensitivity and positive and negutive evidence, i.e., the first two characteristics of the evidence function according to Definition 3.2 which we consider crucial aspects for explaining real-world histopathology prediction tasks.
Inspired by previous works 3 34],
we sampled bags of MNIST images (68), with each instance representing a number between and 9, We defined three MIL tasks for these bags:
4-Bags: The bag label is class if is in the bag, class 2 if 9 is in the bag, class if 8 and are in the bag, and class 0 otherwise. The dataset was proposed by Early et al. [34. In this setting, the model needs learn basic instance interactions.
We define 4, as positive and 5, 7, 9 as negative numbers. The bag label is class 1 if the amount of unique positive numbers is strictly greater than that of unique negative numbers, and class O otherwise. The model needs to adequately weigh positive and negative evidence make correct predictions.
Pos-Neg:
Adjacent Pairs: The bag label is class if it contains any pair of consecutive numbers between and 4, i..., (0,1), (1,2), (2.3) (3,4), and class 0 otherwise. In this case, the impact of instance is contextual, as it depends on the presence or absence of adjacent numbers.
To assess the explanation quality, we first defined valid evidence scores as ground truths according to Definition 3.2. For each dataset, we require one evidence function per predicted class c, denoted by gld) (X, y, xx) c
(c) We assigned ( if x2 supports class c, { -1 if the instance refutes
class and & O if it is irrelevant. We aimed to measure whether an explanation method correctly distinguishes instances with positive, neutral, and negative evidence scores. Therefore, we computed a two-class averaged area under the precision-recall curve (AUPRC-2), measuring if the positive instances received the highest and the negative instances the lowest explanation scores. We assessed AttnMIL and Mll models and repeated each experiment 30 times. The details of the ground truth, the evaluation metric, and the experimental setup are provided in Appendix A.3
Table (dibllasss the test AUROC scores of the three models across datasets, demonstrating that the models solve the tasks to varying degrees, alongside the performances of the explanation methods. We find that XMIL-LRP outperformed the other explanation approaches across MIL models and datasets in all but one setting. It reached particularly high AUPRC-2 scores in the 4-Bags and Pos-Neg datasets while being most robust in the more difficult Adjacent Pairs setting. Attention severely suffered from the presence of positive and negative evidence, which it cannot distinguish by design. While IG performed comparably to xMIL-LRP for AttnMIL models, it was inferior for TransMIL. Notably, the test AUROC of AddMIL was worse all settings, resulting in explanations that are not competitive with the post-hoc explanation methods on AttMMII. and TlansMIL. This supports our point that AddMII. may not perform competitively in difficult prediction tasks. The single perturbation method provided good explanations in the Pos-Neg setting, where numbers have a fixed evidence score irrespective of the other instances in the bag. However, in 4-Bags and Adjacent Pairs, the method's performance decreased, as it always assigns the same score to the same instance regardless of the bag context. In contrast, XMIL-LRP is both context-sensitive and identifies positive and negative instances. Since we expect that these aspects are common features of many
7

---

Table I: Results of the toy experiments, We report AUPRC-2 scores of MIL explanation methods on three toy datasets measuring how well a method identified instances with positive and negative evidence scores (mean + std. over 30 repetitions), The highest mean scores are bold and the second highest are underlined. We also display the model performances ("Test AUROC", mean + std.).
4 4-Bags
Pos- Pos-Neg
Adjucent Pairs
ArtaMLL
TransMIL. 1.00 + 0,00
AddMIL 0.98 2 0.02
AtteMIII 0.97 = 0.00
VIT 0.98 = 0.00
AddMIL 0.89 = 0.03
AttMII 0.88 + 0.08
TransMIL 0,92 + 0.06
AddMIL 0.77
Test AUROC
1,00 + 0.00
Rand
0,31 + 0.00
0.31 + 0.00
0.42 0.00
0.42 # 0.00
0.54 + 0.00
0,54 + 0.00
Altn
0.53 a: 0.00
0.52 + 0.00
0.54 a 0.01
0.45 = 0.00
0.46 = 0.03
0.48 = 0.02
0.61 0.01
0.60 +: 0.01
0.63 a
Single
0.87 + 0:02
0.85 0.07
0.89 0.00
0.91 0.02
0.73 0.06
0.77 0.06
Logits
0.79 0.11
0.68 = 0.17
0.71 0.09
Gx1
0.72
0.40 0.07
0.72
0.44 0.06
0.63 0.05
0.57 0.05
IG
0.88 +: 0.01
0,80 +: 0.09
0.93 a 0.00
0.82 a 0.08
0.75 0.03
0.72 +: 0.07
xMII-I.RP
0.91
0.90 0.02
0.91
0.92 0.91
0.81
real-world histopathological datasets, we conclude that our method is the only suitable approach for such complex settings.
4.2 Histopathology experiments
Datasets and model training. To evaluate the performance of explanations real-world histopathol- ogy prediction tasks, we considered four diverse datasets of increasing task difficulty covering tumor detection, disease subtyping, and biomarker prediction. These datasets had previously been used for
benchmarking in multiple studies [12 33 46 69)
CAMELYON16 70 consists of 400 sentinel lymph node slides, of which 160 carry to-be- recognized metastatic lesions of different sizes. is a well-established tumor detection dataset.
The TCGA NSCLC dataset (abbreviated as NSCLC) contains 529 slides with lung adenocarcinoma (LUAD) and 512 with lung squamous cell carcinoma (LUSC). The prediction task is to distinguish these two non-small cell lung cancer (NSCLC) subtypes.
The TCGA HNSC HPV dataset [22 (abbreviated HNSC HPV) has 433 slides of head and neck squamous cell carcinoma (HNSC). 43 of them were affected by a human papillomavirus (HPV) infection diagnosed via additional testing
(7T). HPV infection is an essential biomarker guiding prognosis and treatment [12]. The task is to identify the HPV status directly from the slides. Label imbalances and the complexity of the predictive signature are key challenges in this task.
The TCGA LUAD TP53 (abbreviated as LUAD TP53) dataset contains 529 lung adenocareinoma (LUUDD) slides, 263 of which exhibit a mutation of the TP53 gene, which is one of the most common mutations across cancers. In lung cancer, it is associated with poorer prognosis and resistance to chemotherapy and radiation (72. Previous works showed that TP53 mutation can be
predicted from LUAD slides 669 73).
We generated patches at 20x magnification and obtained 10,454 + 6,236 patches per slide across all datasets (mean + std.). Features were extracted using the pre-trained CTransPath [43] foundation model and aggregated using AttnMIL or TransMIL Additional details regarding the datasets and training procedure are described in Appendix A.4
We report the mean and standard deviation of the test set AUROC over repetitions in Table 2 In all but one case, TransMIL outperformed AttmMIL, with the largest margin observed in the difficutt TP53 dataset. Our results generally align with performances reported in previous works 12 46 69
Faithfulness evaluation. As the evidence functions E of our histopathology datasets are unknown, we resorted to assessing faihffuless, i.e., how accurately explanation scores reflect the model prediction The primary goal of the faithfulness experiments is to evaluate the ordering of relevance scores (Property 3 of the evidence function in Definition 3.2). Faithfulness can be quantified by progressively excluding instances from the most relevant first (MORF) to the least relevant last and measuring the change in prediction score. The area under the resulting perturbation curve (AUPC)
74 75
"We did not include AddMIL in the real-world experiments, as it is difficult to compare heatmaps from different models without having ground truth like in the toy experiments (Section 4.1) Also notice that faithfulness evaluations are not applicable, since AddMIL explanations are faithful by design [33..
8

---

Table 2: Results of the faithfulness experiments, AUPC values per dataset, MIL model, and expla- nation method (mean std. over all slides). Lower scores indicate higher faithfulness. The best performance per setting (significant minimum based on the paired t-tests) is highlighted in bold. We also display the model performances ("Test AUROC", mean std. over 5 repetitions).
AttaIII.
TransMil.
CAMELYON 0.93 0.00
NSCLC
HNSC HPV 0.88 0.06
LUAD TP53 0.71
CAMELYONIG = 0.01
NSCLC
HNSC HPV
LUAD TP53 0.75
AUROC
0.00
0.05
Rand
0.13
0.97 0.04
0.84 0.14
0.95
0.98 0.08
1.00 0.01
0.14
Alta
0.65 0.46
0.94 0.27
0.65 0.14
0.63 0.45
0.91 0.22
0.95 : 0.15
0.38
Single
0.61 0.43
0.42
0.78 +
0.34 0.16
0.22 0.35
0.53 # 0.26
0.92 #
0.73
GxI
0.19
0.81 0.35
0.82
0.79
0.46
IG
0.62 a: 0.44
0.75 0.38
0.78 a: 0.25
0.38 a: 0.20
0.88 = 0.23
0.99 a: 0.01
1.00 a 0.00
0.99 2 0.01
0.22
0.30
0.26
0.23
indicates how faithfully the identified ordering of the instances affects the model prediction. The lower the AUPC score, the more faithful the method. We calculated AUPC for correctly classified slides. Further methodological details are provided in Appendix [A.5]
In Figure 3 we show the perturbation curves and AUPC boxplots for the patch-dropping experiment for MIL in our four datasets (Figure shows the results for AttmMIL). Additionally, we summarize our results in Table 2 To test the difference in the AUPC values among the baseline explanation methods, we performed paired t-tests between the random baseline vs. all methods and XMIL-LRP vs. all other baselines. The p-values were corrected using the Bonferroni method for multiple comparison correction. All tests resulted in significant differences except for random baseline vs. GxI for CAMELYON16 and attention for HNSC HPV.
XMIL-LRP significantly achieved the lowest average AUPC compared to the baselines, providing the most faithful explanations across all tasks and model architectures. Especially evident with the TransMIL model, xMIL-LRP accurately decomposed the mixing of patch information via self- attention. Notably, the largest margin of x xMIL-LRP to other methods could be observed in the more challenging biomarker prediction tasks of the HNSC HPV and LUAD TP53 datasets.
The results also reflect whether the explanation scores contain meaningful positive/negative evidence for the target class (Property of the evidence function in Definition 3.2 : if so. expect the model's prediction to flip when all patches supporting the target class are excluded. In Figure 3 the model decision always flips when patches excluded based on xMIL-LRP scores, whereas other methods show inconsistent results.
Attention scores, the most widely used explanation approach for MIL in histopathology, did not provide faithful explanations outside the simple tumor detection setting in the CAMELYONI6 dataset. This remarkably highlights their limited usefulness as model explanations and confirms previously reported results in other domains [30 31 32). Passing single instances through the model ("single") achieved good faithfulness scores for simpler tasks and AttnMIL, but performed worse for Transformer-basee biomarker prediction.
5 Extracting insights from xMIL-LRP heatmaps
The identification of predictive features for HPV infection in head and neck carcinoma from histopathological slides is a challenging task for pathologists. In this task, there are partially known morphological patterns associated with the class label. We provide a brief overview of the known his- tological features differentiating HPV-negative and HPV-positive HNSC in Appendix AA.|and and Figure 5 In the following, we demonstrate how faithful xMIL-LRPP explanations can support pathologists in gaining insights about the model strategy and inferring relevaelevant features.
We extracted explanation scores for the best pesfoorrfon TransMIL models. To increase the readability of resulting heatmaps, we clipped the scores per slide at the whiskers of their boxplots, which extended 1.5 times the interquartile range from the first and third quartiles. We then translated them into zero-centered red-blue color map, with red indicating positive and blue negative scores. Notice that the explanation methods operate on different scales. For xMIL-LRP, a positive relevance score indicates support for the explained label, while negative score contradicts it.

---

CAMELYON16
NSCLC
HNBC HPV
LUAD TP53
Target bless probabity
1.0
0.8
0.6
LEP
Siggle
Single
0.2
0.0
1and
Farm
0
40
80
100
0
20
40
60
80
100
20
40
80
100
20
40
60
100
dropped
dropped
dropped
dropped
|RP (3)) sind All (6 Ran
LRP GM sind AuP
Rand
LAP
gindg A
Rand
URP
G ginl AlP (6 Ranlt
Figure 3: Patch dropping results for TransMIL. The first row depicts the perturbation curves, where the solid lines are the average perturbation curve and the shaded area is the standard error of the mean at each perturbation step. Each boxplot the second row shows the distribution of AUPC values for all test slides per explanation methods. In each boxplot, the red line marks the median and the red dot marks the mean. Lower perturbation curves and AUPCs represent higher faithfulness.
We revisit the example of the HNSC tumor with false-positive prediction of HPV infection in Figure [ As previously noted, only XMIL-LRP indicates that the model recognizes evidence of HPV infection in the tumor border, but not the remaining tumor. Despite prediction score close to all relevance scores from the single method were between 0.95-0.97. suggesting that context-free single-instance bags may not be informative in this task. We observed this phenomenon across various slides.
Heatmaps of additional examples are provided in Appendix A.7 In Figure 0 xMIL-LRP accurately delineates and distinguishes HPV-positive tumor islands from the surrounding stroma. In this simple case, attention also provides reasonable explanation. Figure p another correctly classified HPV-positive sample. Here, XMIL-LRP outlines spatially consistent slide regions with clear positive evidence, distinct from regions of negative mixed evidence (top row). Most notably, the subepithelial mucous glands (bottom row), which are not associated with HPV, are correctly highlighted blue, unlike the attention map. In Figure g we display false positive slide. In this case. xMIL-LRP allowed us to identify that the evidence of HPV-positivity can be attributed to an unusual morphology of an HPV-negative tumor that shares some morphological features usually associated with HPV infection (e.g., smaller tumor cells with hyperchromatic nuclei, dense lymphocyte infiltrates).
Conclusion
We introduced xMIL, more general and realistic MIL framework for histopathology. formalizing requirements for MIL explanations via the evidence function. We adapted LRP to MIL as xMIL-LRP, experimentally demonstrated its advantages over previous explanation approaches, and showed how access to faithful explanations can enable pathologists to extract insights from a biomarker prediction model. Thus, xMIL is step toward increasing the reliability of clinical ML systems and driving medical knowledge discovery, particularly in histopathology. Despite being motivated by the challenges in histopathology, our approach presented here can be directly transferred other problem settings that require explaining complex MII. models, e.g., in video, audio, or text domains. Furthermore, a detailed analysis of potentially complex dependencies between instances, especially in the context of multi-moda. inputs, represents a promising direction for future research.
10

---

Acknowledgements
The results shown here are in whole or part based upon data generated by the TCGA Research Network: http://wwwwwwa.e.ggwo/oeooo
This work was in part supported by the German Ministry for Education and Research (BMBF) under Grants
01GQ1115, 01GQ0850, 01IS18025A, 031L0207D, 011S18037A, and BIFOLD24B and by the Institute of Information & Communications Technology Planning & Evaluation (ITTP) grants funded by the Korea government (MSIT) (No. 2019- 0-00079, Artificial Intelligence Graduate School Program, Korea University and No. 2022-0-00984, Development of Artificial Intelligence Technology for Personalized Plug-and-Play Explanation and Verification of Explanation).
References
[1] Thomas G. Dietterich, Richard H. Lathrop, and Tomás Lozano-Pérez. Solving the multiple instance problem with axis-parallel rectangles. Artificial Intelligence, 89(1):31-71 1997.
[2] Oded Maron and Tomás Lozano-Pérez. A framework for multiple-instance learning. Advances in Neural Information Processing Systems, 10, 1997.
[3] Maximilian Ilse, Jakub Tomezak, and Max Welling. Attention-based deep multiple instance learning. In International Conference on Machine Learning, pages 2127-2136. PMLR, 2018.
[4] Zhuchen Shao, Hao Bian, Yang Chen, Yifeng Wang. Jian Zhang, Xiangyang Ji, et al. Transmil: Transformer based correlated multiple instance learning for whole slide image classification. Advances in Neural Information Processing Systems, 34:2136-2147, 2021.
[5] Bin Li, Yin Li, and Kevin W Eliceiri. Dual-stream multiple instance learning network for whole slide image classification with self-supervised contrastive learning. Proceedings of the IEEE/CVF' Conference on Computer Vision and Pattern Recognition, pages 14318-14328, 2021.
[6] Ming Y Lu, Drew FK Williamson, Tiffany Y Chen, Richard J Chen, Mattco Barbieri, and Faisal Mahmood. Data-efficient and weakly supervised computational pathology whole-slide images. Nature Biomedical Engineering, 5(6):555-570) 2021.
[7] Yash Sharma, Aman Shrivastava, Lubaina Ehsan, Christopher A Moskaluk, Sana Syed, and Donald Brown. Cluster-to-conquer: A framework for end-to-end multi-instance learning for whole slide image classification. In Medical Imaging with Deep Learning, pages 682-698. PMLR, 2021.
[8] Jiawei Yang, Hanbo Chen, Yu Zhao, Fan Yang, Yao Zhang, Lei He, and Jianhua Yao. Remix: A general and efficient framework for multiple instance learning based whole slide image classification. In International Conference on Medical Image Computing and Computer-Assiie -sssisted Intervention, pages 35-45. Springer, 2022.
[9] Hongrun Zhang, Yanda Meng. Yitian Zhao, Yihong Qiao, Xiaoyun Yang, Sarah E Coupland, and Yalin Zheng. Difd-mil: Double-tier tier feature distillation multiple instance learning for histopathology whole slide image classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18802-1881 18812, 2022.
[10] Leo Fillioux, Joseph Boyd, Maria Vakalopoulou, Paul-Henry Cournède, and Stergios Christodoulidis. Structured state space models for multiple instance learning in digital pathology. In International Conference on Medical Image Computing and Computer-Assitted Intervention. pages 594-604. Springer, 2023.
[11] Olga Fourkioti, Matt De Vries, and Chris Bakal. Camil: Context-aware multiple instance learning for cancer detection and subtyping in whole slide images. In The Twelfih International Conference on Learning Representations, 2024.
[12] Mohsin Bilal, Robert Jewsbury, Ruoyu Wang, Hammam AIGhamdi, Amina Asif, Mark Eastwood, and Nasir Rajpoot. An aggregation of aggregation methods in computational pathology. Medical Image Analysis, 2023.

---

[13] Gabriele Campanella, Matthew G Hanna, Luke Geneslaw, Allen Miraflor, Vitor Werneck Krauss Silva, Klaus J Busam, Edi Brogi, Victor Reuter, David S Klimstra, and Thomas Fuchs. Clinical-grade computational pathology using weakly supervised deep learning on whole slide images. Nature Medicine, 25(8):1301-130) 2019.
[14] Jakob Nikolas Kather, Lara R Heij, Heike Grabsch, Chiara Loeffler, Amelie Echle, Hannah So- phie Muti, Jeremias Krause, Jan M Nichues, Kai AJ Sommer, Peter Bankhead, et al. Pan-cancer image-based detection of clinically actionable genetic alterations. Nature Cancer, 1(8):789-799, 2020.
[15] Amelie Echle, Niklas Timon Rindtorff, Titus Josef Brinker, Tom Luedde, Alexander Thomas Pearson, and Jakob Nikolas Kather. Deep learning in cancer pathology: a new generation of clinical biomarkers. British Journal of Cancer, 124(4)6686-696, 2021.
[16] Salim Arslan, Julian Schmidt, Cher Bass, Debapriya Mehrotra, Andre Geraldes, Shikha Singhal, Julius Hense, Xiusi Li, Pandu Raharja-Liu, Oscar Maiques, et al. A systematic pan-cancer study on deep learning-based prediction of multi-omic biomarkers from routine pathology images. Communications Medicine, 4(1):48, 2024.
[17] Charlie Saillard, Benoit Schmauch, Oumeima Laifa, Matahi Moarii, Sylvain Toldo, Mikhail Zaslavskiy, Elodie Pronier, Alexis Laurent, Giuliana Amaddeo, Hélène Regnault, et al. Predict- ing survival after hepatocellular carcinoma resection using deep learning on histological slides. Hepatology, 72(6):2000-200 2013, 2020.
[18] Ole-Johan Skrede, Sepp De Raedt, Andreas Kleppe, Tarjei S Hveem, Knut Liestøl, John Maddison, Hanne A Askautrud, Manohar Pradhan, John Arne Nesheim, Fritz Albregtsen, et al. Deep learning for prediction of colorectal cancer outcome: a discovery and validation study, The Lancet, 395(102211:350-366 2020.
[19] Sebastian Lapuschkin, Stephan Wäldchen, Alexander Binder, Grégoire Montavon, Wojeiech Samek, and Klaus-Robert Müller. Unmasking clever hans predictors and assessing what machines really learn. Nature Communications, 10(1):1096, 2019.
[20] Frederick M Howard, James Dolezal, Sara Kochanny, Jefree Schulte, Heather Chen, Lara Heij, Dezheng Huo, Rita Nanda, Olufunmilayo I Olopade, Jakob N Kather, et al. The impact of site-specific digital histology signatures on deep learning model accuracy and bias. Nature Communications, 12(1):4423, 2021.
[21] Miriam Hägele, Philipp Seegerer, Sebastian Lapuschkin, Michael Bockmayr, Wojeiech Samek, Frederick Klauschen, Klaus- Robert Müller, and Alexander Binder. Resolving challenges in deep learning-based analyses of histopathological images using explanation methods. Scientific Reports, 10(1):6423) 2020.
[22] Frederick Klauschen, Jonas Dippel, Philipp Keyl, Philipp Jurmeister, Michael Bockmayr, Andreas Mock, Oliver Buchstab, Maximilian Alber, Lukas Ruff, Grégoire Montavon, and Rlbes-Robert Müller. Toward explainable artificial intelligence for precision pathology. Annual Review of Puthology: Mechanisms of Disease, 19:541-5700 2024.
[23] Alexander Binder, Michael Bockmayr, Miriam Hägele, Stephan Wienert, Daniel Heim, Katha- rina Hellweg, Masaru Ishii, Albrecht Stenzinger, Andreas C. Hocke, Carsten Denkert, Klaus- Robert Müller, and Frederick Klauschen. Morphological and molecular breast cancer profiling through explainable machine learning. Nature Machine Intelligence, 3:355 366, 2021.
[24] Leonille Schweizer, Philipp Seegerer, Hee-yeong Kim, René Saitenmacher, Amos Muench, Liane Barnick, Anja Osterioh, Carsten Dittmayer, Ruben Jödicke, Debora Pehl, et al. Analysing cerebrospinal fluid with explainable deep learning: From diagnosties to insights. Neuropathol- ogy and Applied Neurobiology, 49( 1) 1866,, 2023.
[25] Kristof T Schütt, Farhad Arbabzadah, Stefan Chmiela, HoburR Müller, and Alexandre Tkatchenko. Quantum-chemtnn chemical insights from deep tensor neural networks. Nature Communi- cations, 8(1): 831):13890, 2017.
12

---

[26] John A Keith, Valentin Aussilev-Galinde Bingqing Cheng, Stefan Chmiela, Michael Gastegger, Klaus-Robert Müller, and Alexandre Tkatchenko. Combining machine learning and computa- tional chemistry for predictive insights into chemical systems. Chemical Reviews, (21(16)9816- 9872, 2021.
[27] Jana Lipkova, Tiffany Y Chen, Ming Y Lu, Richard Chen, Maha Shady, Mane Williams, Jingwen Wang. Zahra Noor, Richard N Mitchell, Mehmet Turan, et al. Deep learning-enabled assessment of cardiac allograft rejection from endomyocardial biopsies. Nature Medicine, 28(3):575-582, 2022.
[28] Ming Y Lu, Tiffany Y Chen, Drew FK Williamson, Melissa Zhao, Maha Shady, Jana Lipkova, and Faisal Mahmood, Ai-based pathology predicts origins for cancers of unknown primary. Nature, 394(7861):100-111 2021.
[29] Sophia Wagner, Daniel Reisenbüchler, Nicholas West, Jan Moritz Niehues, Jiefu Zhu, Sebastian Foersch, Gregory Patrick Veldhuizen, Philip Quirke, Heike I Grabsch, Piet A van den Brandt, et al. Transformer-based biomarker prediction from colorectal cancer histology: A large-scale multicentric study. Cancer Cell, 41()): 1650-1661, 2023.
[30] Sarah Wiegreffe and Yuval Pinter. Attention is not not explanation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference Natural Language Processing (EMNLP-IJCNLP), pages 11-20, Hong Kong, China, 2019. Association for Computational Linguistics.
[31] Sarthak Jain and Byron C. Wallace. Attention is not Explanation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguis- fics: Human Language Technologies, pages 3543-3556, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.
[32] Ameen Ali, Thomas Schnake, Oliver Eberle, Grégoire Montavon, Klaus-Robers Müller, and Lior Wolf. Xai for transformers: Better explanations through conservative propagation. In International Conference on Machine Learning, pages 435-451. PMLR, 2022.
[33] Syed Ashar Javed, Dinkar Juyal, Harshith Padigela, Amaro Taylor-Weiner, Limin Yu, and Aa- ditya Prakash. Additive mil: Intrinsically interpretable multiple instance learning for pathology. Advances in Neural Information Processing Systems, 35:20689 207022, 2022.
[34] Joseph Early, Christine Evers, and Sarvapali Ramchurn. Model agnostic interpretability for multiple instance learning. In International Conference on Learning Representations, 2022.
[35] Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Klaus-Robe Robert Müller, and Wojeiech Samek. On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PLoS ONE, 10(7), 07 2015.
[36] Grégoire Montavon, Alexander Binder, Sebastian Lapuschkin, Wojeiech Samek, and Klaus- Robert Müller. Layer-wise relevance propagation: an overview. Explainable Al: interpreting, explaining and visualizing deep learning, pages 193-209, 2019.
[37] Richard J Chen, Ming Lu, Drew FK Williamson, Tiffany Y Chen, Jana Lipkova, Zahra Noor, Muhammad Shaban, Maha Shady, Mane Williams, Bumjin Joo, et al. Pan-cancer integrative histology-genonil analysis via multimodal deep learning. Cancer Cell, 40(8):865-878, 2022.
[38] Julien Calderaro, Narmin Ghaffari Laleh, Qinghe Zeng, Pascale Maille, Loetitia Favre, Anaïs Pujals, Christophe Klein, Céline Bazille, Lara R Heij, Arnaud Uguen, al. Deep learning-based phenotyping reclassifies combined catptc//elllllcccolmmiccccoooooo Nature Communications, 14(1):8290, 2023.
[39] Andrew H Song, Mane Williams, Drew FK Williamson, Sarah SL Chow, Guillaume Jaume, Gan Gao, Andrew Zhang, Bowen Chen, Alexander S Baras, Robert Serafin, et al. Analysis of 3d pathology samples using weakly supervised ai. Cell, 187(10) 2024.
13

---

[40] Omar SM EI Nahhas, Chiara ML Loeffler, Zunamys Carrero, Marko van Treeck, Fiona R Kolbinger, Katherine Hewitt, Hannah S Muti, Mara Graziani. Qinghe Zeng, Julien Calderaro, et al. Regression-basse based deep-learning predicts molecular biomarkers from pathology slides. Nature Communications, 15(1):1253, 2024.
[41] Mare-André Carbonneau, Veronika Cheplygina, Eric Granger, and Ghyslain Gagnon. Multiple instance learning: A survey of problem characteristics and applications. Pattern Recognition, 77:329-353, 2018.
[42] James Foulds and Eibe Frank. A review of multi-instance learning assumptions. The Knowledge Engineering Review, 25(1): 2010.
[43] Xiyue Wang, Sen Yang. Jun Zhang, Minghui Wang, Jing Zhang, Wei Yang, Junzhou Huang, and Xiao Han. Transformer-based unsupervised contrastive learning for histopathological image classification. Medical Image Analysis. 81, 2022.
[44] Jonas Dippel, Barbara Feulner, Tobias Winterhoff, Simon Schallenberg. Gabriel Dernbach, Andreas Kunft, Stephan Tietz, Philipp Jurmeister, David Horst, Lukas Ruff, et al. Rudolfv: A foundation model by pathologists for pathologists. arXiv preprint arXiv:2401.0400 2024.
[45] Eugene Vorontsov, Alican Bozkurt, Adam Casson, George Shaikovski, Michal Zelechowski, Kristen Severson, Eric Zimmermann, James Hall, Neil Tenenholtz, Nicolo Fusi, et al. A foundation model for clinical-grade computational pathology and rare cancers detection. Nature Medicine, pages 1-12, 2024.
[46] Richard J Chen, Tong Ding, Ming Y Lu, Drew FK Williamson, Guillaume Jaume, Andrew H Song, Bowen Chen, Andrew Zhang, Daniel Shao, Muhammad Shaban, et al. Towards a general- purpose foundation model for computational pathology. Nature Medicine, 30(3):550-862. 2024.
[47] Pedro O. Pinheiro and Ronan Collobert. From image-level to pixel-level labeling with con- volutional networks. 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1713-1721, 2015.
[48] Ji Feng and Zhi-Hua Zhou. Deep miml network. In Proceedings of the Phirty-First AAAI Conference on Artificial Intelligence, AAAI 17, page 1884-1890. AAAI Press, 2017.
[49] Wentao Zhu, Qi Lou, Yeeleng Scott Vang, and Xiaohui Xie. Deep multi-instance networks with sparse label assignment for whole mammogram classification. In Medical Image Computing and Computer Assisted Intervention- MICCAI 2017: 20th International Conference, Quebec City, QC, Canada, September 11-13, 2017, Proceedings, Part III 20, pages 603-611. Springer, 2017.
[50] Nikolaos Pappas and Andrei Popescu-Belis. Explicit document modeling through weighted multiple-instance learning. Journal of Artificial Intelligence Research, 58:591-626 2017.
[51] Dawid Rymarczyk, Adriana Borowa, Jacek Tabor, and Bartosz Zieliński. Kernel self-attention for weakly- supervised image classification using deep multiple instance learning. In 2021 1 IEEE Winter Conference on Applications of Computer Vision (WACV), pages 1720-1722 2021.
[52] Dawid Rymarczyk, Adam Pardyl, Jaroslaw Kraus, Aneta Kaczyńska, Marek Skomorowski, and Bartosz Zieliński. Protomil: Multiple instance learning with prototypical parts for whole-slide image classification. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pages 421 421-436. Springer, 2022.
[53] Lukasz Struski, Dawid Rymarczyk, Arkadiusz Lewicki, Robert Sabiniewicz, Jacek Tabor, and Bartosz Zielinski. Promil: Probabilistic multiple instance learning for medical imaging. In ECAI 2023, pages 2210-2217. IOS Press, 2023.
[54] Antoine Pirovano, Hippolyte Heuberger, Sylvain Berlemont, Saïd Ladjal, and Isabelle Bloch. Improving interpretability for computer-aided diagnosis tools whole slide imaging with multiple instance learning and gradient- based explanations. Interpretable and Annotation- Efficient Learning for Medical Image Computing: Third International Workshop, iMIMIC 2020,
14

---

Second International Workshop, MIL3ID 2020, and 5th International Workshop. LABELS 2020, Held Conjunction with MICCAI 2020, Lima, Peru, October 4-8, 2020, Proceedings 3, pages 43-53. Springer, 2020.
[55] Ario Sadañi, Oleksandra Adonkina, Ashkan Khakzar, Peter Lienemann, Rudolf Matthias Hehr, Daniel Rueckert, Nassir Navab, and Carsten Marr. Pixel-level explanation multiple instance learning models in biomedical single cell images. In International Conference Information Processing Medical Imaging, pages 170-182. Springer, 2023.
[56] Scott M Lundberg and Su-In Lee. A unified approach to interpreting model predictions. Advances in Neural Information Processing Systems, 30, 2017.
[57] David Balduzzi, Marcus Frean, Lennox Leary, J. P. Lewis, Kurt Wan-Duo Ma, and Brian McWilliams. The shattered gradients problem: resnets are the answer, then what is the question? In ICML, volume 70 of Proceedings of Machine Learning Research, pages 342-350. PMLR, 2017.
[58] Grégoire Montavon, Wojciech Samek, and Klaus-Robers Müller. Methods for interpreting and understanding deep neural networks. Digital Signal Processing, 73:1-15, 2018.
[59] Leila Arras, José Arjona- Medina, Michael Widrich, Grégoire Montavon, Michael Gillhofer, Klaus- Müller, Sepp Hochreiter, and Wojciech Samek. Explaining and interpreting Istms. Explainable Al: Interpreting. Explaining and Visualizing Deep Learning, pages 211-233 2019.
[60] Oliver Eberle, Jochen Büttner, Florian Kräutli,
Müller, Matteo Valleriani, and Grégoire Montavon. Building and interpreting deep similarity models. IEEE Transactions on Pattem Analysis and Machine Intelligence, 44(3):1 1991119966111 2020.
[61] Thomas Schnake, Oliver Eberle, Jonas Lederer, Shinichi Nakajima, Kristof T Schütt, Klaus- Robert Müller, and Grégoire Montavon. Higher-order explanations of graph neural networks via relevant walks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(11):7581- 7596, 2021.
[62] Simon Letzgus, Patrick Wagner, Jonas Lederer, Wojciech Samek, Klaus-Robert Müller, and Grégoire Montavon. Toward explainable artificial intelligence for regression models: method- ological perspective. IEEE Signal Processing Magazine, (9(4)400-5, 2022.
[63] Grégoire Montavon, Sehastian Bach, Alexander Binder, Wojciech Samek, and Klaus-Robert Müller. Explaining nonlinear classification decisions with deep taylor decomposition. Pattern Recognition, 65:211-222, 2017.
[64] David Baehrens, Timon Schroeter, Stefan Harmeling, Motoaki Kawanabe, Katja Hansen, and Klaus-Robert Müller. How to explain individual classification decisions. The Journal of Machine Learning Research, 11:1803-1831, 2010.
[65] Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Learning important features through propagating activation differences. In Proceedings of the 34th International Conference on Machine Learning Volume 70, ICML' 17, page 3 3145-3153, 2017.
[66] Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In International Conference on Machine Learning. pages 3319-3328, PMLR, 2017.
[67] Samira Abnar and Willem Zuidema. Quantifying attention flow in transformers. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault, editors, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4190 4190-4197, Online, July 2020. Association for Computational Linguistics.
[68] Li Deng. The mnist database of handwritten digit images for machine learning research [best of the web]. IEEE Signal Processing Magazine, 1916)141 2012.
[69] Yanan Wang, Nicolas Coudray, Yun Zhao, Fuyi Li, Changyuan Hu, Yao-Zhong Zhang, Seiya Imoto, Aristotelis Tsirigos, Geoffrey Webb, Roger J Daly, et al. Heal: an automated deep learning framework for cancer histopathology image analysis. Bioinformatics, 37(22)44291- 4295, 2021.
15

---

[70] Babak Ehteshami Bejnordi, Mitko Veta, Paul Johannes Van Diest, Bram Van Ginneken, Nico Karssemeijer. Geert Litjens, Jeroen AWM Van Der Laak, Meyke Hermsen, Quirine F Manson, Maschenka Balkenhol, et al. Diagnostic assessment of deep learning algorithms for detection of lymph node metastases in women with breast cancer. JAMA, 318(22):2119-222) 2017.
[71] Joshua D Campbell, Christina Yau, Reanne Bowlby, Yuexin Liu, Kevin Brennan, Huihui Fan, Alison M Taylor, Chen Wang, Vonn Walter, Rehan Akbani, et al. Genomic, pathway network, and immunologic features distinguishing squamous carcinomas. Cell Reports, 23(1):194-212 2018.
[72] Akira Mogi, Hiroyuki Kuwano, et al. Tp53 mutations in nonsmall cell lung cancer. BioMed Research International, 2011, 1, 2011.
[73] Nicolas Coudray, Paolo Santiago Ocampo, Theodore Sakellaropoulos, Navneet Narula, Matija Snuderl, David Fenyö, Andre L Moreira, Narges Razavian, and Aristotelis Tsirigos. Classifica- tion and mutation prediction from non comll cell lung cancer histopathology images using deep learning. Nature Medicine, 25(10):1559-135 2018.
[74] Wojeiech Samek, Alexander Binder, Grégoire Montavon, Sebastian Lapuschkin, and Klaus- Robert Müller. Evaluating the visualization of what a deep neural network has learned. IEEE Transactions on Neural Networks and Learning Systems, 28(11):2660-25 2673, 2016.
[75] Stefan Bluecher, Johanna Vielhaben, and Nils Strodthoff. Decoupling pixel flipping and occlusion strategy for consistent XAI benchmarks. Transactions Machine Learning Research, 2024.
[76] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017
[77] Narine Kokhlikyan, Vivek Miglani, Miguel Martin, Edward Wang, Bilal Alsallakh, Jonathan Reynolds, Alexander Melnikov, Natalia Kliushkina, Carlos Araya, Siqi Yan, et al. Captum: A unified and generic model interpretability library for pytorch. arXiv preprint arXiv:2009.07896, 2020.
[78] Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. Model-agnostic interpretability of machine learning. arXiv preprint arXiv: 2016.
[79] Torch Vision maintainers and contributors. Torchvision: Pytorch's computer vision library. 2016.
https://gi http::://githb.... www./ooottct/viiieeeeoeo
[80] Ethan Cerami, Jianjiong Gao, Ugur Dogrusoz, Benjamin E Gross, Selcuk Onur Sumer, Bü- lent Arman Aksoy, Anders Jacobsen, Caitlin J Byrne, Michael L Heuer, Erik Larsson, et al. The cbio cancer genomics portal: an open platform for exploring multidimensional cancer genomics data. Cancer Discovery. 2(5):401 -044, 2012.
[81] Jianjiong Gao, Bülent Arman Aksoy, Ugur Dogrusoz, Gideon Dresdner, Benjamin Gross, S Onur Sumer, Yichao Sun, Anders Jacobsen, Rileen Sinha, Erik Larsson, et al. Integrative analysis of complex cancer genomics and clinical profiles using the cbioportal. Science Signaling, 6(269):p11-p11, 2013.
[82] Ino de Bruijn, Ritika Kundra, Brooke Mastrogiacomo, Thinh Ngoc Tran, Luke Sikina, Tali Mazor, Xiang Li, Angelica Ochoa, Gaofei Zhao, Bryan Lai, al. Analysis and visualization of longitudinal genomic and clinical data from the aacr project genie biopharma collaborative in cbioportal. Cancer Research, 83(23):3861-386) 2023.
[83] Nobuyuki Otsu al. A threshold selection method from gray- histograms, Automatica, 22(285-29):23-27, 1975.
16

---

A Appendix
A.1 Baseline MIL explanation methods
Attention maps Attention scores have commonly been used as an explanation of the model by considering attention heatmaps. assuming that they reflect the importance of input features (76).
In AttnMIL, bag representation is computed as an attention-weighted average of instance-level representations, i.e.,
K Σ a& (xx),
((X)
a₁ softmax (w" (tanh(V sigm sigm(Uf(xx)))
where fxxx) is an instance and g(X) the bag representation. The attention scores ag assign to each patch an attribution with S a S 1, and have been used as instance-wise explanation scores B].
In TransMIL, the attention heads deliver self-attention vectors A R(K+1)x(K+3) for each head and Transformer layer I, recalling that the first token is the class token. Mean pooling is often used for fusing the self-attention matrices of different heads, i..., A (A),), The attention scores from the class token to the instance tokens can be used as attribution scores, i.e., (1, Alternatively, attention rollout has been proposed to summarize the self-attention matrices over layers [67]. For a model with L Transformer layers, attention rollout combines {A [1)) as Á I Á where À' 0.5A' + 0.5I, with being the identity matrix. Then, similar to the layer-wise attention scores, the heatmap is defined the attention rollout of the class token to the instances, i.c., A(12)))
Gradient-based methods Gradient-based methods utilize gradient information derive feature relevance scores. Pirovano et el. [54] combined raw gradients to identify the most relevant features and derive tiles that activate these features the most.
Various other gradient-based methods have been proposed in the XAI literature, including saliency maps and Gradient x Input (GxI) 66 66] and Integrated Gradients (IG) [66]. These methods can easily be adapted to compute explanations in MIL. We obtain the gradient of a MIL model prediction ý with respect
patch (xx))
For GxI, we can then define the relevance score of the k-th instance as Da Vi(xx))a=Aa with 2nd being the d-th feature of Xk-
Integrated gradients (IG) (66) computes the gradients of the model's output with respect to the input, integrated over a path from bascline to the actual input. The baseline is typically set to zero, and so we do. The explanation score the k-th instance is computed as (Z(a)),, where the relevance score of the d-th feature of the k-th instance IG(Iaa) is computed as
IG(xka) = Ikdd f
f(aX) Oxad
da,
where of is the model and X is the K x feature matrix the bag (with K being the number of instances and D being the number of features for each instance). We used the implementation of IG available in Captum [7 with the internal batch size set to the number of instances in a hag.
Perturbation- methods The idea of perturbation-base explanation methods is to perturb selected instances of bag and derive importance scores from the resulting change in the model prediction. It builds model-agnostic post-hoc local interpretability methods like LIME (78) and
SHAP [56]
Early et al. 34 proposed and evaluated multiple perturbation-based methods of different complexity. The "single" method passes bags of single patches X₁ = (xx) for k 1,..., through the model and uses the outcome f(Xx) as explanation score. "One removed" drops single patches, i.c. constructs bags X X for & = 1, l,... K and defines the difference to the original prediction score f(X) f(Xx) as explanation. The "combined" approach takes the mean of these two scores. As these methods cannot account for patch interactions, Early al. also propose an algorithm to sample coalitions of patches to be perturbed, called MILLI. They show that MILLI outperforms the
17

---

baselines on toy datasets when instance interactions need to be considered. The complexity of MILLI O(nK2), where n is the number of coalitions and the bag size.
Additive MIL The idea additive MIL 33 is to make the MIL model inherently interpretable by designing the bag-level prediction be sum of individual instance predictions. Let function be feature extractor and v/m, U/ MLPs. In many cases, particularly for Attention MIL [3] MIL model p can be written as
((X) = Uppp
Σ a (xx)
with 0g softmaxx (1/m (X)),
where ag is the attention score of instance k. For Attention MIL, 1/ is defined as the inner part of the softmax function of Equation 2 additive model, the authors suggest to instead compute
and 1hp prediction head outputting class logits. To obtain an
p(X)
Vp [ax (xx)).
(5)
This way, the bag prediction becomes the sum of the individual instance predictions 1.p (arf (xx)). which can be used as instance explanation scores. These instance logits are proportional to the Shapley values of the instances [33] In our experiments, consider the proposed additive variant of Attention MIL (AddMIL).
A.2 Layer-wise Relevance Propagation (LRP)
LRP is a method for explaining neural network predictions by redistributing the output's relevance back through the network the input features. The redistribution follows relevance conservation principle, where the total relevance of each layer is preserved as propagates backward. If r
(
denotes the relevance of neuron j in layer 1, conservation means that Σj ( = E, rt any two layers 4 and 12. As a general principle, LRP posits
(12) holds for
Σ 923
Σ
(++)
(6)
Σx
with dt being the contribution of neuron of layer I relevance r ((+1), There are "propagation rules" for various layer types [36] 588 that specify Qú for different setups.
Feed forward neural network. The following generic rule holds for propagating relevance through
linear layers followed by ReLU [36]:
asp(way) 1 = = + En ag p(wp))
(++)
where aj is the activation of neuron j in layer I, Way the weight from neuron i of layer to neuron of layer? + 1, c a stabilizing term to prevent numerical instabilities, and p(w:)) a modification the weights of the linear layer. For example, if p(w;z) = Waj ymax(w.j. 0), then Equation 7 is called LRP-y rule. For 7 = 0, this equation is called LRP- rule.
LayerNorm. Assume 2 is the embedding of the k-th token and yı = LayerNorm(za) as:
2 E(z}
(8)
std +
where E(z} and std(z} are the expected values and standard deviation of the tokens.
For propagating relevance through LayerNorm, Ali et al. (32) suggested the LN-rule as the following:
Zkdl 1
R(zka) = Σ E, Zid 2ia(d13 (81
R(yaa),
N/
11, ifk=j and 2kd is the d-th dimension of z and R(zra) is the relevance
where ōzj 0, otherwise assigned to it. In practice, LN-rule is implemented by detaching std(z) and handling it as a constant.
18

---

A.3 Toy experiments: Training and evaluation details
Evidence functions. We define the evidence functions for the three datasets as follows. We write X2 ~ n to indicate that instance k represents MNIST number n.
In the 4-Bags dataset, 8 supports classes and but refutes classes and 2, while supports (e) classes 2 and 3 but refutes classes O and 1. Hence, for X ~ 8, we define Ek = 1 for e E {1,3} (c) for E {0, 2). For Xx 9, we set f for c E (2,3) and & = -1 for {0. 1). In all other cases, E O.
and '/
In Pos-Neg, 4, 6, and 8 instances support class and refute class O, and vice versa for 5, 7, 9. (1) Hence, we set (1) and ( - X ~ (4, 8), c -1 and 6 1 xf {5,7,9}. and (/) 0 otherwise.
(0)
In Adjacent Pairs, 4 supports class 1 and refutes class 0 if 3 is also present, but is irrelevant otherwise. That is, for X ~ 4, set c otherwise. The evidence scores for the other numbers are defined accordingly.
(1) 1 and ( 1f 3 is also in in the bag, and 4 0
Evaluation metric. We aim to measure whether explanation method correctly distinguishes instances with positive, neutral, and negative evidence scores. We separate this into two steps: quantify the separation between positive and non non-positive instances, and quantify the separation between negative and non-negative instances. Let ( = [c]
(c)
K be the evidence scores for
(c) some bag X (xI,.... xK} and class c, and s(c) [ ..... C be the class-wise explanation
(c) scores from some explanation method. We define epos min(els) 0) and eney = min( (ee()) 0) as
the binarized positive and negative evidence, and compute
1 (AUPRC(e/C) (c) s(cl + AUPRC (c) -s()))
AUPRC-2 -
(10)
We utilize the area under the precision-recall curve (AUPRC) to account for potential imbalances. Our AUPRC-2 metric can interpreted as the one-vs-all AUPRC score for detecting positive and negative instances. It becomes 1 if all instances with positive negative evidence have been assigned the highest / lowest evidence scores. For each dataset and explanation method, we computed the AUPRC-2 across all classes and test bags and report the average score.
Experimental details. Instead of training end-to-end MIL models, we obtained feature vectors with 512 dimensions for each MNIST image via a ResNet18 model pre- pre-trained on Imagenet from the Torch Vision library [79]. For each bag, we first sampled subset of numbers, where each number was selected with probability of 0.5, and then randomly drew 30 MNIST feature vectors from this subset. We used 2,000 bags for training, 500 for validation, and 1,000 for testing. We trained AttmMLL and TransMIL. models with a learning rate of 0.0001 for maximum of 1000 and 200 epochs for AttMMII. and TransMII,, respectively. We finally selected the model with the lowest validation loss. We repeated each model training 30 times and report means and standard deviations across repetitions. Each experiment with its repetitions was run on single CPUs in less than 24 hours, respectively. We used the same setting for training AddMIL, but with an Adam optimizer as in the original paper (33)
A.4 Histopathology experiments: Data and training details
We downloaded TCGA HNSC, LUAD, and LUSC datasets from TCGA website. The HPV status of HNSC dataset and the TP53 mutations of LUAD dataset were downloaded from cBioPortal
Dataset details.
80
82
We applied the following splits.
CAMELYON16: We used the pre-defined test set of 130 slides, and randomly split the remaining slides into 230 for training and 40 for validation.
NSCLC: As in previous works 4
33
we randomly split the slides into 60% training, 15%
validation, and 25% test data.
HNSC HPV: Due to the low number of HPV-positive samples, we uniformly split the dataset into three cross-validation folds like in previous work (12).
LUAD TP53: We randomly split the slides into 60% training, 15% validation, and 25% test data.
19

---

Preprocessing details. We extracted patches from the slides of 256 x 256 pixels without overlap at 20x magnification (0.5 microns per pixel). We identified and excluded background patches via Otsu's method [3 on slide thumbnails and applied patch-level minimum standard deviation of 8.
Training details. For training, we sampled hags of 2048 patches per slide and passed their feature representations through the MIL model. For validation and testing, we simultaneously exposed all patches of slide the model to avoid sampling biases and statistical instabilities. Due the computational complexity of TransMIL, we excluded slides with more than 24,000 patches (~2 6% of all slides). We did this for all methods to ensure fair comparisons. AttnML models were trained for up to 1,000 epochs with batch size 32, and the TransMIL models for up to 200 epochs with batch size We selected the checkpoint with the highest validation AUC. In the HNSC HPV dataset, we used one fold as validation and test fold and two folds as training folds and repeated this procedure for all possible assignments of folds. We applied a grid search over learning rates and dropout schemes and selected the syper-parameter settings with the highest mean validation AUCs over 5 repetitions. For Att, we found that the best configuration was always a learning rate of 0.002 and no dropout. For TransMIL, we ended up with learning rate of 0.0002 and high dropout (0.2 after the feature extractor, 0.5 after the self-attention blocks and before the final classification layer) for CAMELYONI6 and NSCLC, and learning rate of 0.002 without dropout for HNSC HPV and LUAD TP53. The training was done on an A100 80GB GPU.
A.5
Faithfulness evaluation: Patch flipping
K Given a slide X = {x+] k== and heatmaping function H producing the explanation scores of the instances in X, i.e., H(xx) = ?R. we binned the patches of slide X into 100 ordered groups (Ei.... 1100), where E, is the set of all patches whose attribution scores are between the (100 -)- and (100- - i + 1)-th percentiles of the explanation scores the instances of X, for example, Ei is the set of the most relevant 1% patches of X and 1o is the least relevant 1% of the patches.
Patch dropping. Following the region perturbation strategy introduced in 744 we progressively excluded the most relevant regions from slide X. i.c. in the n-th iteration, the most relevant n%% of patches were excluded. Formally, the perturbation procedure can be formulated as the following:
X (-(0) mort
X
x morr
P(X, n)
(1I)
P(X,n)
U
E.
im++1
where P(X, n)) is a perturbation function that excludes the most relevant n of patches from slide
X. Note that at step 100 all the patches excluded and therefore, x x(100) 0. where 0 is an empty set, for which we pass an array of zeros to the model.
Comparing heatmaps. We define the quantity of interest for the comparison of different heatmaps as the area under the perturbation curve (AUPC) as the following:
101
AUPC(X, H)
Σ
f(x(m) )
(12)
100
7=0
where f is the model's 's function.
That Heatmap Hi is more faithful than H2 if AUPC(X,HH Hi) < AUPC(X, H2). That is, the lower AUPC, the more faithful the explanation method.
We ran all the AttMII.. experiments on an A100 40GB GPU and the TransMll. experiments on single CPU.
A.6 Faithfulness evaluation: Additional results
We present the patch dropping results for AttmMI in Figure
20

---

CAMELYON16
NSCLC
HNSC HPV
LUAD TP53
Target class probabity
1.0
0.8
0.4
K21
Rird
20
40
80
100
40 dropped
100
0
20
80
20
40
80
100
drapped
dropped
dropped
1.0
0.8
0.6
0.4
0.2
0.0
LRP
gind2pp0 AtP Rand
URP Gril gindl pdin
Rand
LRP
singl ABP
Rand
LRP
gindl AM Rand
Figure 4: Patch dropping results for AttnMIL. The first row depicts the perturbation curves, where the solid lines are the average perturbation curve and the shaded area is the standard error of the mean at each perturbation step. Each boxplot the second row shows the distribution of AUPC values for all test slides per explanation methods. In each boxplot, the red line marks the median and the red dot marks the mean. Lower perturbation curves and AUPCs represent higher faithfulness.
A.7 Extracting insights from xMIL-LRP heatmaps: Additional results
Figure 5 summariize and compares the appearance (pathologists call this morphology or histological features) HPV-negative and HPV-positive HNSCC. HPV-positive HNSCC generally do produce keratin (non-keratinizing morphology) and unlike HPV-negative do not origin from the surface epithelium but crypt epithelium of palatine and lingual tonsils. HPV-positive tumor nests are often embedded in lymphocyte rich parts of the tissue (lymphoid stroma) and the tumor nuclei show darker, denser staining (i.e. hyperchromatic nuclei).
HPV-negative
HPV-positive
From TomiToDA.aao TODA- BAJA8D8
From TOGA BA-0059
Figure 5: Exemplary histological features HPV-negative and -positive HNSC.
We display further exemplary heatmaps of TransMIL model predictions in the HNSC HPV dataset in
Figures 7 and g
21

---

Slide
xMIL-LRP
Attention
Single
Figure 6: Heatmaps from different explanation methods for a TransMIL model predicting HPV-status. The model correctly predicted the slide HPV-positive (prediction score: 0.9215). For xMIL-LRP, red indicates evidence for and blue against the HPV-positive class.
Single
Figure 7: Heatmaps from different explanation methods for TransMIL model predicting HPV-status. The model correctly predicted the slide HPV-positive (prediction score: 0.9048). For xMIL-LRP, red indicates evidence for and blue against the HPV-positive class.
Slide
XMIL-LRP
Attention
Figure Heatmaps from different explanation methods for TransMIL model predicting Hta-sta The slide is HeV-negative, but the model predicted HPV-positive (prediction score: 0.9997). For XMIL-LRP, red indicates evidence for and blue against the HPV-negative class.
22

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The contributions, claims, and the scope the paper are made clear in the abstract and introduction (cf. Section T
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The limitations of the framework and previous works are extensively discussed
in the paper, especially in Section 2.2 Additionally, we list further limitations in discussions.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "I "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
23

---

Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: All codes are publicly accessible on GitHub. The experiment pipelines are clearly described in Sections [.1 and 4.2 of the manuscript. Further description of data processing pipelines is given in the main paper in Section 4.2 and in the supplemental material in Section A.4
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g.. a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with open-source dataset instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

---

5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [No]
Justification: Our code for reproducing results and implementation of the used methods is made publicly accessible. Datasets used in this work are public datasets properly described and cited in Sections 4.2 and A.4
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//nips.cc/ pttipss.ccpuublicguudeeee
for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All the details in this regard are provided in Sections 4.2 and A.4
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: our experimental results Section 4.2 we used paired t-tests with multiple comparison corrections for comparing the methods, as well as the standard error of the mean to plot the error bars on the perturbation curves of Figures and
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
25

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables plots. The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The compute resources are mentioned in Sections A.3 and A.4
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: (1) our data are from public datasets that have followed the guidelines of their corresponding ethics committee. (2) we declare no known potential harmful consequence of our submitted work regarding the mentioned points in the code of ethics. (3) if applicable, we implemented all the impact mitigation measures mentioned in the code of ethics, such as disclosing essential elements for reproducibility.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
We anticipate no negative societal impact for our work.
26

---

Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considcrations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning conrectly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We aware of no potential risk for any misuse of our work.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual- use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: (1) We cited the open datasets properly in Section 4.2 (2) if some parts of other repositories have been used in codes, we have cited the repository properly. These repositories have MIT license or GPLv3 license for non-commercial academic use.
Guidelines:
existing assets. The authors should cite the original paper that produced the code package dataset.
The answer NA means that the paper does
The authors should state which version of the asset used and, possible, include URL.
27

---

The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, pacpessssppssiiiipp.. thcode com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: do not release any new assets, except for our codes, which are made publicly accessible alongside documentation.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and sereenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing. The human data are acquired from public datasets, properly cited in the manuscript (please see points 5, 9, 12 of this checklist.).
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Please see point 14.
Guidelines:
28

---

The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
29


Reviews:
[{"review_id": "saPwdHNHcv", "reviewer": "Reviewer_utz5", "summary": "The paper introduces xMIL-LRP, an efficient solution for explainable Multiple Instance Learning (MIL) by incorporating layer-wise relevance propagation (LRP) into MIL. The authors further employ the AH-rule to define the \"propagation rules\" for relevance. Experiments are conducted on three toy datasets and four real-world Whole Slide Image (WSI) prediction tasks, resulting in improved faithfulness scores for the explanations.", "questions": "Please refer to weaknesses.\n\nOne more question is that considering that xMIL-LRP potentially provides more explainable results, does it affect the prediction performance of the original MIL model? How do the authors balance between model explainability and prediction accuracy?", "limitations": "N/A", "rating": 3, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The experimental setup on both toy datasets and real-world WSI tasks is robust, demonstrating the effectiveness of the proposed method.\n\n2. The paper is logically structured, with well-illustrated figures that aid in understanding the results.\n\n3. The development of explainable MIL methods is crucial for the deployment of these models in real-world applications, where understanding the model's decisions is important.", "weaknesses": "Limited Novelty:\nThe method primarily applies existing explainable AI techniques to the MIL domain, which limits its novelty.\nUnclear Method Description:\n\nUnclear Method Description:\nThe definition of the relevance score $𝑝_{𝑖𝑗}$ as the attention score between neurons $j$ and $i$ is unclear. Attention-based MIL typically defines attention at the instance level, making it confusing how relevance is defined and propagated within this context.\n\n\nAbsence of Fair Comparison with Additive MIL:\nThe performance of additive MIL is directly cited without conducting new experiments under the same conditions (data splits, feature extractors, etc.). This could lead to an unfair comparison and underestimation of additive MIL's performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Limited Novelty\n\nRegarding your concerns about the novelty of applying existing XAI methods to MIL, we would like to clarify that it was not the aim of our work to develop a new explanation method for MIL. Instead, the novelty of our work extends across other dimensions:\n\n1. **Methodical novelty**: Despite attempts to apply XAI to MIL models in histopathology (e.g. [1-10]), there exists no formalism guiding the interpretation of the heatmaps and defining their desired properties. Instead, the heatmaps are often considered a prediction of assumed ground-truth instance labels, which we identified as problematic (Section 2.2). **xMIL is a novel framework addressing this gap**. Unlike previous works, it posits that bags possess a ground truth evidence function that quantifies the instances’ impact on the bag label. **Within xMIL, heatmaps estimate this ground truth, which makes interpreting the heatmaps straightforward and insightful.**\n\n2. **Empirical novelty**: Many high-impact medical publications use attention heatmaps or Integrated Gradients to explain histopathological MIL models [1-8]. To our knowledge, no prior research has systematically evaluated whether these methods produce sensible results. Our extensive empirical evaluation on synthetic and real-world histopathology datasets is the first of its kind. **It reveals that the widely used explanation methods may yield misleading results**, as they often fail to reflect the model decisions faithfully. In contrast, **xMIL-LRP sets a new state-of-the-art for explainability in AttnMIL and TransMIL models in histopathology.**\n\n3. **Novelty in insight generation**: Previous studies [9-10] conducted qualitative assessments of heatmaps on easy-to-learn datasets like CAMELYON or TCGA NSCLC. The insights gained in these settings are limited to model debugging, i.e., “Does the model focus on the disease area?” To our knowledge, we are the first to present a method generating **heatmaps that enable pathologists to extract fine-grained insights about the model in a difficult biomarker prediction task.**\n\n> Unclear Method Description\n\nWe apologize for any potential misunderstanding regarding the description of the attention mechanism used in the considered MIL models. We will clarify our description of the AH rule in lines 173-174.\nIn a general attention mechanism, let $x_k$ be the embedding vector of the k-th token and $p_{ij}$ the attention score between tokens i and j. The output vector is $y_j = \\sum_{i} p_{ij} x_i$. The AH rule of LRP treats attention scores as a constant weighting matrix during the backpropagation pass of LRP, not affecting the forward predictions. If $R(y_{j,d})$ is the relevance of the d-th dimension of $y_j$, the AH rule computes the relevance of the d-th feature of $x_k$ as: $R(x_{k,d})=\\sum_{j} \\frac{x_{k,d} p_{kj}}{\\sum_{i} x_{i,d}p_{ij}} R(y_{j,d})$.This formulation can be directly applied to AttnMIL, and also adapted to a QKV attention block in a transformer, where $x_k$ is the embedding associated with the value representation. A figure will be added to illustrate how AH and gamma LRP propagation rules combine to explain multi-head attention.\n\n> Absence of Fair Comparison with Additive MIL\n\nWe agree that just mentioning the performance from the original paper without testing under the same conditions is unfair. We will remove the respective lines from the manuscript (lines 277-278).\n\nA comparison between post-hoc explanations and Additive MIL (AddMIL) is inherently difficult, as we cannot compare heatmaps from different models without having a ground truth. The faithfulness experiments from Section 4.2 are not applicable to Additive MIL, as its explanations are faithful by design [9]. Furthermore, we would like to highlight that the focus of our manuscript is explaining non-additive models, as they are widely used in the computational histopathology community.\n\nRegardless, we ran additional experiments with AddMIL on the toy datasets, where ground truth evidence is available. We used the Additive Attention MIL implementation (with Adam optimizer) provided by Javed et al. [9], and applied the same training protocol as described in Sections 4.1 and A.2 of our manuscript. The results are presented in Table B.1. The test performance of AddMIL was worse in all settings, resulting in explanations that are not competitive with the post-hoc explanation methods on AttnMIL and TransMIL. This supports our point that AddMIL may not achieve competitive performances in difficult prediction tasks (lines 126-128). We will add these results to the manuscript.\n\n\n> Effect of xMIL-LRP on prediction performance\n\n​​​​xMIL-LRP and all the benchmarked methods in this manuscript are post-hoc explanation methods. Therefore, they do not impact the training process or prediction performance of the original model, as they are applied after the model is fully trained. Thus, balancing between model explainability and prediction accuracy is not a concern in this context. We did not investigate utilizing xMIL-LRP scores to further improve prediction performance, but may consider this in future works.\n\n**Additional comments**\n\nMany thanks for the thoughtful and constructive comments! We hope our responses have effectively addressed your concerns and showcased the novelty of our research. Given your acknowledgment of the relevance and soundness of our work, we kindly hope you might reconsider your rating. If you have any further questions or suggestions, we are happy to continue the discussion.\n\n\n**References**\n\nSee Author Rebuttal.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8uGDMrzBgs", "reviewer": "Reviewer_GyUV", "summary": "The authors propose xMIL-LRP which is an explainable MIL method to overcome the shortcomings of previous instance score-based methods (such as attention-based). This can provide positive/negative evidence towards the label and can be slotted into existing MIL methods, since this is a post-hoc interpretability method.", "questions": "See the section on the weaknesses.", "limitations": "Please see above", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "With most of MIL methods reaching saturation, especially with advent of transformer-based modules and very powerful patch-level foundation models, the performance gain to be had with new MIL method would likely be marginal. Therefore it is encouraging that the authors are proposing a tool for interpretability that can fit into any of these MIL methods (if I didn't misunderstand) - I am assuming this also works with any feature encoders. The LRP-based methods have been utilized many times outside this field, so I think it is positive that the authors brought it into the field with great rigor.", "weaknesses": "There are few weakness/confusion that I would like the authors to address to make this a worthwhile contribution to the field.\n\n**Presentation**: I am assuming the authors are very familiar with LRP, but most of the audience in the computational pathology would not be. Therefore, I found lack of motivation/explanation beyone LRP quite confusing. I think Section 3.2 is too compressed (Especially lines 178-182). The section of faithful evaluation was also not easy to understand as this was entirely new analysis. How do these analyses relate to the fact that predicted evidence scores are positive/negative?\n\n- Can we do away with formalizing *evidence function*? It is a good formalization of the concept, but as soon as it was applied to the actual histology data, the concept was abandoned for the lack of ground truth, which I found it quite awkward. Why not go directly into introducing relevance scores (which do not need evidence function formalism) and allocating more explanation for general computational pathology readers?\n\n- I think explaining more rigorously how LRP would concretely be applied to AB-MIL and TransMIL (with few-step explanations) would be immensely helpful, since these are quite different architecture (multihead-attn vs. not) and authors simply bypass this by explaining the chain rules for \"neurons\". Only then readers can be convinced that LRP is easy to apply to any MIL of their choice.\n\n**Clarification vs. other methods**: My understanding is Additive MIL also provides positive/negative instance-level evidence (positive/negative logits) and can function very similarly as proposed LRP. I think the authors need to provide more explanation/experiments differentiating LRP from Additive MIL. Especially the heatmap ablations should probably include additive MIL results to convince the readers. In addition integrated gradient methods have been applied to explain positive/negative evidence towards bag-level labels [1], [2]. Can authors expand on this?\n\n**Restriction to binary classification task**: My understanding is that the current methodology is restricted to binary classification problem. For multi-class subtyping problem and survival prediction (where ordering of predicted risk is important), how can this be adapted? How would evidence scores add up to bag-level label, if the predicted endpoint is continuous (e.g. survival prediction)? I would like authors to expand on this, since only subset of the problems in pathology can be regarded as binary classification problem.\n\n**Speculation on different feature encoders**: While the explicit head-to-head comparison would not be necessary, I am curious to know how different feature encoders, especially with deluge of patch-level foundation models, affect LRP performance. Can authors share their thoughts on this?\n\n[1] Lee, Yongju, et al. \"Derivation of prognostic contextual histopathological features from whole-slide images of tumours via graph deep learning.\" Nature Biomedical Engineering (2022): 1-15.\n[2] Song, Andrew H., et al. \"Analysis of 3D pathology samples using weakly supervised AI.\" Cell 187.10 (2024): 2502-2520.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Details on LRP\n\nWe acknowledge that some descriptions are too condensed. Therefore, we will add a section in the appendix with text and figures to properly describe LRP. \n\n> Faithfulness experiments\n\nWe will clarify the faithfulness experiments and their motivation in the manuscript, with algorithmic details already provided in Appendix A.4. The primary goal of the faithfulness experiments is to evaluate the ordering of relevance scores (property 3 of evidence function in Definition 3.2). However, the results also reflect whether the explanation scores contain meaningful positive/negative evidence for the target class: if such information is present, we expect the model’s prediction to flip when all patches supporting the target class are excluded. In Figure 4, the model decision always flips when patches are excluded based on xMIL-LRP scores, whereas other methods show inconsistent results.\n\n> Formalization of evidence function\n\nThe evidence function is crucial to xMIL. It serves as the ground truth, while the explanation method estimates it. We posit that each instance contains evidence for or against a bag label. Sometimes, the ground truth is (partly) known, such as through pathologist annotations in disease detection tasks. However, many biological associations remain unknown even to experts. Thus, relevance scores provide inexpensive annotation for each patch, as they estimate the ground truth. We will add more details about this to Sections 3.1 and 4.2.\n\n> Applying LRP to AttnMIL and TransMIL\n\nWe will allocate a section in the appendix for LRP details. \nIn a general attention mechanism, let $x_k$ be the embedding vector of the k-th token and $p_{ij}$ the attention score between tokens i and j. The output vector is $y_j = \\sum_{i} p_{ij} x_i$. The AH rule of LRP treats attention scores as a constant weighting matrix during the backpropagation pass of LRP, not affecting the forward predictions. If $R(y_{j,d})$ is the relevance of the d-th dimension of $y_j$, the AH rule computes the relevance of the d-th feature of $x_k$ as: $R(x_{k,d})=\\sum_{j} \\frac{x_{k,d} p_{kj}}{\\sum_{i} x_{i,d}p_{ij}} R(y_{j,d})$.This formulation can be directly applied to AttnMIL, and also adapted to a QKV attention block in a transformer, where $x_k$ is the embedding associated with the value representation. A figure will be added to illustrate how AH and gamma LRP propagation rules combine to explain multi-head attention.\n\n> Additive MIL and integrated gradients (IG)\n\nAdditive MIL (AddMIL) provides positive and negative class-wise evidence. However, AddMIL cannot model arbitrary instance interactions, which we consider crucial in histopathological prediction tasks (lines 109-112). Comparing post-hoc explanations and AddMIL is challenging due to the lack of ground truth for the heatmaps. The faithfulness experiments do not apply to AddMIL, as its explanations are faithful by design [9]. Furthermore, our manuscript focuses on explaining non-additive models, as they are widely used in the computational histopathology community (see e.g. [1-8]). \n\nWe ran additional experiments with AddMIL on the toy datasets, where ground truth evidence is available. We used the Additive Attention MIL implementation (with Adam optimizer) provided by Javed et al. [9], and the same training protocol in Sections 4.1 and A.2 of our paper. The results presented in Table B.1 indicate that AddMIL performed worse in all settings, making its explanations not competitive with the post-hoc explanation methods on AttnMIL and TransMIL. This supports our point that AddMIL may not be competitive in difficult prediction tasks (lines 126-128).\n\nIG has been used to interpret MIL models in histopathology. We implemented IG, and our faithfulness evaluations (Table B.2) show that IG explanation scores are less faithful to model decisions compared to xMIL-LRP, especially for TransMIL. This underscores the importance of our work in rigorously assessing explanation methods for MIL models in histopathology. We will add these additional results to the manuscript.\n\n\n> Multi-class classification\n\nxMIL-LRP is applicable to multi-class classification. All evaluated explanation methods in the paper, except attention, yield one heatmap per class. For example, the 4-Bags dataset in the toy experiments is a 4-class classification problem (Table 1 of the paper).\n\n> Regression and survival prediction\n\nWe do not explicitly evaluate regression tasks, but prior work shows how to apply LRP in regression [11], which can be adapted to histopathological MIL tasks. In survival prediction, outcomes are typically not predicted via regression due to data censorship; instead, survival probabilities at event times are computed, resulting in Kaplan-Meier analyses. While attention heatmaps have been used to interpret survival models (e.g. [12]), explaining these models remains challenging since they are neither purely regression nor classification tasks. We are extending our work to survival analysis in an ongoing project and will address this in a future work section of the paper.\n\n> Feature encoders\n\nA major advantage of our method is that propagation through the feature encoder is not required: the patch scores can be computed at the input level of the MIL aggregation. Thus, the feature encoder’s architecture does not affect the explanation quality. Possible differences in model predictions and heatmaps are due to the quality of the feature embedding.\nTo address this, we repeated the histopathology experiments with the more recent UNI foundation model [13], with the same training parameters as for the best models with CTransPath (CTP) encoder. In our results (Table B.3), xMIL-LRP shows a robust performance in faithfulness experiments in comparison to the results with CTP backbone. Interestingly, while “Single” shows high faithfulness for the LUAD TP53, its performance for NSCLC is much worse in comparison to that with CTP.\n\n**References**\n\nSee Author Rebuttal.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JYnQMaYZrj", "reviewer": "Reviewer_DaUT", "summary": "This work proposed an LRP-based extension for MIL setup in histopathology. It focuses on explanation quality, not derivation of novel architectures.", "questions": "At some point, you are relaxing MIL assumptions so much that is is not MIL anymore, especially when the bag is not a set of instances anymore, because the spatial dependence between instances is important. \n\nWhat is the difference between your work and a straightforward application of LRP to MIL problem?", "limitations": "They are discussed.", "rating": 5, "confidence": 5, "soundness": 1, "presentation": 3, "contribution": 1, "strengths": "It is quite well written in terms of language. \n\nImages are nicely depicting the key ideas.", "weaknesses": "This paper does not follow the MIL paradigm. It references only standard MIL assumptions and methods dedicated to that. There is work extending standard MIL methods to other MIL assumptions such as additive MIL [1], presence and threshold-based MIL [2]. and lastly percentage MIL [3]. \n\nThe novelty is extremely limited because I do not see here more than just the application of LRP to MIL models. \n\nMoreover, there are models such as CLAM [4] and ProtoMIL [5] that are addressing MIL in a more interpretable way, how are they working in your setup? Are they following the limitations? \n\n[1] Javed, Syed Ashar, et al. \"Additive mil: Intrinsically interpretable multiple instance learning for pathology.\" Advances in Neural Information Processing Systems 35 (2022): 20689-20702.\n\n[2] Rymarczyk, Dawid, et al. \"Kernel self-attention for weakly-supervised image classification using deep multiple instance learning.\" Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 2021.\n\n[3] Struski, Łukasz, et al. \"ProMIL: Probabilistic multiple instance learning for medical imaging.\" ECAI 2023. IOS Press, 2023. 2210-2217.\n\n[4] Lu, Ming Y., et al. \"Data-efficient and weakly supervised computational pathology on whole-slide images.\" Nature biomedical engineering 5.6 (2021): 555-570.\n\n[5] Rymarczyk, Dawid, et al. \"Protomil: Multiple instance learning with prototypical parts for whole-slide image classification.\" Joint European Conference on Machine Learning and Knowledge Discovery in Databases. Cham: Springer International Publishing, 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Limited Novelty\n\nRegarding your concerns about the novelty of applying existing XAI methods to MIL, we would like to clarify that it was not the aim of our work to develop a new explanation method for MIL. Instead, the novelty of our work extends across other dimensions:\n\n1. **Methodical novelty**: Despite attempts to apply XAI to MIL models in histopathology (e.g. [1-10]), there exists no formalism guiding the interpretation of the heatmaps and defining their desired properties. Instead, the heatmaps are often considered a prediction of assumed ground-truth instance labels, which we identified as problematic (Section 2.2). **xMIL is a novel framework addressing this gap**. Unlike previous works, it posits that bags possess a ground truth evidence function that quantifies the instances’ impact on the bag label. **Within xMIL, heatmaps estimate this ground truth, which makes interpreting the heatmaps straightforward and insightful.**\n\n2. **Empirical novelty**: Many high-impact medical publications use attention heatmaps or Integrated Gradients to explain histopathological MIL models [1-8]. To our knowledge, no prior research has systematically evaluated whether these methods produce sensible results. Our extensive empirical evaluation on synthetic and real-world histopathology datasets is the first of its kind. **It reveals that the widely used explanation methods may yield misleading results**, as they often fail to reflect the model decisions faithfully. In contrast, **xMIL-LRP sets a new state-of-the-art for explainability in AttnMIL and TransMIL models in histopathology.**\n\n3. **Novelty in insight generation**: Previous studies [9-10] conducted qualitative assessments of heatmaps on easy-to-learn datasets like CAMELYON or TCGA NSCLC. The insights gained in these settings are limited to model debugging, i.e., “Does the model focus on the disease area?” To our knowledge, we are the first to present a method generating **heatmaps that enable pathologists to extract fine-grained insights about the model in a difficult biomarker prediction task.**\n\n> “This paper does not follow the MIL paradigm. It references only standard MIL assumptions and methods dedicated to that. [...]”\n\nIt seems to us that this must be a misunderstanding.\n\nWe are aware that there are various formulations and methods beyond standard MIL, and we explicitly reference and discuss many of them (lines 72-73, 86-88, 95-97, 113-115, 125-128), including additive MIL (lines 95-97, 125-128) and the formulation of threshold-based MIL from Foulds & Frank [14] (lines 72-73, 113-115).\n\nIn fact, we investigated various MIL formulations but didn’t find any that we deemed suitable to cover all the complexities possibly present in histopathology tasks like biomarker or outcome prediction (Section 2.2). Concretely, with respect to the works mentioned:\n\n* Additive MIL is unable to learn arbitrary instance interactions (lines 37-41, 125-128).\n* The MIL formulations in threshold-based and percentage MIL do not include negative evidence, e.g., evidence speaking against a class label. They also don’t account for patch interactions.\n* We argue that including negative evidence and instance interactions may be crucial for capturing the complexities of histopathological datasets (lines 105-112).\n\nThis is why we formulated the xMIL framework, introducing a novel perspective on MIL that overcomes those issues.\n \n> “At some point, you are relaxing MIL assumptions so much that it is not MIL anymore [...]”\n\nThanks for your comment. However, we do not agree that Definitions 3.1 and 3.2 of our manuscript do not formulate a MIL problem.\n\nIn Definition 3.1, we use bags of instances with bag-level labels just as standard MIL and most other MIL formulations (lines 67-68, 135-136). The key difference between our formulation and other formulations is that we do not make assumptions about the relationship between the instances and the bag label, but instead posit the existence of an evidence function assigning context-sensitive evidence scores to the individual instances.\n\nNote that we do not make any assumptions about the (spatial) dependence between the instances. Hence, we neither require nor not rule out that spatial dependence plays a role. Thus, our framework is a general MIL framework that can be reduced to existing MIL formulations with extra assumptions. For example, AttnMIL does not model the correlation among instances, whereas TransMIL does.\n\nWe would appreciate it if you could expand on why you think that our xMIL formulation is not MIL anymore. In particular, we would like to understand what exactly you consider requirements for a MIL formulation.\n\n> “Moreover, there are models such as CLAM and ProtoMIL that are addressing MIL in a more interpretable way [...]”\n\nDespite extending AttnMIL, the explanations from CLAM are also based on attention scores. Therefore, the method is subject to the same conceptual limitations, namely, the lack of faithfulness and not being able to distinguish between positive and negative evidence (lines 35-37, 89-90, 118-120, we cited CLAM in reference [6]).\n\nTo our understanding, ProtoMIL is conceptually different from our approach, as it provides explainability in terms of similarity scores to prototypical instances instead of generating heatmaps over all instances of a bag. Within our xMIL framework, we would therefore argue that it does not estimate the evidence function. Hence, it is unclear to us how to quantitatively or qualitatively compare the evaluated explanation methods with ProtoMIL. We acknowledge that we did not cite ProtoMIL in our paper, but we are happy to include it.\n\n**Additional comments**\n\nAs outlined above, we believe some of the comments and questions may have been based on misunderstandings. We have outlined our key contributions and clarified the raised questions. We are happy to continue the discussion.\n\n**References**\n\nSee Author Rebuttal.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
**Overview**

We thank the reviewers for their comments and valuable feedback. We have addressed their questions and concerns and made the following additions and changes to our manuscript:

* More clarity regarding the contributions of the work (reviewers utz5 & DaUT): We discussed our contributions in detail where the reviewers asked about them.
* Comparisons to Additive MIL (all reviewers): We attached experimental comparisons to Additive MIL in the presence of available ground truth (Table B.1). This confirmed that xMIL-LRP outperformed all baselines, including additive MIL, across all three tasks. We also discussed how Additive MIL is conceptually out of the scope of our manuscript.
* Comparison to Integrated Gradients (IG) as an increasingly popular explanation approach in histopathological MIL papers (reviewer GyUV): We added IG  to our faithfulness evaluation (Table B.2). It reached consistently lower faithfulness compared to xMIL-LRP, and performed poorly for TransMIL.
* Impact of the foundation model on the results (reviewer GyUV): We elaborated upon this point and conducted faithfulness experiments using a different histopathological foundation model as an ablation (Table B.3). We did not observe major changes in the results.
* Clarity of method description (reviewers utz5 & GyUV): We included a more precise description of the xMIL-LRP method. We will also release our code on GitHub upon publication, which will provide further clarity on the implementation details. We also expanded on the motivation and idea of the formalization of the evidence function in the xMIL definition. Finally, we appended a clarification on how our work extends to regression and survival prediction.

In summary, we provided clear additional evidence for the soundness of our evaluation and effectiveness of the xMIL framework. We further clarified the novelty of this contribution by formalizing xMIL using evidence functions and providing faithful patch-level relevance scores for models that are not inherently interpretable but widely used. As appreciated during the review, this is crucial for the deployment of MIL models in real-world applications.


**References**

To avoid redundancies, we collect all references from the individual rebuttals here.

[1] Lu et al. “Data-efficient and weakly supervised computational pathology on whole-slide images.” Nature Biomedical Engineering (2021). _(ref. [6] of the original paper)_

[2] Lu et al. "AI-based pathology predicts origins for cancers of unknown primary." Nature (2021). _(ref. [28] of the original paper)_

[3] Lipkova et al. “Deep learning-enabled assessment of cardiac allograft rejection from endomyocardial biopsies.” Nature Medicine (2022). _(ref. [27] of the original paper)_

[4] Chen et al. “Pan-cancer integrative histology-genomic analysis via multimodal deep learning.” Cancer Cell (2022).

[5] Wagner et al. "Transformer-based biomarker prediction from colorectal cancer histology: A large-scale multicentric study." Cancer Cell (2023). _(ref. [29] of the original paper)_

[6] Calderaro et al. “Deep learning-based phenotyping reclassifies combined hepatocellular-cholangiocarcinoma.” Nature Communications (2023).

[7] Song et al. "Analysis of 3D pathology samples using weakly supervised AI." Cell (2024).

[8] El Nahhas et al. “Regression-based Deep-Learning predicts molecular biomarkers from pathology slides.” Nature Communications (2024).

[9] Javed et al. "Additive MIL: Intrinsically interpretable multiple instance learning for pathology." NeurIPS (2022). _(ref. [33] of the original paper)_

[10] Early et al. “Model agnostic interpretability for multiple instance learning.” ICML (2022). _(ref. [34] of the original paper)_

[11] Letzgus et al. "Toward explainable artificial intelligence for regression models: A methodological perspective." IEEE Signal Processing Magazine (2022).

[12] Chen et al. "Multimodal co-attention transformer for survival prediction in gigapixel whole slide images". ICCV (2021).

[13] Chen et al. "Towards a general-purpose foundation model for computational pathology." Nature Medicine (2024). _(ref. [42] of the original paper)_

[14] Foulds & Frank. “A review of multi-instance learning assumptions.” The Knowledge Engineering Review (2010). _(ref. [38] of the original paper)_

Meta-review: The paper introduces xMIL-LRP for multiple instance learning (MIL) from Whole Slide Images (WSI) by aggregating layer-wise relevance propagation (LRP) into MIL framework and accordingly merits an interpretable module for model’s decision process. The proposed xMIL-LRP provides a tool to fit LRP into any MIL framework that can be used in WSI  downstream tasking for aggregation/classification. Overall the paper is well written and easy to follow. The paper has received mixed reviews. There is consensus among reviewers while the idea of xMIL-LRP is limited in terms of technical novelty but it provides a feasible tool to be used in computational pathology field for transformation of any MIL aggregation model into an interpretable framework. Some concerns were raised by reviewers on (a) lack of motivation and explanation behind LRP for adoption in computational pathology, (b) differentiability of LRP from Additive MIL, (c) limitation to only binary classification of WSI, (d) deployment of existing explainable AI methods into MIL which raises a concern on the novelty of the method, (e) fair comparison to with other MIL approaches such as additive MIL and threshold-based MIL, and percentage MIL, (f) does the explainability module affects the prediction performance of the original MIL, (g) comparison to other interpretable MIL methods such as CLAM and ProtoMIL, and (h) utility of different encoders for patch feature embedding. Authors have addressed several concerns during rebuttal, explained on the novelty of their tool and provided further comparative analysis results. The AC  also finds the proposed tool can be very useful for computational pathology practitioners and they finds enough support from the reviewers to merit the paper for publication. It is highly encouraged for the authors to take the advantage on the discussions raised by reviewers as highlighted above for their final revision using from both pre-/post-rebuttal phase comments.
Decision: Accept (poster)