Model Agnostic Interpretability for Multiple Instance Learning
Authors: Joseph Early
Christine Evers
SArvapali Ramchurn
Venue: ICLR 2022 Poster (2022)
Keywords: multiple instance learning
interpretability
model-agnostic

Abstract: In Multiple Instance Learning (MIL), models are trained using bags of instances, where only a single label is provided for each bag. A bag label is often only determined by a handful of key instances within a bag, making it difficult to interpret what information a classifier is using to make decisions. In this work, we establish the key requirements for interpreting MIL models. We then go on to develop several model-agnostic approaches that meet these requirements. Our methods are compared against existing inherently interpretable MIL models on several datasets, and achieve an increase in interpretability accuracy of up to 30%. We also examine the ability of the methods to identify interactions between instances and scale to larger datasets, improving their applicability to real-world problems.

Published as a conference paper at ICLR 2022
MODEL AGNOSTIC INTERPRETABILITY FOR MULTIPLE INSTANCE LEARNING
Joseph Early, Christine Evers & Sarvapali Ramchurn Agents, Interaction and Complexity Group Department of Electronics and Computer Science University of Southampton {J.A. (J.A.Early) Early, C. E.eves, sdr1 sdr1]@sotoon.ac ac.
ABSTRACT
In Multiple Instance Learning (MIL), models are trained using bags of instances, where only a single label is provided for each bag. A bag label is often only determined by handful of key instances within a bag, making it difficult to in- terpret what information classifier is using make decisions. In this work, we establish the key requirements for interpreting MIL models. We then go on to de- velop several model-aggnstic approaches that meet these requirements. Our meth- ods are compared against existing inherently interpretable MIL models on several datasets, and achieve an increase interpretability accuracy of up to 30%. We also examine the ability of the methods to identify interactions between instances and scale to larger datasets, improving their applicability to real-world problems.
INTRODUCTION
In Multiple Instance Learning (MIL), data is organised into bags of instances, and each bag is given single label. This reduces the burden of labelling, as each instance does not have to be assigned label, making it useful in applications where labelling is expensive, such as healthcare (Carbonneau et al., 2018). However, there are often only few key instances within bag that determine the bag label, so is necessary to identify these key instances in order to interpret a model's decision-makkin (Liu et al., 2012). Directly interpreting the decision-making process of machine learning methods is difficult due to the complexity of the models and the scale of the data on which they trained, so there a need for methods that allow insights into the decision- decision-making processes (Gilpin al., 2018).
In MIL problems, only some of the instances in each bag will be discriminatory, i.e., significant number of the instances in bag will be non-discriminatory or even related other bag classes (Amores, 2013). Identifying the important instances and presenting those as the interpretation of model decision- makiing filters out the non- discriiiiinatory information and provides the explainee with a reduced and relevant interpretation, which will avoid overloading the user (Li et al., 2015). Our work provides the following novel contributions:
1. MIL interpretability definition We identify two questions that interpretability methods for MIL. should be able to answer: 1) Which are the key instances for a bag? 2) What outcome (class/value) does each key instance support? In the rest of this work, we will refer to these two questions as the which and the what questions respectively. As we explore in Section 2, existing interpretability methods are able to answer which questions with varying degrees of accuracy, but can only answer what questions under certain assumptions.
2. MIL. model-agnostic interpretability methods Existing interpretability methods are of- ten model-specific, i.e., they can only be applied to certain types of MIL models. To this end, we build upon the state-of-the-art MIL inherent interpretability methods by develop- ing model-agnostic interpretability methods that are up to 30% more more accurate. The model-agnostic interpretability methods that we propose can be applied to any MIL model, and are able answer both which and what questions.
3. MIL. interpretability method comparison We compare existing model-specific methods with model-agnostic methods on several MIL datasets. Our experiments are also carried out on four types of MIL model, giving a comprehensive comparison of interpretability performance. To the best of the authors' knowledge, this is one of the first studies to compare different interpretability methods for MIL.!
Source code for this project available https: sttps://gii thub. com/ JAAaa/JAaT 4/MI LLL.

---

Published as a conference paper at ICLR 2022
The remainder of this work is laid out as follows. Section provides relevant background knowledge and reviews existing MIL interpretability methods. Section 3 outlines the requirements for MIL interpretability and details our approaches that meet these requirements. Next, Section 4 provides our results and experiments. Section 5 discusses our findings, and Section 6 concludes.
BACKGROUND AND RELATED WORK
The standard MIL assumption (SMIL) is a binary problem with positive and negative bags (Di- etterich al., 1997; Maron & Lozano-Pérez, 1998). bag is positive if any of its instances positive, otherwise it is negative. The assumptions of SMIL can be relaxed to allow more gener- alised versions of MII., e-g., through extensions that include additional positive classes (Scott al., 2005; Weidmann et al., 2003). SMIL also assumes there is no interaction between instances. How- ever, recent work has highlighted that modelling relationships between instances is beneficial for performance (Tu et al., 2019: Zhou et al., 2009). Existing methods for interpreting MIL models often rely on the SMIL assumption, so cannot generalise other MIL problems. In addition, exist- ing methods are often model-specific, i.e., they only work for certain types of MIL models, which constrains the choice model (Ribeiro et al., 201 Identifying the key instances in MIL bags is a form of local interpretability (Molnar, 2020), as the key instances are detected for a particular input to the model. Two of the key motivators for interpreting the decision- makiinn process of a MIL sys- tem are reliability and trust -ast--in muut-identifyinn identifying the key instances allows an evaluation of the reliability of the system, which increases trust as the decision-makin process becomes more transparent. In this the term interpretability rather than explainability convey that the analysis remains tied to the models, i.c., these methods do not provide non-technical explanations in human terms.
work,
Under the SMIL assumption, which and what questions are equivalent there are only two classes (positive and negative), and the only key instances are the instances that are positive, therefore once the key instances are identified, it is also known what outcome they support. However, when there are multiple positive classes, if instances from different classes co-occur in the same bags, answering which and what questions becomes two distinct problems. For example, some key instances will support one positive class, and some key instances will support another. Therefore, solely identifying which are the key instances does not answer the second question of what class they support. Existing methods, such as key instance detection (Liu et al., 2012), MIL attention (Ilse et al., 2018), and MIL graph neural networks (GNNs; Tu et al. (2019)) do not condition their output on particular class, so it is not apparent what class each instance supports, i..., they can only answer which questions. One existing method that can answer what questions is mi-Net (Wang et al., 2018), as it produces instance-leva predictions as part of its processing. However, these instance-level predictions do not take account of interactions between the instances, so are often inaccurate. A related piece of work MIL interpretability is Tibo al. (2020), which considers interpretability within the scope of multi-multi-nnstance learning (MMIL; Tibo et al. (2017); Fuster et al. (2021)). In MMIL, the instances within bag are arranged into into further bags, giving hierarchical bags-of-bags structure. The interpretability techniques presented by Tibo et al. (2020) are model-specific as they are only designed for MMIL networks. In this work, we aim to overcome the limitations of existing methods by developing model-agnostic methods that can answer both which and whal questions.
In single instance supervised learning, model-agnostie techniques have been developed to interpret models. Post-hoe local interpretability methods, such as Local Interpretable Model-agnostic Ex- planations (LIME; Ribeiro et al. (2016b)) and SHapley Additive exPlanations (SHAP: Lundberg & Lee (2017)), work by approximating the original predictive model with locally faithful surrogate model that is inherently interpretable. The surrogate model learns from simplified inputs that rep- resent perturbations of the original input that is being analysed. In this work, one of our proposed methods is a MIL-specific version of this approach.
METHODOLOGY
At the start of this section, we outline the requirements for MIL interpretability (Section 3.1). In Section 3.2, propose three methods that meet these requirements under the assumption that there are no interactions between instances cadependent-instanee methods). In Section 3.3 we remove this assumption and propose our local surrogate model-agnostic interpretability method for MII.,
2

---

Published as a conference paper at ICLR 2022
3.1 MIL INTERPRETABILITE REQUIREMENTS
A general MIL classification problem has C' possible classes, with one class being negative and the rest being positive. This is generalisation of the SMIL assumption, which is special case when == An interpretability method that can only provide the general importance of an instance (without associating with a particular class) can only answer which questions. In order to answer what questions, the method needs to state which classes each instance supports and refutes. For- mally, for a bag of instances X = {ø1,.... .x and a bag classification function F, we want to assign a value to each instance that represents whether it supports or refutes class c:
I(X, F, { - ox)
where I is the interpretability function and Φ; E R is the interpretability value for instance i with respect to class c. Here, assume that d; > 0 means instance i supports class o; < 0 implies instance refutes class G, and the greater lox), the greater the importance of instance i. For some existing methods, such as attention, Φe the same for all classes, and d; > 0 in all cases, these requirements are not met. We later demonstrate these limitations in Section 5. In the next two sections we propose several model-agnostic methods that satisfy these requirements.
3.2 DETERMINING INSTANCE ATTRIBUTIONS
In this section, we propose three model-aggostic methods for interpreting MIL models under the assumption that the instances are independent. This means we can observe the effects of each in- stance in isolation without worrying about interactions between the instances. We exploit a property of MII. models: the ability to deal with different sized bags. As MII. models are able to process bags of different sizes, it is possible to remove instances from the bags and observe any changes in prediction, allowing us to understand what instances are responsible for the model's prediction. Below, we propose three methods that use this property to interpret model's decision making.
Single Given a bag of instances X = (.1,..., Jg and a bag classification function F, we can take each instance in turn and form single instance bag: X₁ {2,} } for i {l...., k}. We then observe the model's 's prediction on each single instance bag Φ; = Fx().). where F, is the output of F for class (i.e., the t entry in the output vector of F(X.)). A large value for de suggests instance i; supports c, and a value close to zero suggest it has no effect with respect to class c. If we repeat this over all instances and all classes, we can build picture of the classes that each instance supports, allowing us to answer what questions. However, this method cannot refute classes (i.e., di 2 0 in all cases). It should be noted that this method gives the same outputs as the inherent interpretability of mi-Net (Wang et al., 2018), but here it is model-agnostic rather than a model model-specific method.
One Removed natural counterpart the Single method is the One Removed method, where each instance is removed from the complete bag in turn, i.c., we form bags X, = X \ {x;}. For particular class c, we can then observe the change in the model's prediction caused by removing a; from the bag: 6; F, (X) - FF FXX)). ). If the prediction decreases, rr supports c, and if increases, a; refutes e, i.e., this method is able to both support and refute different classes. However, there are other instances in the bag that support or refute class we may not observe change in prediction when T, is removed, even xy is a key instance.
Combined In order to access the benefits of both the Single and One Removed methods, we can combine their outputs. A simple approach is to take the mean, i.e., Φ; = [(F(((;;)) + Fx) F.(X) \ {xr))] This method can identify the important instances revealed by the Single method, and also refute outcomes as revealed by the One Removed method.
With these three methods, it is assumed that there is not true for all datasets, therefore, in the next section, we remove this assumption and propose a further method that is able deal with the interactions between instances.
interactions between the instances. This
3.3 DEALING WITH INSTANCE INTERACTIONS
In order to calculate instance attributions whilst accounting for interactions between instances, we have consider the effect of each instance within the context of the bag. With instance interactions,
3

---

Published as a conference paper at ICLR 2022
the co-occurrence of two (or more) instances changes the bag label from what it would be if the two instances were observed independently. The instances have different meanings depending on the context of the bag, L.e, their own they mean something different to what they mean when observed together. One way to uncover these instance interactions is to perturb the original input to the model and observe the outcome. In the case of MII., these perturbations of removing instances from the original bag. By sampling coalitions of instances, and fitting weighted linear regression model against the coalitions and their respective model predictions, is possible to construct a surrogate locally faithful interpretable model that accounts for the instance interactions in the original bag. A coalition is a binary vector E {0. 1] that represents a subset S = [ = },, so the number of ones in the coalition | is equal to the length of the S, The surrogate model Yo takes the form
take the form
k
9e(2) 20
+
i=1
where each coefficient o; E {øi..... } is the importance attribution for each instance 24 E X with respect to class c. Given a collection of n coalitions Z, minimising the loss function
L(Fe: Ic: T) = E(F(s) (
=EZ
means Pe is locally faithful approximation of the original model F for class c. The loss function is weighted by a kernel т, which determines how important it is for ge to be faithful for each individual coalition. Here, ge only approximates F' for class c, i.e., to produce interpretations for bag with respect to all classes, a surrogate model needs to be fit for each class. Similar approaches have been applied in single instance supervised learning in methods such as LIME and KernelSHAP. Both use different choices for LIME employs l2 cosine distance, and KernelSHAP uses weighting scheme that approximates Shapley values (Shapley, 1953). For single instance supervised learning models, when perturbing the inputs, it is not possible to simply 'remove' a feature from an input as the models expect fxxed-izee inputs - either a new model has to be re-trained without that particular feature, appropriate sampling from other data has to be undertaken, which can lead to unrealistic synthetic data. In MIL, as models are able to deal with different size bags, instances can be removed from the bags without the need for re- re-training sampling from other data, meaning these issues do not occur in our setting.
While it is possible utilise the weight kernels from LIME and KernelSHAP for MII,, identify a significant drawback with both methods. Their choice for æ weights all coalitions of the same size equally, i.c., they do not consider the content of the coalitions, only their size. Just because two coalitions are of the same size does not mean it is equally important that the surrogate model is faithful to both of them. Furthermore, the sampling strategies of both approaches lead to very large ([2) close to 1) or very small coalitions ([2) close to 0). Unless the number of samples is very large, samples of average size ([2) close to 0.5) will not be chosen. As we see in Section 4.4, this approach to sampling is appropriate for some datasets, but not for others.
We aim to overcome both of these drawbacks with our own MIL--pecific choice of weight kernel and sampling approach. Below, we propose Multiple Instance Learning Local Interpretations (MILLI). At the core of our approach is a new method for weighting coalitions based on an initial ranking of instance importances r; E {ri...., J.J. For coalition z and ranking of instance importances we define our weight kernel Tar as:
1
k
Th (2,
2i Tr(r),,
(3)
2
i=0
(2a 1)(1 [1 )e Dr. 1-a, a,
if 0,
where T (ri)
(4)
(1 2a (1 + - )el (r: + a, otherwise.
B, if a < 0.5,
and
1-8, otherwise.
Tn is a function that weights instances based on their order in the ranking. Its two hyperparameters, [0, and B E (-o0, oo) define the shape of the function, and ultimately determine the kemnll
4

---

Published as a conference paper at ICLR 2022
Tx.. The value of a dictates whether the sampling should be biased towards instances that highly ranked or not: a > 0.5 means π. is biased towards instances higher in ordering, and a < 0.5 means Th is biased towards values lower in the ordering. The value of / is discussed below, it becomes important when sampling coalitions. We provide an illustration of Tn in Figure
0.1
0.3
0.8
1.00
1.00
1.00
0.75
0.75
0.75
B
0.50
0.50
0.50
B = 0.0
8 0.1
0.25
0.25
0.25
B 1.0
0.00
0.00
0.00
Figure The effect of a and on Tr (Equation 4).
In Txg, the coalition weighted on its content rather than its length, i.e., the distance between a subset and bag is no longer determined simply by the number of instances removed. This is beneficial, as removing many con-discriminatory instances from the bag likely means the label of the bag remains the same, despite being different in size. Conversely, removing one discriminatory instance could change the bag label, even though the bag is only one instance smaller. As well as using na in the weight kernel, we also utilise it to sample coalitions. We can consider the problem of sampling as repeated coin toss, where P(z, = 1) = Pt and P(z, = 0) = P:- In equal random sampling, every value p. E { p& } is equal to 0.5, meaning =[[=[] = 0.5k. However, it is possible to improve upon equal random sampling by changing the value of for each instance change the likelihood of each instance being involved in coalition. To sample
in bag. i.e., more informative coalitions, we set Pt = (Ta)), meaning [[[=|| fo πR dr:
2a-1 1 (e Ab 8k 1) + k(1 a), if 8 k
[|
E
(5)
20 (e-ôk + 3k 1) ka,
otherwise.
k:
If 3 < O, the sampling is biased towards smaller coalitions, and if 3 > 0, the sampling is biased towards larger coalitions. The maximum and minimum E[[=[] is controlled by a. When a 0.5 or =
O, every value p; E (PI..... Pk is equal to 0.5, meaning we have equal random sampling as described above. We provide an illustration how E[|=[] changes Figure 2.
1.0
0.6
0.8
0.3
0.8
0.5
a= 1.0
0.4
0.2
-3
9
The effect of a and 8 B[|a]] (Equation 4). We give the expected coalition size proportion of the bag size, i.e., E||=]//k:
Figure 2:
The final part of MILLI is how to determine the initial ranking of importances. For this, we can use the Single method from Section 3.2, which produces values {Ø1.... Φv), and we convert this to an instance ranking: the new values {ri,.... rk) represent the position for instance in (ø....., Φx) (e.g., the instance with the greatest value for é has 0). MILLI more expensive to compute that the methods outlined in Section 3.2: OCCnk²) compared with OCC)). However, when there interactions between the instances, this extra complexity is required in order build better un- derstanding of the classes that each instance supports and refutes, allowing more accurate answering of what questions. It is important note that MILLI is indeed model agnostic - only requires access to the bag classification function F, and makes no assumptions about the underlying MIL model. As we demonstrate in the next section, this means we can apply it to any MII. model.
5

---

Published as a conference paper at ICLR 2022
4 EXPERIMENTS
We apply our model-agnostic methods to seven MIL datasets. In this section, we detail the evaluation strategy (Section 4.1), the datasets (Section 4.2), models (Section 4.3), and results (Section 4.4). For implementation details, see Appendix A.
4.1 EVALUATION STRATEGY
For our evaluation, we do not assume that the interpretability methods have consistent output do- mains; only assume a larger value implies larger support. Therefore, the best approach for evaluating the interpretability methods is to use ranking metrics - rather than looking at the ab- solute values produced by the methods, we compare the relative orderings they produce. As noted by Carbonneau et al. (2018), although a large number of benchmark MIL datasets exist, many do not have instance labels. Without instance labels, evaluating interpretability is challenging, as we do not have a ground truth instance ordering to compare to. We identify two appropriate evaluation metries: for datasets with instance labels, we propose the use of normalised discounted cumulative gain at (NDCG @ n), and for datasets without instance labels, perturbation curve compared with random orderings (AOPC-R: Samek et al. (2016)). While AOPC- R does not require instance labels, it is very expensive to compute. A significant difference between the two metrics is the way they weight the importance ordering: NDCG@n prioritises performance at the start of the ordering, whereas AOPC-R equally prioritises performance across the entire dering. We are the first to propose both of these metrics for use in evaluating MIL interpretability, and further discuss the advantages of both metrics in Appendix A.3.
propose the use of area under the
4.2 DATASETS
Below we detail the three main datasets which evaluate our interpretability methods. These datasets were selected as they have instance labels, so we can evaluate them using both NDCG@n and AOPC-R. However, we provide further results on the classical MIL datasets Musk, Tiger, Elephant and Fox in Appendix A.4 (Dietterich et al., 1997; Andrews et al., 2002).
SIVAL The Spatially Independent, Variable Area, and Lighting (SIVAL; Rahmani et al. (2005)) dataset consists of 25 classes of complex objects photographed in different environments, where each class contains 60 images. Each image has been segmented into approximately 30 segments, and each segment is represented by 30-dimensional feature vector that encodes information such as the segment's colour and texture. The segments are labelled as containing the object or containing background. We selected 12 of the 25 classes be positive classes, and randomly sampled from the other 13 classes to form the negative class. For additional details see Appendix A.6.
Four-class MNIST-Bags The SIVAL dataset has no co-occurrence of instances from different classes, i.e., each hag only contains one class of positive instances. Therefore, the which and what questions are the same. In order to explore what happens when there are different classes of positive instances in the same bag, we propose an extension of the MNIST-Bags dataset introduced by Ilse et al. (2018). Our extension, four-class 8 in bag, class 2 if in bag, class 3 in and 9 in bag. and class O otherwise. In this dataset, answering what questions goes beyond answering which questions, e.g., the existence of an 8 supports classes one and three, but refutes classes zero and two. For further details see Appendix A.7.
MANIST-Bags (4-MNIST-Bai) Bags)) is setup follows: class 1 if
Colon Cancer To test the applicability of the interpretabiliiy methods on larger bag sizes, we apply it to colorectal cancer tissue classification. The ColoRectal Cancer (CRC) dataset (Sirinukunwattana et al., 2016) is a collection of microscopy images with annotated nuclei. We follow the same setup as Ilse et al. (2018), in which bag is positive if it contains one or more nuclei from the epithelial class. This means the problem conforms to the SMIL assumption, and there are no interactions between instances. Each microscopy image is 500 x 500 pixels, and was split into 27 x 27 pixel patches to give a maximum of 324 patches per slide (patches were discarded if they contained mostly slide background). Not all of the instances are labelled, so tailor our assessment using NDCG @n to only consider labelled instances. For additional details see Appendix A.8.
We use to refer to image from the MNIST dataset that represents the number n, even though that is the assigned class label in the M-NNSST-Bags dataset.

---

Published as a conference paper at ICLR 2022
4.3 MODELS AND METHODS
To highlight the model-agnostic abilities of the proposed methods, for each dataset, we trained four different types of multi-class MIL model: an embedding based multiple instance neural network (MI-Net; Wang et al. (2018)). an instance-hased multiple instance network (mi-Net; Wang et al. (2018)), an attention-based model (MI-Attn; lise et al. (2018)), and a graph neural network model (MI-GNN; Tu et al. (2019)). Aside from MI-Net, each of these models provide their own inherent interpretability method that we can compare our methods against. Additional details on the models given in Appendix A.2. As well evaluating we also compare against LIME and SHAP using both random and guided sampling (choosing coali- evaluating nine different
independent- independentiinstance methods and MILLI,
tions that maximise the weight kernel). This means that in total interpretability methods: inherent interpretability, the three independent-instane. methods (Section 3.2), two LIME methods, two SHAP methods, and MILLI (Section 3.3). Note that, aside from the inherent interpretability methods, these methods are either novel hidependenttinstane methods and MILLI) or are applied to MIL for the time in this study (LIME and SHAP).
4.4 RESULTS
For each dataset, we measured the performance of each interpretability method on each of the four MIL models. For the SIVAL dataset we measured the performance only the negative class and the bag's s true class, but for all the other datasets we evaluated the interpretability over every class. This distinction was made as know that each SIVAL bag can only contain instances from one positive class, therefore it is not necessary to evaluate over all possible classes We present the interpretability results run against the test set for the SIVAL, 4-MNIST-Bags, and CRC datasets in Tables 1. 2 and 3 respectively. The results are averaged over ten repeat trainings of each model, and we also give the test accuracy of each of the underlying MIL models. We discuss our choice of hyperparameters in Appendix A.5.
By analysing the NDCG@n interpretability results across these three datasets, we find that MILLI performs best with an average of 0.85, followed by the GuidedSHAP and Combined methods (both with an average of 0.81). For the average AOPC-R results (including the results on the classi- cal MIL datasets, see Appendix A.4), we find that Combined, GuidedSHAP, RandomL.ME, and MILLI are the best performing methods, however the overall the difference in performance between the methods is much less than what we observe for the NDCG@1 n metric. For the 4-MNIST Bags dataset, the difference in performance of MILLI on NDCG@n vs AOPC-R is due to the difference in weighting between the metries: MILLI achieves better ordering for the most important instances (outperforms other methods on NDCG@n), but gives the same ordering as other methods for less important instances (equal performance on AOPC-R). In the majority of cases, all of our proposed model-agnostic methods outperform the inherent interpretability methods, and are relatively consis- tent in performance across all models. For the SIVAL dataset, the independent-instame perform well, which is expected as the instances are independent. However, on the 4-MNIST-
methods
Table 1: SIVAL interpretability IDC@@n/ AOPC-R results. For all of the interpretability methods, the standard error of the mean was 0.01 or less.
Methods
MI-Net
mi-Net
MI-Attn
MI-GNN
Overall
Model Acc
0.819
0.808
0.813
0.781
0.805
Inherent
N/A
0.813 / 0.265
0.717 0.000
0.586 0.023
0.705 0.098
Single
0.825 / 0.280
0.813 / 0.266
0.801 0.302
0.734/ 0.194
0.793 0.261
One Removed
0.778 0.256
0.837 / 0.308
0.736 / 0.293
0.776 / 0.231
0.782/ 0.272
Combined
0.828 0.291
0.828 / 0.294
0.803 0.316
0.762 / 0.227
0.805 0.282
RandomSHAP
0,801 / 0.284
0.807 / 0.300
0.766 / 0.322
0.784 0.250
0.789 0.289
GuidedSHAP
0.826 0.291
0.819 / 0.290
0.790
0.313
0.765 0.235
0.800 0.282
RandomL IME
0.809 / 0.295
0.815 / 0.310
0.776 / 0.335
0.793 / 0.258
0.798 0.299
GuidedLIMM
0.780 0.259
0.830 0.310
0.742 0.296
0.776 0.782/ 0.233
0.274
MILLI
0.823/ 0.283
0.827 / 0.307
0.794 0.307
0.790 0.239
0.808 0.284
7

---

Published as a conference paper at ICLR 2022
Table 2: 4-MNIST-Bags interpretability NDCG@n / AOPC-R results. The MI-GNN model takes four times as long for a single model pass than the other models, so calculating its AOPC-R results on this dataset was infeasible (see Appendix A.3). For all of the interpretability methods, the standard error of the mean was 0.01 or less.
Methods
MI-Net
mi-Net
MI-Autn
MI-GNN
Overall
Model Ace
0.971
0.974
0.967
0.966
0.970
Inherent
//A N/A
0.723 / 0.136
0.750 / 0.002
0,419 / N/A 0.630 / 0.069
Single
0.722 0.138
0.723 / 0.137
0.778 / 00.16
0.761 N/A 0.746 / 0.146
One Removed
0.811 0.187
0.810/ 0.166
0.809 00.14
0.786 N/A 0.804 / 00.172
Combined
0.775 0.184
0.175 0.15
0.816 0.188
0.804 N/A
0.792 / 0.184
RaPdomSHAP
0.813 0.186
0.809 / 0.185
0.825 / 0.178
0.828 N/A
0.819 / 00.13
GuidedSHAP
0.773 / 0.187
0.773/0.15
0.816 / 0.183
0,805 N/A 0.792 / 0.186
RandomLIME
0.828 / 0.189
0.825
0.189
0.841 /0.179
0.838 N/A 0.833 / 0.186
GuidedLIME
0.760 0.118
0.785 0.187 0.187
0..14
0.776 / N/A 0.769/ / 0.174
MILLI 0.947 0.190 0.943 / 0.189
0.917 / 0.181 0.959 N/A 0.942 / 0.186
Table 3: CRC interpretability NDCG@n results. As this dataset has much larger bag sizes (264 instances per bag on average), it is infeasible to compute its AOPC-R results (see Appendix A.3). For all of the interpretability methods, the standard error of the mean 0.02 less.
Methods
MI-Net mi-Net
MI-Attn
MI-GNN
Overall
Model Ace
0.795
0.795
0.830
0.770
0.797
Inherent
N/A
0.845
0.692
0.684
0.740
Single
0.815
0.845
0.847
0.786
0.823
One Removed
0.698
0.682
0.701
0.815
0.724
Combined
0.815
0.845
0.846
0.803
0.827
Rand 0.695
0.690
0.717
0.703
0.701
GuidedSHAP 0.815
0.845
0.846
0.804
0.827
RandomLIME 0.695
0.702
0.716
0.813
0.731
GuidedLIME
0.687
0.699
0.701
0.818
0.726
MILLI 0.753
0.800
0.810
0.780
0.786
dataset, MILLII excels as it samples informative coalitions that capture the instance interactions. On the CRC dataset, methods that are able to isolate individual instances, (i.e., the Single, Combined, and GuidedSHAP methods) perform well due to instance independence in this dataset. Further- more, if we consider the witness rate (WR; the proportion of key instances in each bag; Carbonneau et al. (2018)) of the datasets, we find that the CRC dataset has a higher WR (27.47% than SIVAL (15.28%) and 4- -aggs (8.04%). This means, with larger coalitions, it becomes more difficult to isolate the contributions of individual instances, which is why the One Removed and Random sampling methods struggle.
DISCUSSION
To demonstrate how MILLI captures instance interactions, and to show the limitations of existing methods that cannot condition their output for particular class, we compare the MILLI interpre- tations with the attention interpretations on the 4-MNIST-Bags dataset (Figure 3). As this bag con- tains an 8 and 9, the correct label for it is class three. Both the MILLI and attention methods have identified the 8 and 9 instances as key instances, i.e., they have both answered the which question. However, only MILLI correctly identifies that the 8 refutes class two and that the 9 refutes class one, answering the what question, something that the attention values do not do. We provide further examples, including interpretability outputs for the SIVAL and CRC datasets, in Appendix A.10.
8

---

Published as a conference paper at ICLR 2022
/
5
7
9
4
6
/
8
Attention
MILLI
MILLI 2
MILLI
Figure 3: An example of the identification of key instances for the 4-MNIST-Bage experiment. The top row shows the attention value for each instance, and bottom three rows show the output MILLI for classes three, two, and one respectively (green support, red = refute).
LIME, SHAP, and MILLI all require the number of sampled coalitions be selected. A greater number of coalitions means there is more data to fit the surrogate model on, therefore it is more likely to faithful the underlying model. However, the more samples that are taken, the more expensive the computation. As shown in Figure 4, MILLI is the most consistent in terms of performance and efficiency. For all methods, it is a case of diminishing returns, where additional samples only lead to small increase in performance.
SIVAL
4-MNIST- -aag
CRC
0.9
RandomSHAP
NDCGON
0.8
0.8
0.8
GuidedSHAP
RendomLIME
0.6
0.7
GuidedLIME
0.6
MILLI
0.5
0.6
200
200
1000
1500
Number Samples
Figure 4: The effect of sample size on interpretability performance for the MI-Attn model. We provide the results of the same study but for the other MIL models in Appendix A.9.
The different sampling approaches used in this work have distinct advantages. When there are in- teractions between instances, RandomSHAP and RandomLIME are better than GuidedSHAP and GuidedLIME as they form larger coalitions that are more likely to capture the instance interac- tions. However, for the CRC dataset, where there a high WR, RandomSHAP, RandomLIME and GuidedLIME struggle as they cannot form small coalitions. MILLI performs relatively well across all datasets the size of its sampled coalitions can be adapted depending on the dataset, demonstrating its generalisability. However, it still out- performed by GuidedSHAP
large number of independent instances and
the CRC dataset. One possible explanation for this is that MILLI is limited to sampling only smaller or larger coalitions, whereas GuidedSHAP samples both large and small coalitions. One approach for improving MILLLI would be incorporate paired sampling (Covert & Lee, 2020), where for every sampled coalition 2;, we also sample its complement coali- tion 1 2,. Following the advice of Carbonneau al. (2018), further MIL studies on more complex datasets, such as Pascal VOC (Everingham et al., 2010), could also be insightful for evaluating MIL. interpretability methods. It would also be beneficial to examine these techniques are applicable to MII. domains beyond classification, e.g. MII. regression as in Wang et al. (2020).
CONCLUSION
In this work, we have discussed the process of model-agnostic interpretability for MII., Along with defining the requirements for MIL interpretability, we have presented our own approaches and com- pared them to existing inherently interpretable MIL models. By analysing the methods across sev- eral datasets, we have shown that independent-inssance methods can be effective, but local surrogate methods required when there interactions between the instances. All of our proposed meth- ods are more effective than existing inherently interpretable models, and are able to not only identify which the key instances, but also say whal classes they support and refute.

---

Published as a conference paper at ICLR 2022
REPRODUCIBILITY STATEMENT
For our work, the main details for the methodology are detailed in Section In addition, we provide details that will aid reproducibility in the Appendix. The dataset sources and a general overview of our implementation
given in Appendix A.I. Further information on the MIL models used in this work can be found in Appendix A.2, and specific details on their architectures and training can found in Appendices A.6, A.7, and A.8 for the SIVAL, 4-MNIST-Bags, and CRC datasets respectively. The codehase for this work can be found on GitHub: https: / //gi thub. com/ JAEarly /MMLLII.
ACKNOWLEDGEMENTS
This work was funded by AXA Research Fund and the UKRI Trustworthy Autonomous Systems Hub (EP/V00784X/1). We would also like to thank the University of Southampton and the Alan Turing Institute for their support.
The authors acknowledge the use of the IRIDIS High Performance Computing Facility and associ- ated support services at the University of Southampton in the completion of this work. IRIDIS-5 GPU-enabled compute nodes were used for the long running experiments in this work.
We also acknowledge the use of SciencePlots (Garrett, 2021) for formatting our Matplotlib figures.
REFERENCES
Jaume Amores. Multiple instance classification: Review, taxonomy and comparative study. Artificial intelligence, 201:81-105, 2013.
Stuart Andrews, Ioannis Tsochantaridis, and Thomas Hofmann. Support vector machines for multiple-instance learning. In NIPS, volume 2, pp. 2002.
Marc-André Carbonneau, Veronika Cheplygina, Eric Granger, and Ghyslain Gagnon. Multiple in- stance learning: survey of problem characteristics and applications. Pattern Recognition, 77: 329-353, 2018.
Ian Covert and Su-In Lee. Improving kemelshap: Practical shapley value estimation via linear regression. arXiv preprint arXiv:2012.01536, 2020.
Thomas G Dietterich, Richard Lathrop, and Tomás Lozano-Pérez, Solving the multiple instance problem with axis-parallel rectangles. Artificial intelligence, 89(1-2):31-71, 1997.
Mark Everingham, Lue Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International journal of computer vision, 88(2): 303-338, 2010.
Saul Fuster, Trygve Eftestøl, and Kjersti Engan. Nested multiple instance learning with attention mechanisms. arXiv preprint arXiv:2111.00947, 2021.
John D. Garrett, garretti403/SciiencPloss. September 2021. doi: 10.5281//eeodoo441666699 URL http:/ / hww./ii.oo.. 10. wwwp////////ee.oeeeeeeeeeeeooo.e 4106649.
Leilani Н Gilpin, David Bau, Ben Z Yuan, Ayesha Bajwa, Michael Specter, and Lalana Kagal. Explaining explanations: An overview of interpretability of machine learning. In 2018 IEEE 5th International Conference on data science and advanced analytics (DSAA), pp. 80-89. IEEE, 2018.
Maximilian Ilse. Jakub Tomezak, and Max Welling. Attention-based deep multiple instance learn- ing. In International conference on machine learning, pp. 2127-2136. PMLR, 2018.
Sirui Li, Weixing Sun, and Tim Miller. Communication human-agent teams for tasks with joint action. In International Workshop Coordination, Organizations, Institutions, and Norms in Agent Systems, pp. 224-241. Springer, 2015.
Guoqing Liu, Jianxin Wu, and Zhi-Hua Zhou. Key instance detection in inllti-instane learning. In Asian Conference on Machine Learning, pp. 253-268. PMLR, 2012.
10

---

Published as a conference paper at ICLR 2022
Scott Lundberg and Su-In Lee. A unified approach interpreting model predictions. arXiv preprint arXiv:1705.07874, 2017.
Oded Maron and Tomás Lozano-Pérez. A framework for multiple-instance learning. Advances in neural information processing systems, pp. 570-576, 1998.
Christoph Molnar. Interpretable machine learning. Lulu, com, 2020.
Rouhollah Rahmani, Sally Goldman, Hui Zhang, John Krettek, and Jason E Fritts. Localized content based image retrieval. In Proceedings of the 7th ACM SIGMM international workshop Multimedia information retrieval, pp. 227-236 2005.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. Model-agnostic interpretability of ma- chine learning. arXiv preprini urXiv:/606e 05386, 2016a.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. "Why should trust you?" Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pp. 134-1144, 2016b.
Wojciech Samek, Alexander Binder, Grégoire Montavon, Sebastian Lapuschkin, and Klaus-Robert Müller. Evaluating the visualization of what a deep neural network has learned. IEEE transactions on neural networks and learning systems, 28( (1:211):0662222 2673, 2016.
Stephen Scott, Jun Zhang, and Joshua Brown. On generalized multiple-instance learning. Interna- tional Journal of Computational Intelligence and Applications, 5(01))21-35, 2005.
Lloyd S Shapley. value for n-person games. Princeton University Press, 1953.
Korsuk Sirinukunwattana, Shan Ahmed Raza, Wah Tsang, David RJ Snead, Ian A Cree, and Nasir M Rajpoot. Locality sensitive deep learning for detection and classification of nuclei in routine colon cancer histology images. IEEE transactions on medical imaging, 1965):119--126, 2016.
Alessandro Tibo, Paolo Frasconi, and Manfred Jaeger. A network architecture for multi-multi- instance learning. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 737-752. Springer, 2017.
Alessandro Tibo, Manfred Jaeger, and Paolo Frasconi. Learning and interpreting multi-multi- instance learning networks. J. Mach. Learn. Res., 21:193-1, 2020.
Ming Tu, Jing Huang, Xiaodong He, and Bowen Zhou. Multiple instance learning with graph neural networks. arXiv preprint 100.::9906004800 2019.
Kaili Wang, Jose Oramas, and Tinne Tuytelaars. In defense of Istms for addressing multiple instance learning problems. In Proceedings the Asian Conference on Computer Vision, 2020.
Xinggang Wang, Yongluan Yan, Peng Tang, Xiang Bai, and Wenyu Liu. Revisiting multiple instance neural networks. Pattern Recognition, 74:15-24, 2018.
Nils Weidmann, Eibe Frank, and Bernhard Pfahringer. A two-level learning method for generalized multi-instance problems. In European Conference on Machine Learning. pp. 468-479 Springer, 2003.
Zhi-Hua Zhou, Yu-Yin Sun, and Yu-Feng Li. Multi-instance learning by treating instances as non- iid samples. In Proceedings of the 26th annual international conference on machine learning, pp. - 1256, 2009.
11

---

Published as a conference paper at ICLR 2022
A APPENDIX
A.1 IMPLRMENTATION DETAILS
All code for this work was implemented in Python 3.8, using the PyTorch library for the machine learning functionality. Hyperparameter tuning was carried out using the Optuna libary. Some lo- cal experiments were carried graphics card with 4GB of VRAM. GPU support for machine learning was enabled through CUDA v11.0. Other longer running experiments, such hyperparameter tuning, were carried on re- mote GPU node utilising a Volta V100 Enterprise Compute GPU with 16GB of VRAM. The longest model train was the GNN for the CRC dataset, which took just under half an hour. The longest running experiment was the hyperparameter analysis for the CRC dataset at just under 40 hours. This was due to high number of samples for the local surrogate methods and the fact that the results were average over 10 different models. The randomisation of data splits was fixed using seeding; the seed values are provided in the code for each experiment. We use the following sources for our data:
Dell XPS Windows laptop, utilising GeForce GTX 1650
The annotated SIVAL dataset was downloaded from the publicly accessible page:
http://pages cs. wisc.edu/ wedu/bbee "bset /ata//
The MNIST dataset was access directly from the PyTorch Python library:
https://pytorch.. http:///sioo///a////eeeeaeee
html tmnist,
The CRC dataset was downloaded from the publicly accessible page:
https: https////www...//. uk http//////oosaaaeeeee ttw/t////taaaaao
abe lednoo leihe/.
The Musk dataset was downloaded from the publicly accessible page:
https://archive ics. www. edu/ml adtu/ml/dataaee htt////aaattt/aaaaaaaeeeeeeeeeee ion
The Tiger, Elephant and Fox datasets were downloaded from the publicly accessible page:
http://wwed lumbia. edu/" andrews 1/ datasets. .ttml
A.2 MODELS
In this work, we trained four different models for each dataset. In this section, we provide further details each of the models. Each model was tuned independently for each dataset, in the later sections we provide details of the specific architectures for each dataset. We also tuned the learning rate, weight decay, and dropout for each of the models independently for every dataset. Again, these are given in the later sections for each specific dataset.
MI-Net Each instance is embedded a fixed size, and then these embeddings are aggregated to give single bag embedding. This aggregation can either take the mean (mil-mean) or max (mil- max) of the instance embeddings. The bag embedding is then classified to give an overall bag prediction. For this model, we tuned the number and size of fully connected (FC) layers, as well as the choice of aggregation function.
mi-Net A prediction is made for each individual instance, and then these aggregated to single outcome for the bag as a whole. Similar to the MI-Net model, we tuned the FC layers and the aggregation function.
MI-Attn Each instance is embedded to a fixed size, and then the mil-attn block produces an atten- tion value for each instance embedding. The instance embeddings are then aggregated to a single bag embedding by performing a weighted sum based on the attention values. The bag embedding then classified to give an overall bag prediction. The mil-attn block is the same as per the MIL attention mechanism proposed by Ilse et al. (2018), i.e., using a single hidden layer. We tuned the the number and size of the FC layers, as well the size of the hidden attention layer.
MI-GNN The GNN model treats the bag as a fully connected graph and uses graph convolutions to propagate information between instances. Initially, the original instances are embedded to a fixed size. These instance embeddings are then passed onto GNNembet block and GNN block, the output of which is used to reduce the graph representation down into a single embedding using differentiable pooling (gnn-pool). The final bag representation is then classified give an overall bag prediction. For more details see Tu et al. (2019). We tuned the number and size of the embedding, GNN, and classifier layers.
12

---

Published as a conference paper at ICLR 2022
A.3 EVALUATION METRICS
In this section, we provide further details on how we use normalised discounted cumulative gain at (NDCG@n) evaluate the interpretability methods for datasets with instance labels, and how use area under the perturbation curve with random orderings (AOPC-R) to evaluate the interpretabil- ity methods for datasets without instance labels.
NDCG@n To use NDCG@n, we first need a ground truth ordering to compare to. For particular class, we can place each instance into one of three groups based their ground truth instance labels: supporting instances, neutral instances, and refuting instances. The ideal instance ordering for that class would then have all of the supporting instances at the beginning, followed by all the neutral instances, and then all of the refuting instances at the end. Then, given interpretability outputs ( (bk for particular class, we rank the outputs from highest to lowest, i.e., in order of how much they support that class. This importance ordering is then compared to the ground truth ordering using the follow metric:
ril(i) logz (i +
NDCG@n IDCG
a
where IDCG
logz + +1
IDCG is the ideal discounted cumulative gain that normalises the scores across different values of n. The relevance funetion ri) is follows:
rel(i)= - {
if the 7th instance in the ranking supports the class, the 7th instance the ranking refutes the class, otherwise.
AOPC-R Although originally designed for single instance supervised learning, here we adapt AOPC-R for MIL. Given an importance ordering, AOPC successively removes the most relevant instances (i.e., those at the start the given importance ordering), and measures the change in prediction. A better ordering will show more rapid decrease in prediction, as instances that are more supportive will be removed first. This rate of decrease prediction with respect to class for classifier F and bag X {æ1...., Fk} is measured follows: given the ordered bag Ox = {o)..... 0x (which is just ordered by instance importance, with the most important instances at the start),
AOPC
F((X) Fx MORF).
(6)
where x (-( MORF = X. (+)
(7)
and MoRF = x (i-1)
MoRF fo,).
(8)
To normalise the results, one approach is to measure the average difference in AOPC for the given ordering to the AOPC of several random orderings:
AOPC-R 1 Σ AOPC(Ox) (PPC(OO))AOC((On)))
where the number of random orderings, and On, is the tth random ordering of X. The repeated calls to F. make AOPC-R very expensive to compute. This can be reduced by examining the first perturbations rather than all - possible perturbations, but that was not something we investigated in this work (we kept A and 10). Furthermore, due to the use of random orderings, this evaluation metric is inherently stochastic. meaning not only do we have variance in the orderings produced in the methods (e.g.. from random sampling), also have variance due measurement. NDCG@n n does not have either of the issues (i.e., its cheap to compute and deterministic), however requires (at least some) instance labels.
13

---

Published as a conference paper at ICLR 2022
A.4 ADDITIONAL RESULTS
In this section, we detail our additional results on the Musk and Tiger, Elephant & Fox (TEF) clas- sical MIL datasets. Although these datasets do not have instance labels, since they are widely used MIL. datasets, it is useful to see how our interpretability methods perform on them. Each of these datasets are instance independent, and as they have a low number of instances per bag. we adapted our local surrogate methods to allow sampling with repeats (otherwise we cannot sample enough coalitions). our previous experiments, we restricted the local surrogate methods to sampling without repeats, which was not an issue with the larger bag sizes in the SIVAL, 4-MNISTT Bags, and CRC datasets. For each of these classic datasets, we used the same model hyperparameters that we used for SIVAL (see Appendix A.6), i.e., we didn't 't retune the training parameters or model archi- tectures. However, we did tune the parameters for the interpretability methods, which we discuss in Appendix A.5. We give the interpretability results well as the model performance for Musk (Table A1), Tiger (Table A2), Elephant (Table A3), and Fox (Table A4) below.
Table AI: Musk interpretability AOPC-R results. Note that we are using the Muskl dataset, rather than Musk2, as the latter has much larger bag sizes, making the use of AOPC-R infeasible.
Methods
MI-Net
mi-Net
MI-Attn
MI-GNN
Overall
Model Acc
0.871 + 0.024 0.836 = 0.027 0.793 + 0.029 0.793 + 0.028 0.823 + 0.016
Inherent
N/A
0.108
0.010
0.002 + 0.015
0.003 + 0.006
0.036 + 0.011
Single
0.123 + 0.010
0.108
0.010
0.113 = 0.010
0.090 + 0.010
0.108 + 0.010
One Removed
0.108 0.009
0.107 = 0.010
0.088 + 0.008
0.076 + 0.009
0.095 = 0.009
Combined
0.124 + 0.010
0.107
0.010
0.116 + 0.010
0.088 + 0.010
0.109 + 0.010
RandomSHAP
0.115 0.009
0.104 = 0.010
0.110 = 0.009
0.102 = 0.009
GuidedSHAP
0,122 + 0.009
0.106
0.010
0.116 = 0.010
0.004
0.107 = 0.009
RandomLIME
0.113 0.009
0.107
0.010
0.110 + 0.009
0.080 + 0.009
0.102 + 0.009
GuidedLIME
0.117 + 0.009
106
0.010
0.102 + 0.008
0.077 + 0.009
0.101 + 0.009
MILLI
0.117 + 0.009
0.105 0.010
0.109 + 0.009 0.084 + 0.010 0.104 + 0.009
Table A2: Tiger interpretability AOPC-R results.
Methods
MI-Net
mi-Net
MI-Attn
MI-GNN
Overall
Model Acc
0.827 + 0.024 0.807 = 0.029 0.807 + 0.019 0,800 + 0.028
0.810 + 0.005
Inherent
N/A
0.124
0.005
0.001 + 0.007 0.000 + 0.003
0.042 + 0.005
Single
0.132 + 0.005
0.125
0.005
0.120 + 0.005
0.082 + 0.003
0.115 + 0.004
One Removed
0.123 + 0.005
0.127
0.005
0.114 + 0.004
0.081 + 0.003
0.111 + 0.004
Combined
0.130 + 0.005
0.127
0.005
0.124 + 0.005
0.082 + 0.003
0.116 + 0.004
RandomSHAP
0.124 0.004
0.122
0.005
0.121 = 0.005
0.080 + 0.003
0.112 + 0.004
GuidedSHAP
0.130 + 0.005
0.125 = 0.005
0.121 + 0.005
0.082 - 0.003
0.115 + 0.004
RandomLIME
0.128 0.005
0.125 = 0.005
0.119 + 0.005
0.081 + 0.003
0.113 1 0.004
GuidedLIME
0.129 + 0.005
0.125 0.005
0.122
+ 0.005
0.082 + 0.003
0.115 + 0.004
MILLI
0.128 + 0.005 0.125 0.005 0.120 + 0.004 0.081 + 0.003 0.114 + 0.004
We find that there is very little difference interpretability performance for each of our proposed methods across these four datasets. This is to be expected, as the instances are independent, there- fore the independent-instance methods as well as the local surrogate methods are able to identify the important instances, i.e., there is little to be gained by sampling coalitions when each instance can be understood in isolation. However, this reinforces the applicability of all of our proposed meth- ods. We note that the attention and GNN inherent interpretability methods perform poorly in these experiments - this is because they are unable to condition their outputs on a specific class (i.e., they can only answer which questions, not what questions). We also note that the performance is much worse on the Fox dataset. Here, the underlying MIL models perform poorly, so it is unsurprising that the interpretability methods also perform poorly.
14

---

Published as a conference paper at ICLR 2022
Table A3: Elephant interpretability AOPC-R results.
Methods
MI-Net
mi-Net
MI-Attn
MI-GNN
Overall
Model Acc
0.857 + 0.017 0.863
0.017 0.867 + 0.016 0.853 +: 0.020
0.860 + 0.003
Inherent
N/A
0.130 = 0.006 0.000 + 0.006 0.000 + 0.003
0.043 + 0.005
Single
0.127 + 0.004
0.131
0.006
0.127 + 0.005
0.105 + 0.004
0.122 + 0.005
One Removed
0,117 0.004
0.129 = 0.006
0.105 = 0.005
0.103 = 0.004
0.114 = 0.005
Combined
0.126 + 0.004
0.130
0.006
0.127 = 0.005
0.106 0.004
0.122 = 0.005
RandomSHAP
0.124 0.004 0.126
0.006
0.119 + 0.005
0.102
+ 0.004
0.118 + 0.005
GuidedSHAP
0.127 + 0.004 0.130
0.006
=.124
0,105
0.004
0.122 + 0.005
RandomLIME
0.124 + 0.004
0.128
0.121 0.006
0.104
0.119 + 0.005
GuidedLIME
0,123 + 0.004
0.130 = 0.006
0.118 0.005
0.119 = 0.005
MILLI
0.124 + 0.005 0.128 0.006 0.121 + 0.005 0.103 + 0.005
0.119 + 0.005
Table A4: Fox interpretability AOPC-R results..
Methods
MI-Net
mi-Net
MI-Attn
MI-GNN
Overall
Model Acc
0.600 + 0.011 0.610 0.017 0.620 + 0.023 0.580 + 0.013 0.603 + 0.007
Inherent
N/A
0.050 = 0.002
0.001 = 0.002 0.000 + 0.001
0.017 = 0.002
Single
0.044 0.002
0.049 = 0.002
0.041 + 0.002
0.021 + 0.001
0.039 + 0.002
One Removed
0.043 + 0.002
0.049 = 0.002
0.040 + 0.002
0.021 + 0.001
0.038 + 0.002
Combined
0.044 = 0.002
0.050 = 0.002
0.041 = 0.002
0.021 0.001
0.039 = 0.002
RandomSHAP
0.044 + 0.002
0.049
0.002
0.041 + 0.002
0.021
+ 0.001
0.039 + 0.002
GuidedSHAP
0.045 + 0.002
0.050
0.002
0.042 + 0.002
0.021
0.001
0.040
+ 0.002 0,039 0.002
RandomLIME
0.044 + 0.002
0.050
0.002
0.041 = 0.002
0.021 + 0.001
GuidedLIME
0.044 = 0.002
0.050 = 0.002
0.041
= 0.002
0.021
1 0.001
0.039 0.002
MILLI
0.043 0.002
0.049 = 0.002
0.041 + 0.002
0.022 + 0.001
0.039 + 0.002
A.5 INTERPRETABILITY METHOD HYPERPARAMETER SELECTION
In this section discuss our method for hyperparameter selection in the interpretability methods, and detail the hyperparameters that we found to be most effective. An advantage of the inherent interpretability and independent-instanee methods is that they do not have hyperparameters, i.c. we only had to select hyperparameters for the local surrogate interpretability methods. First, we discuss our choice of hyperparameters for LIME, and then our choice of hyperparameters for MILLI.
LIME hy typeraarreeters When using the LIME weight kernel, there are two hyperparameterss to tune. Firstly, the distance measures that are commonly used are L2 and cosine distance. our experiments, found very little difference between each of these distance measures, therefore we arbitrarily chose to use L.2 distance in all of our experiments. The second hyperparameter is the kernel width, which determines the weighting of coalitions, i.e., a large kernel width means all coalitions are weighted more evenly, and a small kernel width prioritises larger coalitions. We chose to use kernel width for all of our experiments that is determined by the average bag size, such that the half coalition (i.e., |z| = 0.5k) is weighted at 0.5.
MILLI hyperparameters For MILLI, there were three hyperparameters to tune: the number of samples, a, and B. For the number of samples, (also see Appendix A.9), and chose the number of samples to be at the point where all the methods had reasonably converged. For and / we a grid search over the possible values, and chose the best performing pair of parameters. The hyperparameters were tuned for each dataset, except for the TEF datasets, in which we only tuned on Tiger and then used the same hyperparameters across
generated sample size plots such as Figure
15

---

Published as a conference paper at ICLR 2022
all three datasets. In Table A5, we provide the chosen hyperparameters for each dataset, as well as the expected coalition size B[[=]] determined by a and B. For the CRC, Musk and TEF datasets, the chosen parameters produce small values for B||2 i.c., the sampling is heavily biased towards smaller coalitions, which is expected as these are instance independent datasets. Conversely, the parameters for the 4-MNIST-Bags focus on sampling larger coalitions that are able capture the instance interactions in the dataset. Although the SIVAL dataset is instance independent, the pa- rameters also focus on sampling larger coalitions. This could be because, although the instances are independent, it may be difficult to classify an object from just one instance, i.e., several instances are needed make the correct decision. We also note that, in all cases, a < 0.5, i.e. the sampling is biased towards instances ranked lower in the initial importance ordering used by MILLI. Our ex- planation for this that easy to understand the contribution discriminatory instances, as they have a large effect on the model prediction, but it is more difficult to understand the contribution of non non-discriminatory instances, as they have much less of an effect. Therefore, more samples contain- ing non-discriminatory instances are required to properly understand their effect, hence biasing the sampling towards them.
Table A5: MILLI hyperparameters.
Dataset
Sample Size a
B
E
||
SIVAL
200
0.05
-0.01
13
4-MNIST-Bags
150
0.05
0.01
16
CRC
1000
0.008
-5.0
2
MUSK
150
0.3
-1.0
2
TEF
150
0.3
0.01
3
A.6 SIVAL EXPERIMENT DETAILS
Dataset For the SIVAL dataset, each instance is represented by 30-dimensional feature vector, and there are around 30 instances per bag. We chose 12 of the 25 original classes to be the positive classes, and randomly selected 30 images from each of the other 13 classes to form the negative class, meaning overall we had classes (12 positive and one negative). In total, we had 60 bags for each of the 12 positive classes, and 390 bags for the single negative class, meaning the class distribution was R5 5.4% for each positive class and x= 35.1% for the negative class. The arbitrarily chosen positive classes were: apple, banana, checkeredscurf,, cokecan, dataminingbook, goldmedal, largespoon, rapbook, smileyfacedoll, spritecan, translucentbowl, and wd40can. We normalised each instance according to the dataset mean and standard deviation. No other data augmentation was used. The dataset was separated into train, validation, and test data using an 80/10/10 10 split. This was done with stratified sampling in order to maintain the same data distribution across all splits.
Training When training models against the SIVAL dataset, we used batch size of one, i.e., single bag of, on average, 30 instances. We trained the models to minimise cross entropy loss using the Adam optimiser; the hyperparamater details for learning rate (LR), weight decay (WD) and dropout (DO) are given in Table A6. We utilised carly stopping based on validation loss - the validation loss had not decreased for 10 epochs then terminated the training procedure and reset the model to the point at which it caused the last decrease in validation loss. Otherwise, the maximum number of epochs was 100. The tuned architectures for each model are given in Tables A7 to A10, and the results for each model are comapred in Table All.
Table A6: SIVAL hyperparameters.
Table A7: SIVAL MI-Net architecture.
Model
LR
WD
DO
Layer
Type
Input
Output
Mi-Net
5 x 10-3
1 10
0.45
FC + ReLU + DO
30
128
mi-Net
5 x 10-4
1x 10
0.25
2
FC + ReLU + DO
128
256
MI-Attn
1 x 10-3
1x 10-5
0.15
mil-max
256
256
MI-GNN
10-4
1 10
0.2
4
FC
256
13
16

---

Published as a conference paper at ICLR 2022
Table A8: SIVAL mi-Net architecture.
Table A9: SIVAL MI-Attn architecture.
Layer
Type
Input
Output
Layer
Type
Input
Output
FC + ReLU + DO
30
512
FC + ReLU DO
30
128
2
FC ReLU + DO
512
256
2
FC + ReLU DO
128
256
3
FC ReLU + DO
256
FC + ReLU DO
256
128
4
FC
64
13
4
mil-attn((56)) DO
128
128
5
mil-mean
13
13
5
FC
128
13
Table A10: SIVAL MI-GNN architecture.
Layer
Type
Input Output
FC ReI.U + D0
30
128
2a GNN embed
SAGEConv ReLU + DO
128
128
SAGEConV + ReLU + DO
128
256
SAGEConV ReLU + DO
256
64
2b GNN cluster
SAGEConv Softmax
128
gnn-pool FC + ReLU + DO
64
4
64
128
FC
128
13
Table A11: SIVAL model results. The mean performance was calculated over ten repeat trainings of each model, and the standard error of the mean is given.
Model
Train Accuracy
Val Accuracy
Test Accuracy
MI-Net
0.984 +: 0.004
0.850 + 0.009 0.819 + 0.012
mi-Net
0.967 + 0.005
0.835 = 0.010 0.808 + 0.011
MI-Attn
0.972 + 0.007
0.830 = 0.011
0.813 = 0.012
MI-GNN
0.932 + 0.014
0.803 + 0.014
0.781 + 0.019
A.7 4-MNIST-BAGS EXPERIMENT DETAILS
Dataset In the 4-MNIST-Baae Bags experiments, the bag sizes were draw from a normal distribution, with mean of 30 and variance of We used 2500 training bags, 1000 validation bags, and 1000 test bags. The instances in the training bags were only drawn from the original MNIST training split, and the instances the validation and test bags were only drawn from the original MNIST test split, i.e., there was no overlap between training, validation, and test instances. The classes were balanced, so, on average, there were 625 bags per class in the training data, and 250 bags per class in the validation and test data. We normalised the MNIST images using the PyTorch normalise transformation, with a mean of 0.1307 and a stand deviation of 0.3081. No other data augmentation was carried out.
Training The training procedure was the same as for the SIVAL experiments: batch size of one, early stopping with a patience of ten, and maximum of 100 epochs. The training hyperparamater details are given in Table A12. For each model, we first passed the MNIST instances through convolutional architecture to produce initial instance embeddings. This encoder was tuned for each model (i.e., the architecture was fixed, but the weights were learnt). The architecture for this encoder given in Table A A13, and then the model architectures are given Tables A to A17. The encoder produces features vectors with 800 features, therefore the input size to each of the models 800. The model results are given in Table A18.
17

---

Published as a conference paper at ICLR 2022
Table A13: 4-MNIST-Bage convolutional encod- ing architecture. For the convolutional (Conv2d) and pooling (MaxPool2d) layers, the numbers in the brackets are the kernel size, stride, and padding.
Table A12: 4-MNIST- Bags hyperparameters.
Model
LR
WD
DO
MI-Net
x 10
1 x 10-3
0.3
Layer Type
Input Out
mi-Net
1 X 10-4
x 10-1 0.3
MI-Attn
10-4
1 10
0.15
Conv2d(5, 1, 0) ReLU
20
MI-GNN
X 10-3
1 10-5 0.3
2
MaxPool2d(2, 2, 0) DO
20
20
3
Conv2d(5, I, 0) ReLU
20
50
4
MaxPool2d(2. 2, 0) DO
50
50
Table A15: 4-MNIST-Bags mi-Net architecture.
Table A14: 4-MNIST-Bags MI-Net architecture.
Layer
Type
Input
Output
Layer
Type
Input Output
FC + ReLU + DO
800
512
FC ReLU + DO
800
128
2
FC + ReLU DO
512
128
2 FC + ReLU + DO
128
512
FC + ReLU + DO
128
64
3
mil-mean
512
512
4
FC
128
4
4
FC
512
4
5
mil-mean
4
Table A16: 4-MNIST-Bags MI-Attn architecture.
Layer
Type
Input size Output size
1
FC ReLU Dropout 800
64
2
FC + ReLU Dropout 64
256
3
mil-attn(64) + Dropout 256
256
4
FC+ ReLU Dropout 256
64
5
FC
64
Table A17: 4-MNIST-Bag MI-GNN architecture.
Layer
Type
Input size
Output size
FC ReLU Dropout FC + ReLU + Dropout
800
64
2
64
64
3a GNNembed SAGEConv + ReLU Dropout 64
128
SAGEConv + ReLU + Dropout
128
128
SAGECony + ReLU + Dropout
128
128
3b GNN SAGEConv + Softmax
64
gnn-pool FC + ReLU + Dropout FC
128
128
5
128
64
64
4
Table A18: 4-MNIST-Bags model results. The mean performance calculated repeat trainings of each model, and the standard eтror of the mean is given.
Model
Train Aceuracy
Val Accuracy
Test Accuracy
MI-Net
0.995 + 0.001
0.972 + 0.002 0.971 0.002
mi-Net
0.993 + 0.001
0.973 + 0.002
0.974 + 0.002
MI-Attn
0.991 + 0.002
0.967 + 0.003
0.967 0.003
MI-GNN
0.984 + 0.002
0.968 = 0.002
0.966 0.002
18

---

Published as a conference paper at ICLR 2022
A.8 CRC EXPERIMENT DETAILS
Dataset In the CRC dataset, there are 100 microscopy images, and each image is annotated with four classes of nuclei: epithelial, inflammatory, fibroblast, and miscellaneous. Of these 100 images, 50 are in the negative class (no epithelial nuclei), and 50 are in the positive class (at least one epithelial nuclei). Each image 500 x 500 pixels, and was split into 27 27 pixel patches to give 324 patches per slide. The patches were created by applying non-overlapping grid over the image, where each cell of the grid was 27 x pixel region. The alternative would to extract patches centred on each marked nuclei (as done by llse et al. (201 (2018)), however this method then requires the nuclei to be marked for unseen data. This accounts for the difference between our results and the results of Ilse et al. (2018) - extracting the patches using a fixed grid will include patches that do not contain any marked nuclei, increasing the amount and thus making the problem harder to learn. We removed slide background patches by using brightness threshold - any patch with an average pixel value above 230 (using pixel values 0 to discarded
uon-isscrimiatorry data in each bag
255) was discarded. This left average of 264 patches per image, and one image as it had zero foreground patches (this was also manually verified). The images were normalised using the dataset mean (0.8035, 0.6499, 0.8348) and standard deviation (0.0858, 0.1079, 0.0731). During training, we also applied three transformations to patches at random: horizontal flips, vertical flips. and 90 degree rotations. The dataset was separated into train, validation, and test data using 60/20/20 split. This was done with stratified sampling in order to maintain the same data distribution across all splits.
Training The training procedure was the same for the SIVAL and 4- 4-MNIST-Bags experiments: a batch size of one, early stopping with a patience of ten, and a maximum of 100 epochs. The training hyperparamater details experiments, for each model, we first passed the patches through a convolutional architecture to produce initial instance embeddings. The architecture for this encoder is given in Table A20, and then the model architectures are given in Tables A21 to A24. The encoder produces features vectors with 1200 features, therefore the input size each of the models is 1200. The model results are given in Table A25.
given in Table A19. Following the same procedure the 4-MNIST- --agg
Table A20: CRC convolutional encoding archi-
tecture. For the convolutional (Conv2d) and
Table A19: CRC training hyperparameters.
pooling (MaxPool2d) layers, the numbers in the brackets are the kernel size, stride, and padding.
Model
LR
WD
DO
MI-Net 5 x 10-4
x 10-3
0.3
Layer Type
Input Out
mi-Net
5 x 10-4
1 10-2
0.25
Conv2d(4, 1, 0) ReLU
36
MI-Attn
1 10-3
1x 10-6 0.2
2
MaxPool2d(2, 0) DO
36
36
MI-GNN
1x 10-3
1x 10-2
0.35
3
Conv2d(3, 1, 0) ReLU
36
48
MaxPool2d(2, 2, DO
48
48
Table A21: CRC MI-Net architecture.
Table A22: CRC mi-Net architecture.
Layer
Type
Input Output
Layer
Type
Input Output
1
FC + ReLU + DO
1200
64
1
FC + ReLU + DO
1200
64
2
FC ReLU + DO
64
512
2
FC + ReLU DO
64
64
3
mil-max
512
512
3
FC + ReLU + DO
64
64
FC ReLU + DO
512
128
FC
64
2
4
FC ReLU + DO
128
64
5
mil-max
2
2
FC ReLU + DO
64
19

---

Published as a conference paper at ICLR 2022
Table A23: CRC MI-Attn architecture.
Layer
Type
Input size
Output size
FC ReLU + Dropout
1200
64
2
FC ReLU + Dropout
64
64
3
FC + ReLU + Dropout
256
4
mil-attn(125) Dropout
256
256
5
FC
256
Table A24: CRC MI-GNN architecture.
Layer
Type
Input size
Output size
FC ReLU + Dropout
1200
64
FC + ReLU + Dropout
64
128
3a GNNembed
SAGEConv + ReLU + Dropout
128
128
3b GNNeluster
SAGEConv + Softmax
128
1
4
gnn-pool 6 FC + ReLU + Dropout
128
128
128
128
FC
128
2
Table A25: CRC model results. The mean performance was calculated over ten repeat trainings of each model, and the standard error of the mean is given.
Model
Train Aceuracy
Val Accuracy
Test Accuracy
MI-Net
0.880 + 0.041
0.805 + 0.034 0.795 0.042
mi-Net
0.856 + 0.041
0.810 + 0.043 0.795 + 0.049
MI-Attn
0,870 + 0.014
0,860 = 0.017 0.830 + 0.028
MI-GNN
0.791 + 0.039
0.765 = 0.031 0.770 0.032
A.9 ADDITIONAL EXPERIMENTS
In this section, we provide additional sample size experiments for the SIVAL, 4-MNIST-Bags, and CRC datasets for MI-Net (Figure A1), mi-Net (Figure A2), and MI-GNN (Figure A3). We find the trends are relatively consistent across all the models. GuidedSHAP and MILLI are the most consistent and well performing methods, and MILLI is particularly effective on the 4-MNIST-Baur -aggs dataset. One considerable difference the performance of RandomL.MM and Guidedl IIE on the CRC dataset for the MI-GNN model; for all other models, both LIME methods perform poorly on the CRC dataset. Further investigation is required to understand why this happens. However, the LIME methods are still outperformed by MILLI and GuidedSHAP the CRC dataset, even for the MI-GNN model.
SIVAL
4-MNIST-Bace
CRC
0.9
RandomSHAP
NDCGRN
0.8
0.8
0.8
GuidedSHAP
0.7
Ran
0.6
0
IuidedLIME
0.6
MILLI
0.5
0.4
100
300
500
1000
1500
Number Somples
Figure A1: The effect of sample size on interpretability performance for MI-Net.
20

---

Published as a conference paper at ICLR 2022
SIVAL
1-MNIST-DE -Bags
CRC
0.9
1.0
0.9
RandomSHAP
NDCGRN
0.8
GuidedSTIAP
0.7
Ran
0.6
0.7
IME
0.0
MILLI
100
200
300
100
200
300
500
1000
1500
Number of Samples
Figure A2: The effect of sample size on interpretability performance for mi-Net.
SIVAL
4-MNIST Bags
CRC
0.9
RandomSHAP
XDOGRNN
0.8
0.8
0.8
GuidelSHAP
0.7
RandomLIME
GuidedLMME
0.6
MILLI
0.5
0.4
0.6
1000
1500
Samples
Figure A3: The effect of sample size interpretability performance for MI-GNN.
A.10 ADDITIONAL OUTPUTS
In this section, we provide additional interpretability outputs from our studies, similar to the one in Section 5. First, we provide additional outputs for the 4-MNIST-Bags dataset. Then, we show the interpretability outputs for the SIVAL and CRC datasets. The SIVAL outputs created by ranking the instances in bag according to some interpretability function (i.e., by using MILLI), and then selecting the top n, where the known number of key instances in the bag. Then, the corresponding segment for each instance is weighted by the function logz(i 1)-1, where i is the instance's 's position in the ranking (this the same scaling used NDCG@n) The other instances all received a weighting of zero. The brightness of each segment in the output image then corresponds with relative importance according to interpretability method. For the CRC interpretability outputs, the important patches are found by using an interpretability method to output the top patches that support a particular class, where n is the number known important instances for that class. We then highlight these patches while dimming the other patches to produce interpretation of the model's 's decision-mann making..
5
9
/
7
5
/
9
7
Atsention
MILLI 2
MILLIO
Figure A4: The interpretability output for a class two bag on the 4-MNIST-Bags experiment. The show the output of MILLI
top shows the attention value for each instance, and bottom for classes two and zero respectively (green = support, red = refute). We can see that both Attention and MILLI have identified the 9s important, but only MILLI able to also say that the support class two and refute class zero.
21

---

Published as a conference paper at ICLR 2022
8
4
8
4
0
5
Atteution
MILLI
MILLIO
Figure A5: The interpretability output for a class bag the 4-MNIST-Bage experiment. We can see that both Attention and MILLI have identified the 8s as important, but only MILLI is able to also say that the 8s support class and refute class zero. Also note that the colours are less strong for the final 8 the digit is partially obscured, so the model is less confident.
Mond
Coca-Cla
Coca-olla
CocaCCola
Coca-Cola
Figure A6: The interpretability output for two examples of the cokecan class in the SIVAL dataset. The first column shows the original images, the second column the ground truth segments that contain the object, and the final column is the output from MILLI. The interpretations show the model is relying heavily on the colour red an indicator of the object's location, as it also picks up on the red in the background as being important.
22

---

Published as a conference paper at ICLR 2022
Figure A7: The interpretability output for two examples of the dataniningbook class in the SIVAL dataset. Again, the interpretations show the model is relying heavily on the colour orange as an indi- cator of the object's location, as it also picks up on the similar colours the background, including the reflection of the book in the table.
Figure A8: The interpretability output for two examples the goldmedal class the SIVAL dataset. Here, the interpretations show the model is ignoring the gold medal itself, and instead focusing on the red ribbon, i.e., if it were shown the medal without the ribbon, it would likely misclassify it.
23

---

Published as a conference paper at ICLR 2022
WT
WD- &
WD+
Figure A9: The interpretability output for two examples of the wd40can class in the SIVAL dataset. Here, the interpretations show the model predominantly focusing on strong yellow colour at the top of the can, and sometimes picks out the letters and red cap. The rest of the bottle is largely ignored, meaning if the top half were obscured, the model would likely misclassify it.
Figure A10: The interpretability output for an image in the CRC dataset. From left to right, the figures shows: the original image, the image with background patches removed, the ground truth patches for the image's 's label, and the interpretability output from MILLI. In this example, the image class zero (i.e., non-epithelial), meaning the highlighted patches contain mostly fibroblast and inflammatory nuclei. As shown here, MILLI has identified that the model is using the same patches as the ground truth to make a decision, indicating that the model has learnt to correctly identify fibroblast and inflammatory nuclei as supporting instances for non-epithelial images.
24

---

Published as a conference paper at ICLR 2022
The interpretability output for positive image in the CRC dataset. In this example, the ground truth patches all contain at least one epithelial nuclei. MILLI has identified that the model is using most of the same patches the ground truth to make decision, indicating that the model has learnt to correctly identify epithelial nuclei as supporting class one.
Figure All:
Figure A12: The interpretability output for negative image in the CRC dataset. In this example, the patches are sparse there are a lot of background patches that are removed, and the key patches spread out (as opposed being connected in the previous examples). Again, MILLI is able to correctly identify the patches that support the negative class.
Figure A13: The interpretability output for positive image in the CRC dataset. In this example, MILLI has identified some of the ground truth patches, but also additional patches that are not labelled epithelial in the ground truth labelling. As the labelling was exhaustive, these patches may contain epithelial nuclei without being labelled as such. By using MILLI, it is apparent that the model is using these patches in its ceciiionnmakinng therefore further investigation into the types of nuclei in these patches would reveal more information about how the model makes its decisions; something that would not possible without the interpretability output.
25


Reviews:
[{"review_id": "pysH_RKF2V", "reviewer": "Reviewer_xwk6", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "On the positive side, the manuscript is well written and to a good extent its presentation and flow of contents is clear. \n\n- The proposed method is simple and sufficient details are provided regarding its implementation and parameters. In this regard, I would not expect significant difficulties on the re-implementation of the proposed methods by third-parties. \n\n- I appreciate the fact that the complexity of the proposed/discussed methods is provided. This is a detail that is sometime put aside.\n\n- Finally, to the best of my knowledge, this might be one of the first works aiming at the task of explaining MIL methods.\n\nOn the negative side, I have the following concerns with the manuscript:\n\n- While the manuscript covers an evaluation in multiple datasets, it would have been more insightful a more exhaustive analysis considering different MIL scenarios (e.g. different MIL assumptions, MIL regression vs. classification, other data modalities, etc.) see Wang et al., 2020 for reference.\nA more comprehensive analysis would a provided deeper insight on the performance of the proposed method under different scenarios.\n\n- When reporting results in Tables 1-3, more than one parameter is changed for the SHAP-based methods. In the manuscript, it is not properly motivated why this is the case. \n\n- Reported results are not sufficiently conclusive, several trends can be observed in the results reported in Tables 1-3. More concretely, depending on the table different explanation methods lead the results.\nSimilarly, regarding the effect of sample size reported in Figure 3, it is stated that WeightedSHAP seems to be the most sample efficient of the SHAP variants while GuidedSHAP provides diminishing returns. However, these observations does not seem to hold in the other datasets as reported in Figures A13 and 14 from the appendix. Actually if you look at the plots from all the datasets together, it seems that GuidedSHAP provides the best tradeoff.\n\n- The used performance metric seems to rely internally on the surrogate classes (supportive, neutral, refutive) that are also integrated as part of some of the proposed methods. Considering this observation makes me wonder whether the selected metric does work in favor of the proposed method.\nIn this regard, I would suggest to also report results using the perturbation-based method proposed by Samek et al., 2017 which can be applied without the assumption of such surrogate internal classes.  \n\n- Finally, when describing the proposed methods (e.g. Equation 1), it is stated that data is grouped per classes, however, it is not clear where this class information comes from. Could specify whether it comes from the ground-truth class annotations or the predicted classes?\n\nReferences\n- Wojciech Samek  Alexander Binder, Grégoire Montavon, Sebastian Bach, and Klaus-Robert Müller\nEvaluating the visualization of what a Deep Neural Network has learned. TNNLS 2017\n\n- Kaili Wang, Jose Oramas, and Tinne Tuytelaars, In Defense of LSTMs for addressing Multiple Instance Learning Problems. ACCV 2020.", "summary_of_the_paper": "The manuscript aims at the design of a model-agnostic method of the the interpretability of models and methods addressing multiple instance learning (MIL) problem.\nTowards this goal, six explanation methods are proposed, three of which are based on kernel-SHAP.\nThe proposed method is validated on the SIVAL, ColoRectal Cancer and a variant of the MNIST-Bags dataset. ", "summary_of_the_review": "As stated on my review, there are quite some merits regarding the presentation/reproducibility of the manuscript, and the simplicity of the proposed method. \nHowever, I have concerns regarding the validation of the proposed method. On the one hand, it seems to cover a reduced set of MIL scenarios. Therefore, its generality cannot be guaranteed. In addition, at this point, it is hard to assess how conclusive the reported results really are. The fact that some of the proposed methods perform better in one or another dataset, suggests that the overarching goal of the manuscript -  of proposing an explanation method - has not been achieved. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "irGp1Z2UCo6", "reviewer": "Reviewer_smzi", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n1. This paper formulates the interpretability requirements. Important points are 1) defining the conditional contribution for a particular class ; 2) allowing the contribution to be negative, which indicates that the instance refutes some class. \n2. Several straightforward methods and Shapely value based methods are evaluated on three datasets.  \n3. Authors propose a novel weight-sampling strategy to improve sampling in SHAP. \n4. The paper is well written and easy to read.\n\nWeaknesses\n1. I'm not sure whether the interpretability requirements for MIL are original ideas, since MIL is not a new problem. Moreover, these requirements have already been defined and addressed in studies of feature importance for standard single instance models. Computing instance importance and feature importance are similar problems. \n2. Related methods such as LIME[1] should be discussed. LIME is also a model-agnostic local interpretability method. It should be naturally suitable for interpreting MIL models. \n3. Classical MIL datasets (MUSK1, MUSK2, FOX, TIGER, ELEPHANT) used in [2,3] are missed in the experiments. Can authors explain why?\n4. In the first part of Section 5, it's not clear why the demonstrated case shows interactions between instance 8 and 9? And what's the evidence that WeightedSHAP succeeds in taking account of the interaction?\n\n[1] Marco Tulio Ribeiro, Sameer Singh, Carlos Guestrin. “Why Should I Trust You?\": Explaining the Predictions of Any Classifier. \n\n[2] Xinggang Wang, Yongluan Yan, Peng Tang, Xiang Bai, and Wenyu Liu. Revisiting multiple instance neural networks.\n\n[3] Maximilian Ilse, Jakub Tomczak, and Max Welling. Attention-based deep multiple instance learning.", "summary_of_the_paper": "This paper introduces approaches to interpret Multiple Instance Learning (MIL) models. These approaches are designed to quantify the contribution of each instance given a specific class. Authors provide empirical studies about these methods. The single instance evaluation and leave one out method show good performance when instances are independent with each other. The Shapely value based methods are much better when instances have interactions. All these methods exhibit higher interpretability than the inherent interpretations provided by the selected MIL training methods. \n", "summary_of_the_review": "I recommend weak rejection for now, considering the weaknesses listed above.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "SGZUwCb29I8", "reviewer": "Reviewer_MxkS", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is nicely written and easy to follow, but, in my opinion, it totally overlooks existing prior work that should be cited. In particular, I would like to see comparisons with the techniques presented in the following paper: https://www.jmlr.org/papers/volume21/18-811/18-811.pdf It is from 2020 and already mentioned interpretability, and it should be used as a baseline. \n\nThe reason mentioned above is the strongest one to ground my rejection score for this paper. Another one is that SHAP looks like an arbitrary choice from the paper. There are many explanation mechanisms that can be used, and it is not clear why SHAP is the only one you try (besides the simple three methods you introduce at the beginning)\n\nApart from those negative comments, the paper is well written and easy to read. Only one comment regarding the terminology you use in your paper.  The term \"interpretable\" usually refers to the fact that you can look at the model and understand how predictions are computed. Yours is more an \"explainability\" method.", "summary_of_the_paper": "The paper presents six methods for explaining the output of MIL models. The idea is pretty simple but, indeed, effective. The most advanced method is based on SHAP, and it is used to assign to each instance in the bag a value that weights its importance.\nExperiments confirm that the method is able to improve over competitive baselines.", "summary_of_the_review": "The paper presents a SHAP-based explainability method for MIL models. The paper is easy to read, but it lacks the sufficient comparison with proper baselines. I mentioned a paper that should be used to build a stronger baseline.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "2urEOQjb5s", "reviewer": "Reviewer_F67D", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper presents a suite of sampling-based approaches to compute Shapley values for interpretable multiple instance learning. The focus of the paper is on answering \"which\" questions (for identifying key instances in a bag) and answering \"what\" questions (for identifying which positive classes are supported by key instances).\n\nThe motivation for this paper is solid as there is not much work in interpretable MIL, especially with more than one positive class. The algorithms make technical sense and I can see them being useful in real-world applications such as colon cancer classification.\n\nUnfortunately, the paper suffers from a number of weaknesses. The main problem with the paper is its lack of novelty. The instance attribution ideas in Section 3.2 are fairly obvious things to do. Secondly, the key ideas for dealing with instance interactions (e.g. Shapley values for explainability, sampling for Shapley value computation) come from existing work and the authors are performing minor tweaks to them through their weighted sampling technique.\n\nFurthermore, the evaluation could be much stronger. There are only three datasets used in the evaluation section, with two of them having independent instances. It would help to have more real-world datasets involving instance interactions. The authors use the 4-MNIST-Bags dataset as an example of a dataset with instance interactions. The positive classes are artificially generated, which is acceptable for evaluation, but the authors could produce many more variants of the 4-MNIST-Bags for a more thorough evaluation.\n\nThere are two terms used in this paper that are used somewhat differently in the context of the wider machine learning literature and this use is confusing. Explicitly clarifying these terms would greatly improve the text.\n\nThe first is \"interpretability\". From the text, I think the authors intend interpretability to be the correct identification of which are the key instances as well as the correct identification of what classes the key instances support. This use of the term interpretability different from what is commonly referred to as interpretability in the XAI literature, in which interpretability captures how well a human user can understand why a machine learing algorithm makes a particular prediction. This latter definition of interpretability is a human-centric concept and thus requires a user study to measure it; it is not captured by the NDCG@N metric.\n\nThe second confusing term is \"interaction between instances\". I think the authors are referring to the fact that multiple instances can support different classes. However, other work in the machine learning literature (e.g. Adams and Marlin (2017), Guan et al. (2016)) consider \"interactions\" to be relationships between the instances e.g. the instances are intervals of a time series and are thus correlated with each other.\n\nThis second point of confusion makes it unclear how the sampling approaches in Section 3.3 capture interactions between instances. The random sampling treats each member of the coalition to be independently drawn. The guided sampling adds a weight to the random sampling to provide a bias towards small coalitions, but this weight still treats each member independently. Finally, the weighted sampling relies on independent instance methods to rank. This section of the paper could be improved if the authors could explicitly clarify how the sampling approaches moves away from the independence of instances and capture interactions.\n\nReferences\n\nAdams, R. J. and Marlin, B. M. (2017). Learning Time Series Detection Models from Temporally Imprecise Labels. In Proceedings of the 20th International Conference on Artificial Intelligence and Statistics.\n\nGuan, X., Raich, R. and Wong, W-K. (2016). Efficient Multi-Instance Learning for Activity Recognition from Time Series Data Using an Auto-Regressive Hidden Markov Model. In Proceedings of the 33rd International Conference on Machine Learning.\n", "summary_of_the_paper": "This paper presents model-agnostic Shapley value approaches for interpretable multiple instance learning, with a focus on identifying key instances and identifying which positive classes these key instances support. ", "summary_of_the_review": "The paper addresses the interpretability of multiple instance learning, which could use more attention in the literature, but it has weaknesses in terms of the novelty of ideas, the clarity of terms and the experimental evaluation.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper studies interpretability in multi instance learning (where model is trained with a label provided for a bag of instances). The author proposes model-agnostic weight-sampling strategy to improve sampling in prior methods such as (SHAP), and evaluate their performance on three datasets (and authors provided results on more datasets during rebuttal). 

All reviewers agree the paper is well written and well motivated. The paper presents a simple but meaningful extensions to existing interpretability study and will be helpful for the community. Reviewers had some concerns with the comprehensiveness of the evaluation, the strength of their proposed results, and the originality/novelty of the paper. The authors have provided further experimental results on new datasets as well as additional baselines. Given the study of MIL setting in interpretability is scarce, I am leaning towards the acceptance.
Decision: Accept (Poster)