Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper introduces approaches to interpret Multiple Instance Learning (MIL) models. These approaches are designed to quantify the contribution of each instance given a specific class. Authors provide empirical studies about these methods. The single instance evaluation and leave one out method show good performance when instances are independent with each other. The Shapely value based methods are much better when instances have interactions. All these methods exhibit higher interpretability than the inherent interpretations provided by the selected MIL training methods. 

main_review: Strengths
1. This paper formulates the interpretability requirements. Important points are 1) defining the conditional contribution for a particular class ; 2) allowing the contribution to be negative, which indicates that the instance refutes some class. 
2. Several straightforward methods and Shapely value based methods are evaluated on three datasets.  
3. Authors propose a novel weight-sampling strategy to improve sampling in SHAP. 
4. The paper is well written and easy to read.

Weaknesses
1. I'm not sure whether the interpretability requirements for MIL are original ideas, since MIL is not a new problem. Moreover, these requirements have already been defined and addressed in studies of feature importance for standard single instance models. Computing instance importance and feature importance are similar problems. 
2. Related methods such as LIME[1] should be discussed. LIME is also a model-agnostic local interpretability method. It should be naturally suitable for interpreting MIL models. 
3. Classical MIL datasets (MUSK1, MUSK2, FOX, TIGER, ELEPHANT) used in [2,3] are missed in the experiments. Can authors explain why?
4. In the first part of Section 5, it's not clear why the demonstrated case shows interactions between instance 8 and 9? And what's the evidence that WeightedSHAP succeeds in taking account of the interaction?

[1] Marco Tulio Ribeiro, Sameer Singh, Carlos Guestrin. “Why Should I Trust You?": Explaining the Predictions of Any Classifier. 

[2] Xinggang Wang, Yongluan Yan, Peng Tang, Xiang Bai, and Wenyu Liu. Revisiting multiple instance neural networks.

[3] Maximilian Ilse, Jakub Tomczak, and Max Welling. Attention-based deep multiple instance learning.
summary_of_the_review: I recommend weak rejection for now, considering the weaknesses listed above.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper presents six methods for explaining the output of MIL models. The idea is pretty simple but, indeed, effective. The most advanced method is based on SHAP, and it is used to assign to each instance in the bag a value that weights its importance.
Experiments confirm that the method is able to improve over competitive baselines.
main_review: The paper is nicely written and easy to follow, but, in my opinion, it totally overlooks existing prior work that should be cited. In particular, I would like to see comparisons with the techniques presented in the following paper: https://www.jmlr.org/papers/volume21/18-811/18-811.pdf It is from 2020 and already mentioned interpretability, and it should be used as a baseline. 

The reason mentioned above is the strongest one to ground my rejection score for this paper. Another one is that SHAP looks like an arbitrary choice from the paper. There are many explanation mechanisms that can be used, and it is not clear why SHAP is the only one you try (besides the simple three methods you introduce at the beginning)

Apart from those negative comments, the paper is well written and easy to read. Only one comment regarding the terminology you use in your paper.  The term "interpretable" usually refers to the fact that you can look at the model and understand how predictions are computed. Yours is more an "explainability" method.
summary_of_the_review: The paper presents a SHAP-based explainability method for MIL models. The paper is easy to read, but it lacks the sufficient comparison with proper baselines. I mentioned a paper that should be used to build a stronger baseline.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper presents model-agnostic Shapley value approaches for interpretable multiple instance learning, with a focus on identifying key instances and identifying which positive classes these key instances support. 
main_review: The paper presents a suite of sampling-based approaches to compute Shapley values for interpretable multiple instance learning. The focus of the paper is on answering "which" questions (for identifying key instances in a bag) and answering "what" questions (for identifying which positive classes are supported by key instances).

The motivation for this paper is solid as there is not much work in interpretable MIL, especially with more than one positive class. The algorithms make technical sense and I can see them being useful in real-world applications such as colon cancer classification.

Unfortunately, the paper suffers from a number of weaknesses. The main problem with the paper is its lack of novelty. The instance attribution ideas in Section 3.2 are fairly obvious things to do. Secondly, the key ideas for dealing with instance interactions (e.g. Shapley values for explainability, sampling for Shapley value computation) come from existing work and the authors are performing minor tweaks to them through their weighted sampling technique.

Furthermore, the evaluation could be much stronger. There are only three datasets used in the evaluation section, with two of them having independent instances. It would help to have more real-world datasets involving instance interactions. The authors use the 4-MNIST-Bags dataset as an example of a dataset with instance interactions. The positive classes are artificially generated, which is acceptable for evaluation, but the authors could produce many more variants of the 4-MNIST-Bags for a more thorough evaluation.

There are two terms used in this paper that are used somewhat differently in the context of the wider machine learning literature and this use is confusing. Explicitly clarifying these terms would greatly improve the text.

The first is "interpretability". From the text, I think the authors intend interpretability to be the correct identification of which are the key instances as well as the correct identification of what classes the key instances support. This use of the term interpretability different from what is commonly referred to as interpretability in the XAI literature, in which interpretability captures how well a human user can understand why a machine learing algorithm makes a particular prediction. This latter definition of interpretability is a human-centric concept and thus requires a user study to measure it; it is not captured by the NDCG@N metric.

The second confusing term is "interaction between instances". I think the authors are referring to the fact that multiple instances can support different classes. However, other work in the machine learning literature (e.g. Adams and Marlin (2017), Guan et al. (2016)) consider "interactions" to be relationships between the instances e.g. the instances are intervals of a time series and are thus correlated with each other.

This second point of confusion makes it unclear how the sampling approaches in Section 3.3 capture interactions between instances. The random sampling treats each member of the coalition to be independently drawn. The guided sampling adds a weight to the random sampling to provide a bias towards small coalitions, but this weight still treats each member independently. Finally, the weighted sampling relies on independent instance methods to rank. This section of the paper could be improved if the authors could explicitly clarify how the sampling approaches moves away from the independence of instances and capture interactions.

References

Adams, R. J. and Marlin, B. M. (2017). Learning Time Series Detection Models from Temporally Imprecise Labels. In Proceedings of the 20th International Conference on Artificial Intelligence and Statistics.

Guan, X., Raich, R. and Wong, W-K. (2016). Efficient Multi-Instance Learning for Activity Recognition from Time Series Data Using an Auto-Regressive Hidden Markov Model. In Proceedings of the 33rd International Conference on Machine Learning.

summary_of_the_review: The paper addresses the interpretability of multiple instance learning, which could use more attention in the literature, but it has weaknesses in terms of the novelty of ideas, the clarity of terms and the experimental evaluation.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The manuscript aims at the design of a model-agnostic method of the the interpretability of models and methods addressing multiple instance learning (MIL) problem.
Towards this goal, six explanation methods are proposed, three of which are based on kernel-SHAP.
The proposed method is validated on the SIVAL, ColoRectal Cancer and a variant of the MNIST-Bags dataset. 
main_review: On the positive side, the manuscript is well written and to a good extent its presentation and flow of contents is clear. 

- The proposed method is simple and sufficient details are provided regarding its implementation and parameters. In this regard, I would not expect significant difficulties on the re-implementation of the proposed methods by third-parties. 

- I appreciate the fact that the complexity of the proposed/discussed methods is provided. This is a detail that is sometime put aside.

- Finally, to the best of my knowledge, this might be one of the first works aiming at the task of explaining MIL methods.

On the negative side, I have the following concerns with the manuscript:

- While the manuscript covers an evaluation in multiple datasets, it would have been more insightful a more exhaustive analysis considering different MIL scenarios (e.g. different MIL assumptions, MIL regression vs. classification, other data modalities, etc.) see Wang et al., 2020 for reference.
A more comprehensive analysis would a provided deeper insight on the performance of the proposed method under different scenarios.

- When reporting results in Tables 1-3, more than one parameter is changed for the SHAP-based methods. In the manuscript, it is not properly motivated why this is the case. 

- Reported results are not sufficiently conclusive, several trends can be observed in the results reported in Tables 1-3. More concretely, depending on the table different explanation methods lead the results.
Similarly, regarding the effect of sample size reported in Figure 3, it is stated that WeightedSHAP seems to be the most sample efficient of the SHAP variants while GuidedSHAP provides diminishing returns. However, these observations does not seem to hold in the other datasets as reported in Figures A13 and 14 from the appendix. Actually if you look at the plots from all the datasets together, it seems that GuidedSHAP provides the best tradeoff.

- The used performance metric seems to rely internally on the surrogate classes (supportive, neutral, refutive) that are also integrated as part of some of the proposed methods. Considering this observation makes me wonder whether the selected metric does work in favor of the proposed method.
In this regard, I would suggest to also report results using the perturbation-based method proposed by Samek et al., 2017 which can be applied without the assumption of such surrogate internal classes.  

- Finally, when describing the proposed methods (e.g. Equation 1), it is stated that data is grouped per classes, however, it is not clear where this class information comes from. Could specify whether it comes from the ground-truth class annotations or the predicted classes?

References
- Wojciech Samek  Alexander Binder, Grégoire Montavon, Sebastian Bach, and Klaus-Robert Müller
Evaluating the visualization of what a Deep Neural Network has learned. TNNLS 2017

- Kaili Wang, Jose Oramas, and Tinne Tuytelaars, In Defense of LSTMs for addressing Multiple Instance Learning Problems. ACCV 2020.
summary_of_the_review: As stated on my review, there are quite some merits regarding the presentation/reproducibility of the manuscript, and the simplicity of the proposed method. 
However, I have concerns regarding the validation of the proposed method. On the one hand, it seems to cover a reduced set of MIL scenarios. Therefore, its generality cannot be guaranteed. In addition, at this point, it is hard to assess how conclusive the reported results really are. The fact that some of the proposed methods perform better in one or another dataset, suggests that the overarching goal of the manuscript -  of proposing an explanation method - has not been achieved.
review_title: null

Meta-review: The paper studies interpretability in multi instance learning (where model is trained with a label provided for a bag of instances). The author proposes model-agnostic weight-sampling strategy to improve sampling in prior methods such as (SHAP), and evaluate their performance on three datasets (and authors provided results on more datasets during rebuttal). 

All reviewers agree the paper is well written and well motivated. The paper presents a simple but meaningful extensions to existing interpretability study and will be helpful for the community. Reviewers had some concerns with the comprehensiveness of the evaluation, the strength of their proposed results, and the originality/novelty of the paper. The authors have provided further experimental results on new datasets as well as additional baselines. Given the study of MIL setting in interpretability is scarce, I am leaning towards the acceptance.
Decision: Accept (Poster)