Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes interpretable models in multiple instance learning for WSI classification. The authors identify problems in using directly attention mechanism for interpretation. The experiments conducted indicates that high attention values are still generated in negative slides, indicating positive patches. To address this issue, the authors propose Bayesian MIL incorporating 1) probabilistic instance-wise attention module for uncertainty visualization 2) slide-dependent patch regularizer for learning the correct attention distribution 3) an approximate convolutional conditional random field for encoding spatial information. The experiments on two WSI datasets showed that the proposed Bayes-MIL outperforms baselines in path and slide levels metrics. 
strength_and_weaknesses: Strength
The proposed approach improves upon existing MIL approaches on different aspects: It improves the performance at slide and patch level. It improves the interpretability by introducing calibrated model uncertainty.
The paper uses a 10-fold cross validation in the evaluation, which is not frequently seen in the literature due to the computational cost.

Weakness
The scalability of the approach is not discussed. How much Bayes-MIL is slower than baselines in training and inference time. 

clarity,_quality,_novelty_and_reproducibility: The paper is well-written and well-organized. For reproducibility, the code has been provided. The novelty of the paper is in the combination of different elements that improve the SOTA. 

summary_of_the_review: The paper introduces a new approach for MIL problem which brings uncertainty measure in the model and improves interpretability. This is highly relevant for pathology applications where annotations are available only at slide level.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper studies multiple instance learning for WSI analysis. They pointed out that the interpretability of previous attention-based MIL methods for WSI diagnosis was either untrustworthy or unsatisfactory and proposed Bayes-MIL to induce patch-level uncertainty as a new measure of MIL interpretability. To impose constraints derived from the MIL assumption, they further designed a slide-dependent patch regularizer. Last, they applied an approximate convolutional conditional random field to encode the spatial information in WSI. The proposed method was evaluated on public datasets.
strength_and_weaknesses: Strength:
1. The discussed problem is interesting and important for AI-assisted pathology diagnosis.
2. The proposed slide-dependent patch regularizer (SDPR) explicitly encodes the basic assumption of MIL into the training process, which is a long-ignored issue in weakly-supervised learning for WSI analysis.
3. Good performance on two breast cancer datasets in normal/tumor classification tasks.

Weakness:
1. The authors should discuss more about the relationship between the WSI pathology diagnosis and MIL. It is easy to understand that the normal/tumor classification in WSI diagnosis followed the basic assumption of MIL and can benefit from the proposed SDPR. However, other tasks in WSI diagnosis may not meet the same requirements. For example, both tumor subtype classification with more than two classes and tumor stage classification tasks cannot be dealt with the proposed Bayes-MIL and SDPR. Does this mean that the current solution is limited in the simplest positive/negative setting in MIL?
2. As for the tumor/normal classification, it is far more important to localize the ROI in tumor WSI instead of the normal WSI. However, in the Tumor WSIs in both Fig 2. And Fig 8., CLAM seems to provide a more accurate (the highlighted regions have better shape and is denser) visualization than the proposed method. 
3. Is it ok to conclude that in previous works, high attention weights are used to indicate it associated instances are cancerous image patches? If the slide-level prediction for a WSI is normal (without cancer), how do we interpret the high attention weights?
4. From Table 1., we can see that without SDPR and APCRF, the proposed method is not better than previous attention-based works. Does this mean the data uncertainty is not necessary? Can we apply the SDPR and APCRF to previous attention-based works and achieve satisfactory improvement? More ablation study is needed to validate the motivation and effectiveness of the proposed Bayes-MIL.
5. The experiments are limited to the breast cancer datasets and normal/tumor classification task.
6. The std of the experiments is not provided in the paper.

clarity,_quality,_novelty_and_reproducibility: This paper studies an interesting problem in MIL-based WSI analysis. While some details about the method design are not clear and the experimental part is a little weak.  
summary_of_the_review: This paper discusses an interesting problem in MIL-based WSI analysis, while the current version need further improved in regard of clarification and evaluation.

After response: 

The new clarification and experiments have partially addressed my concerns. Therefore,  I would raise my score to 6: marginally above the acceptance threshold.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a method for attention-bases multi-instance-learning (MIL) on digital pathology whole slides (WSI) for tumor classification. The method improves upon SOTA MIL methods on WSI by approaching the problem using a probabilistic framework. A drawback of current attention-based WSI MIL methods is that attention is not focused on positive (tumor) instances only. The proposed approach ensures that attention is focused on positives instances, making the results more interpretable as high attention areas match tumor areas. A Bayesian modeling of MIL provides calibrated uncertainty at the patch level. A variational inference framework regularizer produces dense localization of positive patches. Finally the spatial correlations are captures by a conditional random field CRF. The empirical section compares the proposed approach to SOTA MIL WSI approaches such as DSMIL, TransMIL on two public datasets, showing significant improvements. The results are also ablated showing the contribution of individual modules.

strength_and_weaknesses: Pros:

* The method improves SOTA both on interpretability (the high attention patches correspond to tumor areas) and accuracy.

* The spatial correlation of patches is learned by a CRF, which conforms to the tendency of tumors to locally grow.

* The contribution of each module is clearly reported and show that each of them provides additional improvements.

Cons:

* The feature extractor is simply a resnet50 (imagenet). For the WSI task, other works have shown that self-supervised models trained on WSI data do have superior results.

* There are no speed information given for training and inference. How long does it take to classify a slide on average ? How long does it take to train the feature extractor, the MIL model ? Also a pseudo-code of the algorithm would help.

clarity,_quality,_novelty_and_reproducibility: * The paper is very well written, well organized and clear.

* The paper contains several novel technical contributions within its bayesian attention-MIL framework for WSI.

* The author provide the codes, which should help reproduce the approach

summary_of_the_review: A well-written paper that proposes a novel way to improve on interpretability of WSI classification, while also improving SOTA accuracy on 2 large public datasets.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper presents a new approach for multiple instance learning by integrating uncertainty into MIL. 
The strength: 
1) It has two main novelties to encode the underlying logic of MIL into the training process and encode spatial information via approximation to convolutional CRF.
2) The results justify the effectiveness of the proposed methods.
Weakness: The validation is done in digital pathology and its usefulness in other areas is not clear.

 

justification_for_why_not_higher_score: While the reviewers agreed that the paper has sufficient novelty, they also comment that the paper is not an outstanding one that might attract many researchers. The applied area is relatively narrow.
justification_for_why_not_lower_score: The rebuttal has addressed the reviewers' concerns and there is a consensus that the paper has sufficient novelty.
Decision: Accept: poster