Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors utilization whole slide image (WSI) classification as a classic instance and bag classification problem as most tasks based on whole slide images have slide-level (bag) classification labels and small patches which compose the slides (instances) for the the instance labels are unknown.  This formulation is necessary because WSI contain too much data to be process on a single GPU. Multiple instances learning paired with an aggregator function, like RNN, has been proven to deliver clinical level results, most classically for "tumor/not tumor" tasks.  The authors suggest that these method are limited in tasks where instance level classifications are dichotomized as challenging and not challenging and that certain bags with few levels of challenging instances can be false negatives.  All of this is a set up for their model which benefits from work in the field of knowledge distillation.   Knowledge distillation is a pairing of two models labeled student and teacher, where the teacher network is a MIL-type model using bags of instances with attention and the the student is an instance level classifier that uses all instances in negative bags as negatives and instances with "positive attention" from the teacher network labeled as positives (authors call them soft pseudo labels).  The authors also employ a "Hard Positive Instance Mining" HPM strategy whereby the instances where are predicted to be positive with high probability are excluded from the teacher training to make teacher task more challenging. The authors then implemented the strategy using synthetic datasets with CIFAR-10 and CRC-100K datasets with instance labels.  Camelyon16 was used as a real cancer dataset with both instance and bag level label. Finally, the model was trained and tested on real-world datasets without known instance level classifications, TCGA NSCLC dataset and and in house cervical cancer dataset.  Using both ABMIL and DSMIL, adding knowledge distillation improved overall bag level classification performance as measured by AUC. 
strengths_and_weaknesses: This work is very strong because the authors have properly formulated the many challenges of whole slide image classification and incorporated emerging methods from the field of knowledge distillation/transfer learning to improve upon existing state of the art methodologies in WSI classifications.  From my own work, I have observed that attention maps derived from MIL training provide what appears to be very useful information but that information, as far as I know, has not been effectively harnessed in a practical way.  By formulating the problem with interactions between student and teacher networks, the bag level classifier can provide soft labels for the classification of tiles while the classification results from the student network can be harnessed to "stress test" the bag level classifier.  

While the authors describe in detail the concept of HPM, the study and results are very weakly developed and presented.  Thus I think that perhaps including this as part of the work is a bit premature and the remaining work to my eyes, stands on its own two feet.  
questions: This method appears to heighten the negative consequences of incorrectly labeled data (specifically true positives false labeled negative) because those instances labeled negative are going to adversely train the student network. Have you considered pairing this work with networks trained to catch this situation?
limitations: The major weakness of the paper is that the methods has not been used on well-benchmarked classification tasks.  For example, does this strategy actually enhance methodologies for which MIL+RNN already deliver clinical grade results or is this method only going to improve results for which the dataset is small and/or the task is not clinically significant.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This work presents a method to improve instance classification in whole slide images (WSI) using only class labels of WSI. To this end, the authors consider a knowledge distillation method where a teacher and a student model learn from each other alternatively. The teacher model is a bag (WSI) classifier while the student model is an instance classifier. The knowledge transfer between the two models is instance score attention which indicates the contribution of the instance to the bag prediction. Experimental validation was performed over 2 synthetic datasets and 3 real-world datasets for histology data, and the proposed method outperforms several SOTA methods.
strengths_and_weaknesses:  Strengths:
+ The paper is clearly written and organized. Their problem statement and the proposed approach are clearly described. 
+ The paper discusses an important application that is improving the instance classification in WSI data.
+ The empirical results show the benefit of the method, and some ablation studies are provided.

Weaknesses:
 - On page 2, lines 63-66, the authors mentioned that using soft labels as they did helps to alleviate the noisy pseudo-labels used in other methods. It is not clear how it is the case. From the description of the method, the pseudo-labels in this work change every SGD step since the network that builds them changes constantly. Therefore, they are noisier, especially at the beginning of the training. It is not clear why the authors claimed that their pseudo-labels alleviate the noise issue. In addition, there is no empirical evidence to support this claim.
 - On page 2, lines 70-72, the authors mentioned that sharing the feature extractor between student and teacher enhances knowledge distillation. It is not clear why it is the case.
 - In the entire paper, the pseudo-labels were described as the attention scores. It is not until eq.12 (page 6) that the authors describe the soft labels as being single scores that simulate probability. The text should be clear about this detail from the outset. In addition to fig.2,  there should be an operation after attention scores come out from the teacher, before being used by the student’s CE loss.
 - Hard positive instance mining (hpm) proposed in this paper is similar to hard mining in class-activation maps (CAMs). In the latter, strong activations are suppressed, thereby pushing the network to seek more positive regions. The same here, the authors drop high positive instances simulating suppression. Therefore, the proposed mining strategy is not really new; and literature discussion is necessary. See an example for CAMs ib Wei, Y., Feng, J., Liang, X., Cheng, M.-M., Zhao, Y., and Yan, S. (2017). Object region mining with adversarial erasing: A simple classification to semantic segmentation approach. CVPR 2017.
- The main issue in this work is that it is unclear why there is an improvement? The authors are not using additional supervision. In fact, the pseudo labels are expected to be noisier and change constantly. It is unclear why training a student in parallel helps instance classification even though there are no instance labels. Authors are encouraged to provide further analysis to explain the observed improvement. If I had to guess, the main element that improves performance is in eq.12, in particular the zero-label of instances in negative bags. Authors explicitly provided instance labels but only for negative instances, giving their model a large advantage. Knowing what is not positive helps the model. 
- Finally, there is an important element missing in the experimental section: the classification accuracy per type of instance (positive/negative, ...). We do not know what pushed the AUC at instance-level to improve. Is it improving the positive instances or negative instances? My guess is the latter helps more since labels of negative instances are explicitly used.
-  The code was not made available. Although the method is well described, there is a concern that the results in this paper would be difficult for a reader to reproduce.
questions: Can you please clarify what is the 'norm' in Equation 6?
limitations: Please refer to the weaknesses described in Strengths And Weaknesses.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper presents a novel method to train a teacher-student attention-MIL model with weak supervision in a way that the student can be used to improve the teacher. Dubbed WENO, the method shares an encoder between the teacher and student model and while the teacher's bag-level attention scores are used to train the student (distillation), the student's instance model is used to select hard examples for the teacher's bag. This is designed to avoid a common pitfall of MIL models where the teacher tends to use only easy-to-classify positive instances from the bag and becomes weak at classifying hard examples. The method can be added to existing attention-based MIL methods. The method is evaluated on 2 synthetic and 3 real-world datasets of WSI histology images, and is using 2 SOTA MIL methods (ABMIL and DSMIL). The method is shown to outperform SOTA by a significant margin on datasets where the prevalence of the positive class is low, and by a small margin for real-world datasets. An ablation study shows that that each key part of the method (distillation, shared-encoder and hard-example-selector) all participate in improving the baseline.

strengths_and_weaknesses: strengths:
- the method combines for the first time multi-instance-learning (MIL) with weakly-supervised knowledge distillation.
- the method addresses the issue with attention-based MIL methods where the teacher's performance stops improving after a while and even decreases because it focuses more and more on a few easy-to-classify examples. The proposed HPM (hard-positive-mining) module to alleviate this issue is novel and uses the student's predictions after a set number of epochs to select hard examples for the teacher's bag.
- the method improves upon SOTA for 3 public datasets of WSI histology. It comes ever-closer (within 3%) to fully-supervised models trained with instance-level labels (expensive to obtain).
- the ablation study is useful and convincing.
- the method is not specific to WSI histology images and can be applied to any MIL setting
- the method does not affect inference complexity, only training is affected. This is a key factor in most real-world applications.
- the paper is very well written and clarity is excellent

weaknesses:
- the achieved improvement over SOTA is relatively small for the real-world datasets.
- hyperparameter tuning has to be performed for the HPM module to determine the best threshold. This is discussed in the supplemental material. However, another hyperparameter, the number of epochs after which to bring the HPM module online is not discussed.

questions: no questions.
limitations: Beyond the author-stated limitation related to hyperparameter tuning, i see no further limitations of the approach.
review_title: null

Meta-review: metareview: This submission was reviewed by three reviewers. All three reviewers provided detailed comments during the review period. The authors provided detailed responses to the initial set of reviews. The rebuttals lead to improved scores of some reviewers while other reviewers confirmed that their concerns have been addressed. Given the above evaluations and interactions, an accept is recommended.
confidence: Certain
Decision: Accept