Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposed a novel weakly-supervised whole-slide image classification method called SCL-WC, which is constructed based on a domain-specific self-supervised learning feature extractor and a task-specific feature aggregator. The proposed SCL-WC method outperforms state-of-the-art weakly-supervised whole-slide image classification studies over four publicly available datasets for the binary/multiple classification tasks.
strengths_and_weaknesses: Strength：
1. The overall framework is novel and interesting. The use of positive-negative-aware module (PNM) and a weakly-supervised cross-slide contrastive learning (WSCL) module to achieve both intra-WSI local patch separation and inter-WSI global feature contrast is smart.
2. The motivation and introduction are clear.
3. The multi-class classification used in multi-instance learning is novel.

Weakness：
3. The application of contrastive learning, i.e., MoCov3, in histopathological image classification is very common. 
4. The assignment of the three hyper-parameter α，β，γ of the loss function mentioned in this paper lacks sufficient experiments.
questions: 1. Why use swin transformer as the feature encoder?
2. Why use MoCo V3 as self supervised learning? what is the detailed implementation process of Moco V3 in the whole framework ?
limitations: see weakness
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper proposed a novel framework for Weakly-Supervised Whole-Slide Image Classification. It uses a self-supervised contrastive representation encoder trained with large-scale data. It implements three components to aggregate the obtained patch-level representations:  deep attention-based class-specific MIL aggregation(CDA), a positive-negative-aware model that pushes away positive and negative patches given its relevance to the slide-level prediction (PNM), and a weakly-supervised cross-slide contrastive learning with both hard negative and negative sample mining. 

strengths_and_weaknesses: * The paper adapted existing ideas to a different application given the characteristics of the problem they aim to solve.
* The paper is well written and easy to follow. The main contributions in terms of the method are well-explained. 
* The proposed methods are well justified and intuitively understandable. They also achieved very strong performance on multiple datasets. 
* Ablation studies are conducted to help understand the importance of each proposed component. 
  
questions: 1. How is the class-agnostic weight score computed precisely in line 163? 
2. How are the hyperparameters selected ($\alpha$, et.al)?
3. The presentation of tables on page 7 needs refinements. 
limitations: The author mentioned about validating in larger cohorts from real-world clinical setting as future work.
review_title: null

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: Summary:
This paper explores the multiple instance learning problems, i.e. the whole slide image classification task. The authors proposed a weakly supervised WSI classification method using cross-slide contrastive learning. Their presented methodology is dependent on self-supervised feature pre-extraction and task-specific weakly-supervised feature refinement and aggregation for WSI-level prediction. The main goal of the weakly-supervised cross-slide constructive learning module is to pull WSIs with the same disease types closer and push different WSIs away. 


strengths_and_weaknesses: Strengths:
The authors have investigated one of the challenging research questions in the field, i.e. whole-slide image classification. 
The authors have demonstrated the superior performance of their approach to breast cancer and prostate cancer. 

Weaknesses:
The authors have used a pre-trained feature extractor, i.e. swin transformer from unlabeled patches from TCGA and PAIP, however, they have provided neither intuition behind their choice of feature extractor network compared to other approaches such as SimCLR nor provided their class-specific distribution nor patch sizes, etc. It is unclear to the reviewer why the authors have not provided how well the pre-trained feature extractor has worked in comparison to existing pre-trained models. I would suggest authors take a systematic comparison between different components of their model, for example, compare your model, WSCL using different feature extractors, and empirically demonstrate the performance of your choice of feature extractor. Table 6 could have been extended ImageNet (P+N+HN) analysis as well.
In addition, the pre-trained model is on TCGA slides whereas they have evaluated their model performance on Camelyon16, PANDA, BRACS, and DiagSet. While these datasets are relatively small, why there is no evaluation on the TCGA database? The authors could have demonstrated the generalizability of their approach on larger cancer indications from TCGA instead of focusing only on prostate cancer and breast cancer. 
What is the difference between the utilized class-specific deep attention (section 3.3) and the presented attention pooling mechanism in reference 15: Attention-based Deep Multiple Instance Learning? Equation 1 in the manuscript is the same as equation 9 in reference 15. 
In table 1. what are the patch sizes? what is the magnification level of your readings? There is a very limited description regarding the implementation specifications, which would make the readers more confused.
Authors could have benefited from a better comparison of their presented approach with its counterparts. For example, they could have described how their attention pooling is different from other state-of-the-art methodologies. In addition, the comparisons between their evaluations with other studies are limited. It is unclear how well their approach is generalizable to other cancer indications. 



questions: The authors need to evaluate the performance of their approach on different cancer indications. For example, there are 32 different cancer indications. I would like to see if your model still performs well enough on a large proportion of those indications.

In addition, the implementation specifications are missing in the manuscript. 

The methodology part is not well described and I cannot see its comparison with state-of-the-art. I would suggest the authors provide more clarity on what is the main contribution of this work and how it is different from others. 
limitations: 
The authors have used a pre-trained feature extractor, i.e. swin transformer from unlabeled patches from TCGA and PAIP, however, they have provided neither intuition behind their choice of feature extractor network compared to other approaches such as SimCLR nor provided their class-specific distribution nor patch sizes, etc. It is unclear to the reviewer why the authors have not provided how well the pre-trained feature extractor has worked in comparison to existing pre-trained models. I would suggest authors take a systematic comparison between different components of their model, for example, compare your model, WSCL using different feature extractors, and empirically demonstrate the performance of your choice of feature extractor. Table 6 could have been extended ImageNet (P+N+HN) analysis as well.
In addition, the pre-trained model is on TCGA slides whereas they have evaluated their model performance on Camelyon16, PANDA, BRACS, and DiagSet. While these datasets are relatively small, why there is no evaluation on the TCGA database? The authors could have demonstrated the generalizability of their approach on larger cancer indications from TCGA instead of focusing only on prostate cancer and breast cancer. 
What is the difference between the utilized class-specific deep attention (section 3.3) and the presented attention pooling mechanism in reference 15: Attention-based Deep Multiple Instance Learning? Equation 1 in the manuscript is the same as equation 9 in reference 15. 
In table 1. what are the patch sizes? what is the magnification level of your readings? There is a very limited description regarding the implementation specifications, which would make the readers more confused.
Authors could have benefited from a better comparison of their presented approach with its counterparts. For example, they could have described how their attention pooling is different from other state-of-the-art methodologies. In addition, the comparisons between their evaluations with other studies are limited. It is unclear how well their approach is generalizable to other cancer indications.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper presents a weakly supervised contrastive learning approach for the classification of whole-slide images (WSI). Specifically, the proposed method consists of two main steps: 1) a domain-specific self-supervised feature extractor pretrained using a Swin-Transformer, and 2) task-specific feature aggregation modules to achieve distinguishable features that can facilitate both inter-class separability and intra-class compactness for WSI representation. The proposed method has been validated on four publicly available large-scale datasets, across two different organs, including prostate (PANDA and DiagSet) and breast (Camelyon16 and BRACS), for binary and multi-class classification tasks. The proposed technique shows significant improvement over the other weakly supervised benchmark methods across all four datasets, with several ablation experiments that demonstrate the efficacy of the proposed idea. 

Overall, in my opinion, the idea is impressive, and it is novel in the context of weakly-supervised learning for WSI classification.
strengths_and_weaknesses:  — A series of novel feature aggregation modules (such as class-specific deep attention (CDA), positive-negative-aware modeling (PNM), and weakly-supervised cross-slide contrastive learning (WSCL)) presented in this paper is a clever and well-engineered technique that captures both intra- and inter-WSI feature separability. This strategy is very impressive and generic, which can be extended well beyond other weakly-supervised applications such as survival predictions or treatment outcomes in computational pathology. 
          
— Comparison against a robust set of baseline methods (such as CLAM and Trans-MIL) and ablation experiments that have been conducted demonstrates the usefulness of the proposed technique in challenging four weakly-supervised WSI classification tasks.
          
— Overall, the paper is well written and easy to follow the core idea.
questions: I have a few minor concerns with this paper:

— In Section 3.3, Equation 4 (instance-level loss), it is unclear how these pseudo-labels are obtained from the attention network since we don’t have any patch-level labels in a weakly-supervised setting. Further, what is the chosen value of top-K and bottom-k in the experimentation for identifying high-attention and low-attention patches?

— For SSL pretraining, was the method trained on complete TCGA datasets across multiple organs, or was it trained on a specific subset of TCGA. Please, clarify this in the paper.
limitations: Yes, the authors have adequately addressed the limitations and potential negative societal impact of their work
review_title: null

Meta-review: metareview: This work presents a method to obtain slide level representations in computational pathology. The specific contributions of this work are a positive-negative-aware module (PNM) and a weakly-supervised cross-slide contrastive learning (WSCL) module and a loss to encourage intra-WSI local patch separation and inter-WSI global feature contrast. The idea is to use the attention weights in a MIL framework as patch level pseudo labels. These are used to compute "weights" for positive and negative patches (the latter of which is assumed to be significantly larger in number for a given WSI). By using these weights in a contrastive manner to push the representations from positive patches away from those of negative patches, the model learns more effectively since it is less susceptible to the noise from the negative patches.

The reviewers found these contributions novel. During the review, the largest source of concern was along the choices made during the empirical evaluation. Specifically, the manuscript in its current form lacked empirical backing for several of the choices made regarding the neural architecture, the algorithm for self-supervised learning etc. In response to this the authors conducted several different kinds of ablation studies (which were incorporated into the supplement) during the rebuttal process, which other reviewers found convincing as a potential explanation of the outcomes.

Overall, I found the main contributions of this work (leveraging patch level attention weights as psuedo labels) to be an interesting use case for computational pathology to better focus the learning signal on positive patches. My additional comment is that I think the additional ablation experiments (and references) are an important part of the contributions of this work and should be incorporated into the main paper rather than in the appendix (several equations can be compressed to make space in the manuscript in addition to the additional page). 
confidence: Less certain
Decision: Accept