Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper presents a dual-curriculum contrastive Multiple Instance Learning (MIL) method for predicting cancer prognosis in whole-slide histopathology images (WSI’s). The proposed method consists of two key steps to reduce intra-WSI redundancy and increase inter-WSI discrimination: 1) a saliency-guided weakly-supervised multi-scale instance representation learning approach supervised by risk stratification; 2) an instance-level feature aggregation approach via a contrastive-enhanced soft-bag prognosis technique with survival time as supervision. The proposed method has been validated on three publicly available large-scale TCGA datasets: colon adenocarcinoma (COAD), hepatocellular carcinoma (LIHC), and bladder urothelial carcinoma (BLCA). Extensive experiments and comparisons with previous state-of-the-art methods demonstrate the effectiveness of the proposed idea. 
strengths_and_weaknesses: — Overall, in my opinion, the dual-step curriculum learning strategy based on two easy-to-hard curriculums is novel and generic, which is applicable to multi-cancer prognosis prediction tasks in computational pathology. 

— A robust set of baseline comparisons with existing state-of-the-art methods, including clustering, graph network, and multi-instance learning approaches, demonstrates the strength of the proposed idea. Further, an in-depth ablation experiment depicts the importance of encoding multi-magnification instance representations, along with constrained self-attention and the two-tier contrastive learning modules for prognosis prediction across three cancer types.

—  Overall, the paper is well written and easy to follow the methodological contributions of the paper.
questions: There are very few minor concerns with this paper:

— What is the ratio of training and validation sets used across 5-fold cross-validation in experiments.

— How is the best model selected for the validation set to measure the predictive performance on the testing set. 

— Is there any specific reason why three year survival time threshold is adopted? Is there any specific reason behind the 3-year survival time with the three cancer types that have been used in this paper?

— Please clearly state the difference between Figure 4 and Figure 5 (Captions) in the Supplementary Material.

— In Table 2, Please abbreviate the short forms that are used in the paper for better readability

— In Line 241, Section 4.3, there is a typo: Table 3 --> Table 1.
limitations: The authors have adequately addressed their work's limitations and potential negative societal impact.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors propose a framework for cancer diagnosis of WSI by combining a saliency-guided weakly supervised multiscale instance encoding, and soft bag prognosis, using contrastive multi instance learning. The proposed method is tested on three datasets of different cancer tissue and outperforms five other relevant approaches in survival rate prognosis. An extensive ablation study is presented, targeting the contributions of the individual components of the framework.

strengths_and_weaknesses: Strengths:
- The introduction is very well formulated to motivate the issue addressed by this paper.
- The paper is well structured and written.
- The evaluation is extensive with a thorough ablation study.
- The proposed method targets a very relevant problem in biomedical imaging, in which labels are very expensive obtain
- The authors succeed combine a big number of state of the art learning techniques to significantly improve the classification performance 
- The results are verified on three different types of cancer tissue.
- The results can be visualized in an interpretable way.
- The mathematical background and the notation are thorough.

Weaknesses:

- The method targets one particular task of application in particular and is not generally transferable to other tasks, though I believe a number of concepts are.
- The framework consists of a large number of steps and components, each of which have to be tuned for other applications and it is not clear how sensitive the method is to various hyperparameters.

questions: 1.) How were training and test sets chosen? Can WSI from one patient be in the training as well as the test set?
2.) Are the areas highlighted by the network as visualized in Fig. 3 also clinically relevant or would a pathologist identify other areas of cancerous tissue when making a diagnosis?
3.) How stable is the training with respect to the chosen hyperparameters?

limitations: The authors do not address potential negative impacts of their study. Could this framework be applied to remote sensing/satellite data?
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposed an effective framework for Cancer Prognosis Analysis, which outperforms existing methods on three datasets. First, this framework can leverage knowledge from multi-scale information from multi-magnification WSIs. This design is referred to by Saliency-guided Weakly-supervised Instance Encoding with Cross-scale Tiles in the paper. Second, it implements two-tier contrastive learning to enhance intra-bag and inter-bag discrimination.
strengths_and_weaknesses: Strengths:
1. The proposed method implemented several novel ideas that are designed to accommodate particular characteristics of WSI. 
2. The authors conducted comprehensive experiments and validated the effectiveness of the proposed framework. 
3. Ablation studies are also provided to show the impact of each introduced component. 

Weakness:

The paper provides a complicated system but the writing makes it very hard to understand. 

Specific problems include:
* Figure 1 is hard to render by many PDF readers and makes it slow to load. The figure itself is not as informative as the authors may expect due to the overloaded details. Personally, I found Figure 1 in the supplementary materials very helpful in addition to the mathematical explanations. 
* There are many denotations introduced first without any explanation or being explained very late in the paper. It gives the reader hard time understanding the system.  
* The authors may consider using less mathematical denotations. It is not necessary to write down math formulas for many concepts/implementations, such as MLP, attention, et al, which are standard in a deep learning context. The current way in fact makes it too complicated to focus on the novel components proposed in the paper.
* In line 137, it sounds to explain how to obtain w, however, w has never appeared in the part below. 


questions: 1. What does "task-specific" mean in the work? It appears in lines 30, 245, and 262 and is stated as a unique feature of the paper. However, it has never been introduced officially in the paper. Is it relative to using imagenet Pre-trained CNNs as feature extractors? 
2. Where is the architecture of the feature extractor F different from what commonly used ResNet variants? Why do you decide to introduce such architectural-wise changes? How will it perform if we adopt, e.g. ResNet 50, pre-trained on ImageNet, and fine-tune the network? 
3. How to understand the choice of $N_B \ll N_n$? Why does it perform the best to use only 3 instances per bag?   
limitations: I did not find a discussion about the limitation of the proposed methods.
review_title: null

Meta-review: metareview: This is a clear accept. Congratulations!
confidence: Certain
Decision: Accept