Title: Leveraging Tumor Heterogeneity: Heterogeneous Graph Representation Learning for Cancer Survival Prediction in Whole Slide Images
Venue: NeurIPS 2024
Accepted: yes
Keywords: Whole Slide Image

Survival Prediction

Tumor Heterogeneity

Heterogeneous Graph

Graph Convolutional Network
Abstract: Survival prediction is a significant challenge in cancer management. Tumor micro-environment is a highly sophisticated ecosystem consisting of cancer cells, immune cells, endothelial cells, fibroblasts, nerves and extracellular matrix. The intratumor heterogeneity and the interaction across multiple tissue types profoundly impacts the prognosis. However, current methods often neglect the fact that the contribution to prognosis differs with tissue types. In this paper, we propose ProtoSurv, a novel heterogeneous graph model for WSI survival prediction. The learning process of ProtoSurv is not only driven by data but also incorporates pathological domain knowledge, including the awareness of tissue heterogeneity, the emphasis on prior knowledge of prognostic-related tissues, and the depiction of spatial interaction across multiple tissues. We validate ProtoSurv across five different cancer types from TCGA (i.e., BRCA, LGG, LUAD, COAD and PAAD), and demonstrate the superiority of our method over the state-of-the-art methods.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors present ProtoSurv, a model that leverages heterogneous graph representation learning to predict survival risk. Their model can be decoupled into two basic submodules: 1)Structrure view module whish is responsible for injectiing the topological ingormation of the wsi into the model and the 2) histology view module  which incorporates pathological priors reflecting tumor heterogeneity. The authors conducted experiments across five WSI datasets, demonstrating superior performance, supported a series of by ablation studies.\n\nSTRENGTHS:\n- The paper is clear and well-written.\n- The methodology that is introduced is novel, tackling the inherent heterogeneity of pathology images\n- There is a detailed series of abalation studies justifying the use of each component in the model\n\nWEAKNESSES:\n- The prototyes that are used in histology view are not learnt, but are rather selected before training based on prior knowledge, potentially hindering the adaptability of the model since preselected prototypes may not capture the full variability or complexity of the data. This could also introduce generalisation issues because the model may struggle to generalize to new, unseen data that differ significantly from the prototypes.\n\nQUESTIONS:\n- Andrew H. Song et al. have recentrly published PANTHER [Morphological Prototyping for Unsupervised Slide Representation Learning in Computational Pathology](https://openaccess.thecvf.com/content/CVPR2024/papers/Song_Morphological_Prototyping_for_Unsupervised_Slide_Representation_Learning_in_Computational_Pathology_CVPR_2024_paper.pdf)  that explores a similar idea of extracting morphological signatures using protypes from WSIs in an unsupervised way showcasing superior results. Have you considered comparing PANTHER to your model?\n\nLIMITATIONS:\n- Apart from the comparisons with SOTA methods, it would be interesting to shift the focus also to the interepretability of the model. The prototypes are instrumental in enhancing the model's performance, offering insight into how prototype-guided decisions contribute to the predicted outcome. There is only one attention map in the suplementary material.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper analyzes the limitations of the existing MIL method in survival prediction with WSIs 1) overfitting, 2) numerous redundant and irrelevant instances, and 3) insufficient exploration of the interaction between local, regional features and global contextual features in WSI. To address these issues, the paper proposes Multiple Instance Learning with Hierarchical Graph Transformer over Super-Pixel (HGTSP). HGTSP consists of three novel modules 1) a Pixel-based Pseudo-Bag Division (SPBD), 2) a Super-Pixel Region Sampling (SPRS), and 3) a Hierarchical Graph Transformer (HGT). The experiments on TCGA verify the effectiveness of HGTSP. However, the modules proposed in this paper only partially correspond to the issues it claims to solve. In addition, HGTSP contains a large number of hyper-parameters, which makes it challenging to apply this method to new datasets quickly.\n\nSTRENGTHS:\nThe limitations of previous methods have been well analyzed: 1) overfitting, 2) numerous redundant and irrelevant instances, and 3) insufficient exploration of the interaction between local, regional features and global contextual features in WSI. To address these limitations, this paper proposes 1) a Pixel-based Pseudo-Bag Division (SPBD), 2) a Super-Pixel Region Sampling (SPRS), and 3) a Hierarchical Graph Transformer (HGT). The experiments on TCGA verify the effectiveness of HGTSP.\n\nWEAKNESSES:\n1.\tThis paper claims to address the overfitting via SPBD in the abstract. However, according to the introduction, SPBD is mainly used to address the risk of pseudo-bag mislabeling and inconsistency between original bags and pseudo-bags. Therefore, it would be better to refine the claims in the abstract.\n2.\tThere are a large number of hyper-parameters. According to the appendix, these hyper-parameters highly influence the performance of HGTSP.\n\nQUESTIONS:\n1.\tAs HGTSP combines multiple techniques, including K-means clustering, adaptive sampling, and Graph Transformer, comparing the time required for inferring the same number of samples is more appropriate than only comparing FLOPs and the number of parameters.\n2.\tIs there a method to automatically set hyper-parameters?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-12"}

comments: SUMMARY:
The authors proposed ProtoSurv, a graph model for WSI survival prediction. The key contribution is learning different prototypes for each node type, and aggregating nodes using cross attention and learned prototypes.

STRENGTHS:
- Outcome prediction for cancer patients is a very relevant and important problem and has a large impact. Recently, there has been a lot of interest in the field of computational pathology to solve this problem.
- ProtoSurv has a good motivation to identify and aggregate based on prototypes, which provides some inherent interpretability to the model.
- ProtoSurv seems to outperform various baselines reported by the authors.

WEAKNESSES:
- Having a fixed set of hand crafted node types and a fixed number of multi-prototypes is limiting, and restricts the model’s ability to learn from previously unknown factors / phenotypes that may contribute to patient outcome.
- Evaluation benchmarks should include popular aggregation baselines, including ABMIL (used by UNI authors for benchmarking) and TransMIL.
- Ablation studies is not complete. The authors should consider benchmarking: 
  - Use UNI features directly for prototype learning, without considering node types;
  - Removing graph network, or replacing it with a transformer.

QUESTIONS:
Please address the points in the weakness section.

LIMITATIONS:
Limitations are addressed in the paper.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-07-12

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces ProtoSurv, an algorithm which performs survival prediction for Whole Slide Images (WSIs) of tumour samples, by taking into account the interaction between different tissue types and tumour heterogeneity. ProtoSurv proposes leveraging prior tissue knowledge by constructing a graph where the nodes represent patches and edges represent spatial relationship. The attributes of the nodes correspond to embedded feature representation of the patch, as well as a tissue category label. This tissue category label is assigned by a finetuned feature extractor (UNI), which was previously modified by adding a classifier head and training it on patches with know tissue type labels. Once the graph is initialised, the pipeline employs a dual-view architecture, composed of a Structure view and a Histology view. In the Structure view, the graph with its embedded feature representation is passed through a GCN (which correspond to the Patch-GCN architecture) and a final representation H is stored. In the Histology view multiple prototypes are learned for each tissue category, aiming to capture tissue heterogeneity by using cross-attention between the initial feature vector and the prototypes. Cross-attention is then also employed to guide the fusion of the information from the Structure and the Histology view. The loss function employed during training correspond to composition of the Cox regression loss + a modified compatibility loss + an orthogonality loss. The pipeline is applied to 5 TCGA cancer datasets and compared to several SOTA baseline models. The results are presented using C-index. There is ablation on different model components, the number of tissue categories, the feature extractor used and number of prototypes. The results provided show the algorithm outperforms against the baseline models.\n\nSTRENGTHS:\nThis paper presents an interesting dual-stream architecture for cancer survival prediction, combining a spatial graph based approach, with a prototype learning module. Overall the paper is well structured, and the incorporation of prior tissue knowledge to guide cross-attention is a nice contribution which grounds the model in clinical understanding. The method is tested comprehensively on 5 benchmark methods and ablation is provided on the different components of the model. The results obtained show improvement in Survival prediction across the datasets.\n\nWEAKNESSES:\nBelow I list what I perceive to be the main weaknesses in this paper:\n\n1 - This is not a heterogeneous graph neural network: the edges in the graph are created based on spatial proximity of patches, regardless of their tissue type, and while each node has two types of attributes (Feature vector and Tissue category label) the tissue label isn't actually used in the GNN, which operates on a standard homogeneous graph structure. I find this problematic because its a significant misrepresentation of the core methodology in my opinion. Instead, this is a dual stream architecture that processes the data in two different ways (Patch-GCN + prototype learning), before combining them using cross-attention. Given this fact it would be good to compare how this method compares to a simple ensemble of Patch-GCN and a prototype-based model to verify model performance against this baseline. \n\n2 - As this pipeline relies heavily on cross-attention, both in the histology view and in the fusion module, I think the authors should discuss computational requirements and scalability of the model to large WSIs, which is an important consideration in the healthcare setting. \n\n3 - The ablation on the PGF module is unclear: it seems the PGF module is replaced with an aggregation method from Dong et al., where the order of Q and KV in the cross-attention mechanism is swapped. This doesn't completely remove the fusion between the two views, rather it changes how the fusion is performed and hence does not provide a clear understanding of the individual contribution of the PGF module. Maybe adding a comparison to a simple concatenation and classification could add clarity to this point.  \n\n4 - The authors claim improved interpretability due to the use of prototypes and they have an interesting Figure 10 in the Appendix, which I think should be referenced in text. It would be good to clarify the model obtains attention maps per category and per prototype, and if possible include the Figure in the main body of the text as this would back the improved interpretability claim. \n\n5 - One important comment is that the structural view doesn't simulate viewing at multiple magnification or scales as stated in the text, rather it merely extends the receptive field. This provides multi-hop neighbourhood information, but not multi-scale information.\n\nQUESTIONS:\nI have addressed my questions in the Weaknesses section. Below are some general comments:\n\n- Line 1 - \"Tumors are\"\n- Line 18 - slide\n- Line 21 - molecular alterations\n- Line 22 - no need to redefine WSIs acronym\n- Line 24 - you could cite Ilse et al. 2018 here.\n- Line 26 - In MIL aggregation to bag-level representation can be done using non-learnable or learnable aggregation layers. MIL methods which use learnable aggregation do learn about interrelationship among instances, depending on the approach employed. \n- Line 29 - I don't agree with this statement, subtyping and staging can absolutely require a holistic view of the tumour micro-environment. \n- Line 47 - I would qualify this statement: \"it could be affecting\" - there is a lot of research and debate the performance of GNNs on homophilous vs heterophilous graph. For example see [1]. \n- Line 56 - 59 - I don't find these sentences very clear, maybe you could reformulate them?\n- Line 75 - I imagine most approaches using CNNs would also be employing a MIL based approach. \n- Line 80 + 87 - \"context-awareness\" \n- Line 93 - This was done before HEAT, see [2]\n- Line 106 - See also [3]\n- Line 114 - $e_{i,j}$\n- Line 115 - \"where $x_i$ is the feature ...\" ?\n- Line 130 - \" by the current consensus as* highly relevant\"\n- Line 213 - \"We employ*\"\n- Line 225 - \"given the morphological alterations found in frozen sections*\"\n- Table 1 - underlined*\n- Line 310 - \"In our model, we incorporate five tissue categories based on pathological knowledge of prognosis-related tissues.\"\n\n1 - [1] Platonov etal., A critical look at evaluation of GNNs under heterophily: Are we really making progress, ICLR, 2023.\n \n2 -  [2] Pati et al., Hierarchical graph representations in digital pathology, MIA, 2022. \n\n3 - [3] Yu et al., Prototypical multiple instance learning for predicting lymph node metastasis of breast cancer from whole-slide pathological images, MIA, 2023.\n\nLIMITATIONS:\nThe authors mention an important limitation, which is that the tissue categories are fixed and obtaining the labels remains an obstacle. I would expand on this and mention the method relies on a pre-trained tissue classifier, but there's limited discussion on how errors in this classification might impact the overall performance. I also think the heavy use of cross attention mechanism used could introduce problem for scaling to a clinical setting.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-13"}