Deep Biological Pathway Informed Pathology-Genomic Multimodal Survival Prediction
Authors: Lin Qiu
Aminollah Khormali
Kai Liu
Venue: Submitted to ICLR 2023 (2023)

Abstract: The integration of multi-modal data, such as pathological images and genomic data, is essential for understanding cancer heterogeneity and complexity for personalized treatments, as well as for enhancing survival predictions. Despite the progress made in integrating pathology and genomic data, most existing methods cannot mine the complex inter-modality relations thoroughly. Additionally, identifying explainable features from these models that govern preclinical discovery and clinical prediction is crucial for cancer diagnosis, prognosis, and therapeutic response studies. We propose PONET- a novel biological pathway informed pathology-genomic deep model that integrates pathological images and genomic data not only to improve survival prediction but also to identify genes and pathways that cause different survival rates in patients. Empirical results on six of The Cancer Genome Atlas (TCGA) datasets show that our proposed method achieves superior predictive performance and reveals meaningful biological interpretations. The proposed method establishes insight on how to train biological informed deep networks on multimodal biomedical data which will have general applicability for understanding diseases and predicting response and resistance to treatment.

Under review as a conference paper at ICLR 2023
DEEP BIOLOGICAL PATHWAY INFORMED PATHOLOGY- GENOMIC MULTIMODAL SURVIVAL PREDICTION
Anonymous authors Paper under double-blind review
ABSTRACT
The integration of multi-moda data, such pathological images and genomic data, is essential for understanding cancer heterogeneity and complexity for personalized treatments, as well for enhancing survival predictions. Despite the progress made in integrating pathology and genomic data, most existing methods cannot mine the complex inter-modality relations thoroughly. Additionally, identifying explainable features from these models that govern preclinical discovery and clinical prediction is crucial for cancer diagnosis, prognosis, and therapeutic response studies. We propose PONET- a novel biological pathway informed pathology-genomt deep model that integrates pathological images and genomic data not only to improve survival prediction but also to identify genes and pathways that cause different survival rates in patients. Empirical results on six The Cancer Genome Atlas (TCGA) datasets show that our proposed method achieves superior predictive performance and reveals meaningful biological interpretations. The proposed method establishes insight how train biological informed deep networks on multimodal biomedical data which will have general applicability for understanding diseases and predicting response and resistance to treatment.
INTRODUCTION
Manual examination of haematoxylin and eosin (H&rE)-stained slides of turourr tissue by pathologists
currently the state-of-the-art for cancer diagnosis (Chan 2014). The recent advancements in deep learning for digital pathology have enabled the use of whole-slide images (WSI) for computational
image analysis tasks, such as cellular segmentation (Pan et al.] 2017 Hou et al. 2020). tissue classification and characterisation (Hou et al.) 2016; Hekler et al. 2019: lizuka al. 2020). While
H&EE slides are important and sufficient to establish a profound diagnosis, genomics data can provide deep characterisation of the tumour the molecular level potentially offering the chance for prognostic and predictive biomarker discovery.
Cancer prognosis via survival outcome prediction is standard method used for biomarker discovery. stratification of patients into distinct treatment groups, and therapeutic response prediction (Cheng al. 2017 Ning et al. 2020). WSIs exhibit enormous heterogeneity and can be large as 150,000 150,000 pixels. Most approaches adopt two-stage multiple instance learning- -bain (MIL) approach for representation learning of WSIs, which: 1) instance-level feature representations are extracted from image patches in the WSI, and then 2) global aggregation schemes are applied to the bag of instances to obtain WSI-level representation for subsequent supervision (Hou et al., 2016 Courtiol et al. 2019 Wulczyn et al. 20200 Lu et al.] (02T). Therefore, multimodal survival prediction faces an additional challenge due to the large data heterogeneity gap between WSIs and genomies, and many existing approaches use simple multimodal fusion mechanisms for feature integration, which
prevents mining important multimodal interactions (Mobadersany et al. 2018 Chen et al. 2022b a).
The incorporation of biological pathway databases in model takes ad advantage of leveraging prior biological knowledge so that potential prognostic factors of well-known biological functionality can be identified (Hao et al. 2018). Moreover, encoding biological pathway information into the neural networks achieved superior predictive performance compared with established models (Elmarakeby
al. 2021)
Based on the current challenges in multimodal fusion of pathology and genomics and the potential prognostic interpretation link pathways and clinical outcomes in pathway-base -based analysis,

---

Under review as a conference paper at ICLR 2023
30B
+
AC
Giess
Mulicooki Segranee
Partwey
Special Pailway
Dats embolding
Hierarchical multimods! ftasion
Interpretattion
Figure I: Overview of PONET model.
propose novel biological pathway informed pathology genomic deep model, PONET, that uses H&E WSIs and genomic profile features for survival prediction. The proposed method contains four major contributions: 1) PONET formulates biological pathway informed deep hierarchical multimodal integration framework for pathological images and genomic data; 2) PONET captures and cross modality relations among different data
diverse and comprehensive modality-specine sources based on factorized bilinear model and graph fusion network; 3) PONET reveals meaningful model interpretations on both genes and pathways for potential biomarker and therapeutic target discovery; PONET also shows spatial visualization of the top genes/pathways which has enormous potential for novel and prognostic morphological determinants; 4) We evaluate PONET six public TCGA datasets which showed superior survival prediction comparing to state-of-the-al methods. Fig. shows our model framework.
RELATED WORK
Multimodal Fusion. Earlier works on multimodal fusion focus on early fusion and late fusion. Early fusion approaches fuse features by simple concatenation which cannot fully explore intra-modality dynamics (Wöllmer et al. 2013 Poria et al., 2016 Zadeh et al.) 2016). contrast, late fusion fuses different modalities by weighted averaging which fails to model cross-modal interactions
(Nojavanasghari et al. 2016 Kampman et al. 2018). The exploitation of relations within each modality has been successfully introduced cancer prognosis via bilinear model (Wang et al. 2202bb and traph-baeed model (Subramanian et al. (022)). Adversarial Representation Graph Fusion (ARGF) (Mai al. 2020) interprets multimodal fusion as hierarchical interaction learning procedure where firstly bimodal interactions are generated based unimodal dynamics, and then trimodal dynamics are generated based on bimodal and unimodal dynamics. We propose new hierarchical fusion framework with mppecic and cross-modality attentional factorized bilinear modules to mine the comprehensive modality interactions. Our proposed hierarchical fusion framework different from ARGF in the following ways: 1) We take the sum of the weighted modality-speciffe representation as the unimodal representation instead of calculating the weighted average of the modality-specifee representation in ARGF; 2) For higher level's fusion, ARGF takes the original embeddings of each modality as input while we use the weighted modality-specific representations; 3) We argue that ARGF takes redundant information during their trimodal dynamics.
Multimodal Survival Analysis. There have been exciting attempts on multimodal fusion of pathology and genomic data for cancer survival prediction (Mobadersany al. 2018 Cheerla & Gevaert 2019) Wang et al.) 2020). However, these multimodal fusion based methods fail to explicitly model the interaction between each subset of multiple modalities. Kronecker product considers pairwise interactions of two input feature vectors by producing a high-dimenssoonll feature of quadratic expansion (Zade et al. 2017). and showed its superiority cancer survival prediction (Wang et al., 2021b: Chen et al. 2022b a). Despite of promising results, using Kronecker product in multimodal fusion may introduce large number parameters that may lead to high computational cost and a risk of overfitting (Kim et al., 2017 Liu et al. 2021), thus limiting its applicability and improvement in performance. To overcome this drawback, hierarchical factorized bilinear fusion for cancer survival prediction (HFBSurv) (Li et al. 2022) uses factorized bilinear model to fuse genomic and image features which dramatically reduces computational complexity. PONET differs from HFBSurv in two ways: 1) PONET's multimodal framework has three levels of hierarchical fusion module
2

---

Under review as a conference paper at ICLR 2023
including unimodal, bimodal, and trimodal fusion while HFBBSurv only considers within- modality and cross- modality fusion which we argue it is not adequate for mining the comprehensive interactions; 2) PONET leverages biological pathway informed network for better prediction and meaningful interpretation purposes.
Pathway-associate Sparse Neural Network. Pathway-based analysis is approach that number of studies have been investigated to improve both predictive performance and biological interpretabil- ity (Jin et al., 2014] Cirillo et al., 2017: Hao et al.) 2018 Elmarakeby et al.) 2021). Moreover, pathway-based approaches have shown more reproducible analysis results than gene expression data analysis alone (Li al. 2013 Mallavarapu
al. 2017). These pathway-bat -based deep neural networks can only model genomic data which severely inhibits their applicability in current biomedical re-
search. Additionally, the existing pathway- aathway-associated sparse neural network structures are limited for disease mechanism investigation: there is only one pathway layer in PASNet (Hao et al.] (0188) which contains limited biological prior information to deep dive into the hierarchical pathway and biological process relationships; P-NET (Elmarakeby et al. 202 I calculates the final prediction by taking the average of all the gene and pathways layers' outputs, and this will bias the learning process because it will put more weights for some layers' outputs while underestimates the others.
3.
METHODOLOGY
3.1 PROBLEM FORMULATION AND NOTATIONS
The model architecture of PONET is presented in Fig. I where three modalities are included as input: gene expression ! Rd,, pathological image p E Rd,, and copy number (CNV) mutation (MUT) +MT E Id, with d, being the dimensionality of and so on. We define a hierarchical fac- torized bilinear fusion model for PONET. We build a sparse biological pathway-informed embedding network for gene expression, and a fully connected (FC) embedding layer for both preprocessed patho- logical image feature (fp) and the copy number + mutation (fe) to map feature into similar embedding space for alleviating the statistical property differences between modalities, the three network archi- tecture details are in Appendix C.I We label the three modality embeddings as hm. m E [g, p, c). the superscript/subseript b. and represents unimodal fusion, biomodal fusion and trimodal fusion. After that, the embeddings of each modality is first used as input for unimodal fusion to generate the modality-specific representation h. = w hens w represents the modality-speciin importance, the feature vector of the unimodal fusion is the sum of all modality-speciffi representations h. Σm h% In the bimodal fusion, modality-specific representations from the output of unimodal fusion are fused to yield cross modolityy representations Kmimg = Wm my hm my. M1, m2 E {p, g and m1 # m2. Wiamm my represents the corresponding cross-modality importance. Similarly, the feature vector of bimodal fusion is calculated as hy Em.,, h Amimy. We propose to build a trimodal fusion to take each cross-modality representation from the output of bimodal fusion to thoroughly mine the interactions. Simiarly as the bimodal fusion architecture, trimodal fusion feature vector will be h4 = Σ mas mm. Wmy mmymmmy A m M1, m2, m3 E {p, e, c. and m. # m2 * my, Win magma represents the corresponding trimodal importance. Finally, PONET concatenates hus hs, hy to obtain the final comprehensive multimodal representation and pass it to the Cox porportional harzard model (Cox 1972, Cheerla & Gevaert (019) for survival prediction. In the following sections we will describe our hierarchical factorized bilinear fusion framework, I, o, s represents the dimensionality of hm Zm, hymim
3.2 SPARSE NETWORK
We design the sparse gene-pathway network consisting of one gene layer followed by three pathway layers. patient sample of e gene expressions formed as a column vector, which is denoted by
X = x= Ze). each node represents one gene. The gene layer is restricted to have connections reflecting the gene-pathway relationships curated by the Reactome pathway dataset (Fabregat et al., 2020). The connections are encoded by binary matrix M E Ba where is number of pathways and e is number of genes, an element of M, mij. is set to one if gene j belongs to pathway i. The connections that do not exist in the Reactome pathway dataset will be zero-out. For the next aathway-pathwer layers, a similar scheme is applied to control the connection between consecutive layers to reflect the parent-child hierarchical relationships that exist in the Reactome dataset. The
3

---

Under review as a conference paper at ICLR 2023
Faler Sporet
TUTTE PATROpg
734 p14purg O3
Figure 2: Overall framework of the visual representation extraction using pre-trained self-suuprvvsed vision transformer.
output of each layer is calculated as
(I) where / is the activation function, M represents the binary matrix, W is the weights matrix, X is the input matrix, the bias vector, and is the Hadamard product. We use tanh for the activation of each node. We allow the information flow from the biological prior informed network starting from the first gene layer to the last pathway layer, and we label the last layer output embeddings of the sparse network for gene expression as hig
y f((M W) +
3.3 UNIMODAL FUSTON
Bilinear models (Tenenbaum Freemann 2000) provide richer representations than linear models. Given two feature vectors in different modalities, e.g., the visual features E R= for an image and the genomic features y E Rax1 for a genomic profile, bilinear model uses quadratic expansion of linear transformation considering every pair of features:
x." WWW
where W; E Rmxn is a projection matrix, 2; R the output of the bilinear model. Bilinear models introduce large number of parameters which potentially lead to high computational cost and overfitting risk. To address these issues, Yu et al. (017)) develop the Multi-modal Factorized Bilinear pooling (MFB) method, which enjoys the dual benefits of compact output features and robust expressive capacity.
Inspired by the MFB Yu et al. 2017) and its application in pathology and genomic multimodal learning (Li et al. (022).. we propose unimodal fusion to capture mudality-specifff representations and quantify their importance. The unimodal fusion takes the embedding of each modality hm as input and factorizes the projection matrix W; in Eq. as two low-rank matrices:
1 Wahm Σ NT au ham
zs
(3)
d=1
TT VI hm), m E (,
get the output feature 2m
2m = SumPooling (UThin VI
m Nm k) k). E {p.c,g]
(4)
where l: the latent dimensionality of the factorized matrices. SumPooling (ar, k)) function performs
sum pooling over x by using 1-D non-overlapped window with the size k, Üm E Rixko and
Vn E Ixx are 2-D matrices reshaped from Um and Vm, U.. =(Um,,,.... Um,h) )) R and
Vm = [V..I..., km E Rixxxo, Each modality-specific representation ii E BBlto is obtained as:
hm = hm Ozm, m E cp
(5)
where denotes vector concatenation. We also introduce an modality attention network Atten R - IR_ to determine the weight for each modality-specift representation to quantify its impor-
tance:
Wm Atten Auen), m (p.c,g)
(6)
where wm is the weight of modality m. In practice, Atten consists a sigmoid activated dense layer parameterized by O Atten- Therefore, the output of each modality in unimodal fusion, by denoted
4

---

Under review as a conference paper at ICLR 2023
Wm P. E RR' m (p.c.g). Accordingly, the output of unimodal fusion, hus is the sum of each weighted -specit representation wm/m. m {p, c, which different from ARGF (Mai et al. 2020) that used the weighted average of different modalities as the unimodal fusion output.
3.4 BIMODAL AND TRIMODAL FUSION
The goal of bimodal fusion is to fuse diverse information of different modalities and quantify different importance for them. After receiving the aedality-specifff representations N% from the unimodal fusion, we can generate the cross mross-modaaity representation hmmmm R similar to Eq. (4)
hm1.m = Sum Pooling (U hmm oVT m2 Kmn k))
mi, m2 E (p, m1 + m2
where UH [R(+o)xks and VT R(+oooxa are 2-D matrices reshaped from Um, and Vma
and Umi [Um........ al E [[+taxxkx and V [Vmg.1..... V Vuu.s] E [[(+ooxkxs, We leverage bimodal attention network (Mai et al. 2020) identify the importance of the cross- modality representation. The similarity Smimz E R of hm and hm is first estimated as follows:
140
Chmng Ringo
Sm, m22 =
=
(8)
Ej=1 elizhingg
i=1
jj=
where the computed similarity is in the range of 0 to 1. Then, the cross cross cros-mtodality importance Wmymm obtained by:
Omgny
Wmy + Wm2
Wmyg m2
Wmy m2
Em
Smnyy + So
where So represents a pre-defined term controlling the relative contribution of similarity and modality- specific importance, and here is set to 0.5. Therefore, the output of bimodal fusion, h2, is the sum of each weighted cross-mo representation Wm, m2 hm my m2, m2 {p, g. and my # m2.
In the trimodal fusion, each bimodal fusion output is fused with the unimodal fusion output that does not contribute to the formation of the bimodal fusion. The output for each corresponding trimodal representation is mmma. In addition, trimodal attention was applied to identify the importance of each trimodal representation, Wmmmmmm The output of the trimodal fusion, hes the sum of each weighted trimodal representation Wm, mamn Pem Mi, M2, my E {p, c. .. and mi F m2 F my.
3.5 SURVIVAL Loss FUNCTION
We train the model through the Cox partial likelihood loss (Cheerla Gevaert 2019) with 4 regularization for survival prediction, which is defined as:
((9) - Σ =E
he (ax) log Σ exp (bes (1:)) + A (1
(10)
J.T.TT
where the values E,, T3 and x; for each patient represent the survival status, the survival time and the feature, respectively. E, means event while E, = represents censor: he is the neural network model trained for predicting the risk of survival, θ is the neural network model parameters, and is a regularization hyperparameter avoid overfitting.
EXPERIMENTS
4.1 EXPERIMENTAL. SETUP
Datasets. To validate our proposed method, we used six cancer datasets from The Cancer Genome Atlas (TCGA), a public cancer data consortium that contains matched diagnostic WSIs and genomic
5

---

Under review as a conference paper at ICLR 2023
data with labeled survival times and censorship statuses. The genomic profile features (mutation
status, copy number variation, RNA-Seq expression) are preprocessed by Porpoise I
[Chen et al.
2022b). For this study, used the following cancer types: Bladder Urothelial Carcinoma (BLCA)
(n = 437), Kidney Renal Clear Cell Carcinoma (KIRC) (n = 350), Kidney Renal Papillary Cell
Carcinoma (KIRP) (n 284), Lung Adenocarcinoma (LUAD) (n 515). Lung Squamous Cell
Carcinoma (LUSC) (n 484). Pancreatic adenocarcinoma (PAAD) (n = 180). We downloaded the same diagnostic WSIs from TCGA website that used in Porpoise study to match the paired genomic features and survival times. The feature alignment table for all the cancer type is in Appendix A For each WSI, automated segmentation of tissue was performed. Following segmentation, image patches of size 224 x 224 were extracted without overlap at the 20 X equivalent pyramid level from all tissue regions identified while excluding the white background and selecting only patches with at least 50% tissue regions. Subsequently, visual representation of those patches are extracted with vision transformer (Wang et al. 2021a) pre-trained on the TCGA dataset through a supe-supe constructive learning approach, such that each patch is represented as 1 X 2048 vector. Fig. 2 shows the framework for the visual representation extraction by vision transformer (VIT). Survival outcome information is available at the patient-level, we aggregated the patch level feature into slide level feature representations based on attention-based method (Lu et al. 2021 lsse et al. 2018), please check the algorithm details in Appendix B.4
Baselines. Using the same 5-fold cross-validation splits for evaluating PONET, we implemented and evaluated six state-of-the-an art methods for survival outcome prediction. Additionally, we included three variations of PONET: a) PONET-O represents only genomic data and pathway architecture for the gene expression are included in the model; b) PONET-OH represents only genomic and pathological image data but without pathway architecture in the model; c) PONET is our full model. For all methods, we use the same VIT feature extraction pipeline for WSIs, as well as identical training hyperparameters and loss function for supervision. Training details and the parameters tuning can be found in the Appendix C.2
CoxPH (Cox 1972) represents the standard Cox proportional hazard models (Appendix B.1 DeepSurv (Katzman et al., 2018) is the deep neural network version of CoxPH model. Pathomic Fusion (Chen et al. 2022a) as pioneered deep-learning based framework for predicting survival outcome by fusing pathology and genomic multimodal data, in which Kronecker product is taken to model pairwise feature interactions across modalities.
GPDBN (Wang et al.] (2021b) adopts Kronecker product to model inter- motal-yo and intra-modality relations between pathology and genomic data for cancer prognosis prediction.
HFBSurv (Li et al. 2022) extended GPDBN using factorized bilinear model to fuse genomic and pathology features a within-modalin and cross tross-modaittiss hierarchical fusion.
Porpoise (Chen et al. 2022b) applied discrete survival model and Kronecker product to fuse pathology and genomic data for survival prediction (Zadeh Schmid 2020).
Evaluation. For each cancer dataset, we used the cross- valdsaadl concordance index (C-Index) (Appendix (B.22) (Harrell et al. 1982) measure the predictive performance of correctly ranking the predicted patient risk scores with respect to overall survival.
4.2 RESULTS
Comparison with Baselines, In combing pathology image, genomics, and pathway network via PONET, our approach outperforms CoxPH models, unimodal networks, and previous deep learning based approaches on pathology-genen wer----enommmasseeeeeee survival outcome prediction (Table D From the results, deep learning-bast based approaches generally exhibit better performance than CoxPH model. PONET achieves superior C-index value in all six cancer types. All versions of PONET outperform Pathomic Fusion by a big margin. Pathomic Fusion uses Kronecker product to fuse the two modalities and that's also the reason why other advanced fusion methods, like GPDBN and HFBSurv, achieves better performance. Also, we argue that Pathomic Fusion extracts the region of interest of pathology image for feature extraction might limit the understanding of the tumor microenvironmenn of the whole slide. HFBSury shows better performance than GPDBN and Pathomic Fusion which is consistent with their findings, and these results further demonstrate that the hierarchical factorized

---

Under review as a conference paper at ICLR 2023
Table 1: C-Index (mean + standard deviation) of PONET and ablation experiments in TCGA survival prediction. Top performers highlighted in bold.
Medel
TOSA-MI
TC34- KTRC
TOSA-
TOGA-121AT
TCBA-1U82
TCSA-PAM
CosPH (Age Gender)
1972
0.525 0.130
0.550 0.070
0.544 + 0,050
0531 0.082
0.532 : 0.094
0.539 0,092
DorpSurv
Kampman
2018
0,580 =: 0.062
0.600 0.043
0,560:2 0.063
0.534 100077
0.541 :: 0.066
0.544 1: 0,076
(PDRN Wing al
2021
0612 = 0.042
0.67 0.073
0.669 + 0.109
0.565
0.057
0.545 :: 0.063
0:571 + 0.060
HFBSurv Li
21R3
0.622 = 0:045
0.667
0.053
0,769 + 0.109
0.581 + 0.017
0.548 # 0.049
0.591 + 0,052
Pathomic Fusion Chea
2012a
0,586 + 0:062
0.398
0,060
0.577 + 0.032
0.543
0,065
0.523
+0.045
0.545 + 0.064
Porpoise Chen 20226
0.617 =: 0.048
0.711 0.051
0.811 d: 0.089
0.586 0:056
0.527
:: 0.043
0.591 :: 0.064
PONET-O (ours)
0,596 :: 0.056
0.664
1: 0.056
0,761 :: 0.093
0.623 0.066
0.538 :: 0.037
0.5%8 1: 0.027
PONET-OR (ours)
0.65 = 0.063
0.695 0.043
0.776 4: 0.123
0618 0.049
0.551
= 0.049
0.591 0,050
PONET (ours)
1,037
0.726 0.056
=: 0.646 0.044
0.639 0.088
Table Evaluation of PONET on different fusion methods and pathway designs by C-index (mean + standard deviation). Best performer is highlighted in bold.
Methods
POGA BLCA
TCSA KIGP
7OGA LUAD
TCGk L39C
TOGA PARD
Single fusion
Simple concateration
0.585 + 0.045
0.682 +: 0,029
0.554 + 0.005
0,525 = 0.066
0.508 + 0.075
Element-uiee additio
0.592 0.047
0.655
4: 0.055
0.587 0.065
0,522 0.046
0.588 4 0.055
Tensor fusion Zadeh et all 2017
0.055 0.046
0.775 + 0.053
0.505
0.000
0.545 0.045
0.592 +: 0.061
Hierarchical fusion
Unimodal Bimodal
0.596
0.611
0.066
0.553
0.802
0.062
0.289 1: 0.053
0,501
0.066
0.552
0.051
0.598 1 0.083
ARGF Mai et 2020
0,597 1: 0.054
0,792
:: 0.043
0.614 0.051
0.556
0.063
0,602 1 0.065
Unimbour 5mico3
0.614 0,052
0,803
0,061
0,631
0.044
0.578
0.058
0,615 + 0u87
Pathway design
PASNet HHap et 2018
0,606
0.045
0,793 1: 0.051
0.621
0.061
0,551
0.069
0.625 1: 0.057
Elmarakeby PONET
0.622 0.047
0,802
0.625
0.562
0.627 1:
0.643 0.037
0.829 :: 0.054
0.641 0.046
0.567
0.066
0.639 + 0.070
bilinear model can better mine the rich complementary information among different modalities compare to Kronecker product. Porpoise performs similarly with PONET on TCGA-KIRC and TCGA-KIRP and outperformed HFBSury in these two studies, this probably is due Porpoise partitioned the survival time into different non-overlapping bins and parameterized it as a discrete survival model (Zadch & Schmid (020) which works better for these two cancer types. In other cases, Porpoise performs similarly with HFBSurv. Note: the results of Porpoise are from their paper (Chen et al. 2022b).
Additionally, we can see that PONET consistently outperforms PONET-O and PONET-OH indicating that the effectiveness of the biological pathway-infonme informedd neural network and the contribution of pathological image for the overall survival prediction.
A Ablation Studies. To assess the impact of hierarchical factorized bilinear fusion strategy is indeed effective, we compare PONET with four single-fusion methods: 1) Simple concatenation: concatenate each modality embeddings; 2) Element- wise addition: element-wise addition from each modality embeddings: 3) Tensor fusion (Zade et al. 2017): Kronecker product from each modality embed- dings. Table 2 shows the C-index values of different methods. We can see that PONET achieves the best performance and shows remarkable improvement over single-fusion methods on different cancer type datasets. For example, PONET outperforms the Simple concatenation by 8.4% (TCGA-BLCCA) 27% (TCGA-KIRP), 1.5% (TCGA-LUAD), 8.0% (TCGA-LUSC), and 11.4% (TCGA-PAAD)), etc.
Furthermore, we adopted five different configurations of PONET to evaluate each hierarchical component of the proposed method: 1) Unimodal: unimodal fusion output as the final feature representation; 2) Bimodal: bimodal fusion output as the final feature representation; 3) Unimodal + Bimodal: hierarchical (include both unimodal and bimodal feature representation) fusion; 4) ARGF: ARGF (Mai et al. (020)) fusion strategy; 5) PONET: our proposed hierarchical strategy by incorporating unimodal, bimodal and trimodal fusion output. As shown in Table 2 Unimodal + Bimodal performs better than Unimodal and Bimodal which demonstrates that Unimodal + Bimodal can capture the relations within each modality and across modalities. ARGF performs worse than Unimodal+ + Bimodal and far worse than PONET across all the cancer types. PONET outperforms Unimodal + Bimodal in 4 out of 5 cancer types indicating that three layers of hierarchical fusion can mine the comprehensive interactions among different modalities.
To evaluate our sparse gene- eete-aa network design, we compare PONET with PASNet (Hao et al.. 2018) and P-NET (Elmarakeby et al.) (2021) pathway architecture, PASNet performs the worst due to the fact that it only has one pathway layer in the network, and thus limited prior information was used to predict the outcome. PONET constantly outperforms P-NET across all the cancer types, which
7

---

Under review as a conference paper at ICLR 2023
H1
H2
HB
H4
Con miciiiaooo of mosplr aomilro
noroira vesplr Appiling DUADD
POUDS
MAGI
Carmikinee retoree foerd
Rignol em alliatiiioo
CTNNOI
ROUOU
enoenesiie par cereettcon
Low tivves pyreeii
MiPaaea
Ragulatio piDA zanecipton
cu reteaat of suvaried thtve
Signalig by TGF lera Racapler Complax Cancer
-0MG3
Comple ioleeeneee
237 (D40 mediaed incurtin of the
Taispot of -tanti and soddoe
oggatic
TCGA-02 -LGA--2-A55Z Survival Month 14.05
Ropdatoo artiity
Cáuodiaa idael evels
The idlexzaxx
Figure 3: Inspecting and interpreting PONET on TCGA-KIRP. a: Sankey diagram visualization of inner layers of PONET shows the estimated relative importance of different nodes in each layer. Nodes in the first layer represent genes; the next layers represent pathways; and the final layer represents the model outcome. Different layers are linked by weights. Nodes with darker colours are more important, while transparent nodes represent the residual importance of undisplayed nodes in each layer, presents gene layer and H2-H4 represent pathway layers; b: Co-attention visualization of top 4 ranked pathways in case of
demonstrates that averaging all the intermediate layers' output for the final prediction cannot fully capture the prior information flow among the biological hierarchical structures.
Model Interpretation. We discuss the model interpretation results for cancer type TCGA-KIRP here and the results for other cancer types included in the Appendix C.3] To understand the interactions between different genes, pathways and biological processes that contributed to the predictive performance and to study the paths of impact from the input to the outcome, we visualized the whole structure of PONET with the fully interpretable layers after training (Fig. a). To evaluate the relative importance of specific genes contributing the model prediction, we inspected the genes layer and used the Integrated Gradients attribution (Sundararajan et al., (017) method to obtain the total importance score of genes, and the modified ranking algorithm details are included in the Appendix [3.] Highly ranked genes included KRAS, PSMB6, RACI, and CTNNBI which are known kidney cancer drivers previously (Yang et al., 2017) 7| (Shan al.] [017: Al-Obaidy et al.) 2020) Guo et al.) 2022). GBN2, member of the guanine nucleotide-bindngg proteins family, which has been reported that the decrease of its expression reduced tumor cell proliferation (Zhang et al. 2019).
A recent study identified strong dependency on BCL2L1, which encodes the BCL-XL anti apoptotic protein, in a subset of kidney cancer cells (Grubb et al., 2022). This biological interpretability revealed established and novel molecular features contributing kidney cancer. In addition, PONET selected a hierarchy of pathways relevant to the model prediction, including down regulation of TGF-3 receptor signaling, regulation of PTEN stability and activity, the NLRP1 inflammasome, and noncanonical activation of NOTCH3 by PSENI, PSMB6, and BCL2L1. TGF-3 signaling is increasingly recognized as key driver in cancer, and in progressive cancer tissues TGF-3 promotes tumor formation, and its increased expression often correlates with cancer malignancy (Han et al., 2018). Noncanonical activation of NOTCH3 was reported to limit tumour angiogenesis and plays vital roles in kidney disease (L et al. 2017).
To further inspect the pathway spatial association with WSI slide we adopted the co-attention survival method MCAT (Chen et al. (202)) between WSIs and genomic features on the top pathways of the sec- ond layer, visualized as a WSI-level attention heatmap for each pathway genomic embedding in Fig. 3 3 (algorithm details are included in the Appendix B.6). We used the gene list from top 4 pathways as the genomic features and trained MCAT on TCGA-KIRP dataset for survival prediction. Overall, observe that high attention in different pathways showed different spatial pattern associations with the slide. This heatmap can reflect senotype-phenotyy relationships in cancer pathology. The high attention regions (red) of different pathways in the heatmap have positive associations with the pre-
8

---

Under review as a conference paper at ICLR 2023
PONET-O
PONET-OH
PONET-ARGE
PONET
Proportion surviving
0.3
0.1
0.2
0.5
0.3
Louree
Lanh 201
A4
Time (months)
Figure 4: Kaplan-Meke Meier analysis of patient stratification of low and high risk patients via four variatons of PONET on TCGA-KIRP. Low and high risks are defined by the median 50% percentile of hazard predictions via each model prediction. Logrank test was used to test for statistical significance in survival distributions between low and high risk patients.
dicted death risk while the low attention regions (blue) have negative associations with the predicted risk. By further check the cell types in high attention patches we can gain insights of prognostic mor- phological determinants and have better understanding about the complex tumor microenvironment.
Patient Stratification. In visualizing the Kaplan-Mett Meier survival curves of pre- dicted high risk and low risk patient populations, we plot four variations of PONET in Fig. 4 PONET-ARGF rep-
Table 3: Comparison of model complexity
Methods
Number of Parameters
FLOPS
Pathomic Fusion
175M
168G
resents the model that we use the hier- archical fusion strategy of ARGF in our pathway-informed PONET model. From the results, PONET enables easy sepa- ration of patients into low and high risk groups with remarkably better stratifica- tion Value=6..000- in comparison to the others.
GPDBN
82M
91G
HFBSurv
0.3M
0.5G
PONET
2.8M
3.1G
Complexity Comparison. We compared PONET with Pathomic Fusion, GPDBN, and HFBSury since both Pathomic Fusion and GPDBN are based on Kronecker product to fuse different modalities while GPDBN and HFBSurv modeled moter-modality and intra-modality relations which have similar consideration to our method. As illustrated in Table 3, PONET has 2.8M (M Million) trainable parameters, which approximately 1.6%, 3.4%, and 900% of the number of parameters Pathomic Fusion, GPDBN, and HFBSurv. To assess the time complexity of PONET and the competitive methods, we calculate floating point operations per second (FLOPS) each method in testing. The results in Table show that PONET needs 3.1G during testing, compared with 168G, 91G, and 0.5G in Pathomic Fusion, GPDBN, and HFBSurv. The main reason for fewer trainable parameters and number of FLOPS lies in that PONET and HFBSurv performs multimodal fusion using factorized bilinear model, and can significantly reduce the computational complexity and meanwhile obtain more favorable performance. PONET has one additional trimodal fusion which explains why has more trainable parameters than HFBSurv.
5 CONCLUSION
In this study, we pioneer propose a novel biological pathway-informee hierarchical multimodal fusion model that integrates pathology image and genomic profile data for cancer prognosis. In comparison to previous works, PONET deeply mines the interaction from multimodal data by conducting unimodal, biomodal and trimodal fusion step by step. Empirically, PONET demonstrates the effectiveness of the model architecture and the pathway informed network for superior predictive performance. Specifically. PONET provides insight on how to train biological informed deep networks multimodal biomedical data for biological discovery in clinic genomic contexts which will be useful for other problems in medicine that seek to combine heterogeneous data streams for understanding diseases and predicting response and resistance treatment.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Khaleel I Al-Obaidy, John N Eble, Mehdi Nassiri, Liang Cheng, Mohammad K Eldomery, Sean R Williamson, Wael A Sakr, Nilesh Gupta, Oudai Hassan, Muhammad Idrees, et al. Recurrent kras mutations in papillary renal neoplasm with reverse polarity. Modern Pathology, 33(6):1157-116 2020.
Laura Antolini, Patrizia Boracchi, and Elia Biganzoli. time-dependent discrimination index for survival data. Statistics in Medicine, 24(24):3927-3944 2005.
Norman Edward Breslow. Discussion of the paper by d.r. cox. Journal of the Royal Statistical Society: Series (Statistical Methodology), 34(2):216-2) 1972.
John KC Chan. The wonderful colors of the hematoxylin-eosin stain in diagnostic surgical pathology. International Journal of Surgical Pathology, 22(1):12-322 2014.
Anika Cheerla and Olivier Gevaert. Deep learning with multimodal representation for pancancer prognosis prediction. Bioinformatics, 35:1446-i454, 2019.
Richard J Chen, Ming Y Lu, Wei-Hung Weng, Tiffany Y Chen, Drew FK Williamson, Trevor Manz, Maha Shady, and Faisal Mahmood. Multimodal co-attention transformer for survival prediction in gigapixel whole slide images. /EEE/CVF International Conference on Computer Vision, pp. 3995 4005, 2021.
Richard J Chen, Ming Lu, Jingwen Wang, Drew FK Williamson, Scott Rodig, Neal Lindeman, and Faisal Mahmood. Pathomic fusion: An integrated framework for fusing histopathology and genomic features for cancer diagnosis and prognosis. IEEE Transactions on Medical Imaging, 41(4):757-770, 2022a.
Richard Chen, Ming Y Lu, Drew FK Williamson, Tiffany Y Chen, Jana Lipkova, Zahra Noor, Muhammad Shaban, Maha Shady, Mane Williams, Bumjin Joo, et al. Pan-cancer integrative histology-genomic analysis via interpretable multimodal deep learning. 108iv:2/08.02278, 2022b.
Jun Cheng, Jie Zhang, Yatong Han, Xusheng Wang. Xiufen Ye, Yuebo Meng, Anil Parwani, Zhi Han, Qianjin Feng, and Kun Huang. Integrative analysis of histopathological images and genomic data predicts clear cell renal cell careinoma prognosis. Cancer Research, 77(21)xe11ee000, 2017.
Elisa Cirillo, Laurence Parnell, and Chris T Evelo. review of pathway-bast based analysis tools that visualize genetic variants. Frontiers in Genetics, 8(174), 2017.
Pierre Courtiol, Charles Maussion, Matahi Moarii, Elodie Pronier, Samuel Pilcer, Meriem Sefta, Pierre Manceron, Sylvain Toldo, Mikhail Zaslavskiy, Nolwenn Le Stang, et al. Deep learning-based classification of mesothelioma improves prediction of patient outcome. Nature Medicine, 25(10): 1519-1525, 2019.
David R Cox. Regression models and life-tables. Journal of the Royal Statistical Society: Series (Statistical Methodology), 34(2):187-22 1972.
Haitham A Elmarakeby, Justin Hwang, Rand Arafeh, Jett Crowdis, Sydney Gang, David Liu, Saud H AlDubayan, Keyan Salari, Steven Kregel, Camden Richter, et al. Biologically informed deep neural network for prostate cancer discovery. Nature, 598:348-352, 2021.
Antonio Fabregat, Steven Jupe, Lisa Matthews, Konstantinos Sidiropoulos, Mare Gillespie, Phani Garapati, Robin Haw, Bijay Jassal, Florian Korninger, Bruce May, et al. The reactome pathway knowledgebase. Nucleic Acids Research, 48(D1):D498-D) D5033 2020.
Treg Grubb, Smruthi Maganti, John Michael Krill-Burger, Cameron Fraser, Laura Stransky, Tomas Radivoyevitch, Kristopher A. Sarosiek, Francisca Vazquez, William G. Kaelin Jr., and Abhishek A. Chakraborty. A mesenchymal tumor cell state confers increased dependency on the bel-xl anti- apoptotic protein in kidney cancer. bioRxiv, 2022.
Jing- Yi Guo, Zuo-qian Jing, Xue-jic Li, and Li-yuan Liu. Bioinformatic analysis identifying psmb 1/2/3/4/6/8/9/10 as prognostic indicators in clear cell renal cell carcinoma. International Journal of Medical Sciences, 19(5):796-5 2022.
10

---

Under review as a conference paper at ICLR 2023
Zhezhu Han, Dongxu Kang, Yeonsoo Joo, Jihyun Lee, Geun Hoen- Oh, Soojin Choi, Suwan Ko, Suyeon Je. Hye Jin Choi, and Jae Song. Tgf-beta downregulation-iiuuee innuuced cancer cell death is finely regulated by the sapk signaling cascade. Experimental and Molecular Medicine, 50(12):162, 2018.
Jie Hao, Youngsoon Kim, Tae- Kyung Kim, and Mingon Kang. Pasnet: pathway sparse deep neural network for prognosis prediction from high-throughput data. BMC Bioinformatics, 19: 510, 2018.
Frank E Harrell, Robert M Califf, David B Pryor, Kerry Lee, and Robert A Rosati. Evaluating the yield of medical tests. Journal of the American Medical Association, 247(18):2543-2546 1982.
Achim Hekler, Jochen S Utikal, Alexander H Enk, Wiebke Solass, Max Schmitt, Joachim Klode, Dirk Schadendorf, Wiebke Sondermann, Cindy Franklin, Felix Bestvater, et al. Deep learning outperformed 11 pathologists in the classification of histopathological melanoma images. Europe Journal of Cancer, 118:91-6, 2019.
Le Hou, Dimitris Samaras, Tahsin M Kure, Yi Gao, James Davis, and Joel H Saltz. Patch-based convolutional neural network for whole slide tissue image classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2424-2433, 2016.
Le Hou, Rajarsi Gupta, John S Van Arnam, Yuwei Zhang, Kaustubh Sivalenka, Dimitris Samaras, Tahsin M Kurc, and Joel Saltz. Dataset of segmented nuclei in hematoxylin and cosin stained histopathology images of ten cancer types. Scientific Data, 7(1):185, 2020.
Osamu lizuka, Fahdi Kanavati, Kei Kato, Michael Rambeau, Koji Arihiro, and Masayuki Tsuneki. Deep learning models for histopathological classification of gastric and colonic epithelial tumours. Scientific Report, 10(1):1504 2020.
Maximilian Ilse, Jakub Tomczak, and Max Welling, Attention- based deep multiple instance learning. In Proceedings of the 35th International Conference on Machine Learning, volume 80, pp. 2127- 2136, 2018
Ly Jin, Xiao- Yu Zuo, Wei- Yang Su, Xiao-Lei Zhao, Man-Qiong Yuan, Li-Zhen Han, Xiang Zhao, Ye-Da Chen, and Shao-Qi Qi Rao. Pathway- -bsed analysis tools for complex diseases: review. Genomics Proteomics Bioinformatics, 12(5):210-220 2014.
Onno Kampman, Flham J Barezi, Dario Bertero, and Pascale Fung. Investigating audio, video, and text fusion methods for end-to-end automatic personality prediction. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Pupers), pp. 606-6114 2018.
Jared L. Katzman, Uri Shaham, Alexander Cloninger, Jonathan Bates, Tingting Jiang, and Yuval Kluger. Deepsurv: personalized treatment recommender system using cox proportional hazards deep neural network. BMC Medical Research Methology, 18(24):187-20) 2018.
Jin-Hwa Kim, Kyoung- Woon On, Woosang Lim, Jeonghee Kim, Jung- Woo Ha, and Byoung- Tak Kyoung-Woon
Zhang. Hadamard product for low-rank bilinear pooling. In Proceedings of International Confer- ence on Learning Representations, pp. 1-14, 2017.
Ruiqing Li, Xingqi Wu, Ao Li, and Minghui Wang. Hfbsurv: hierarchical multimodal fusion with factorized bilinear models for cancer survival prediction. Bioinformatics, 3889):257722594 2022.
Yanming Li. Bin Nan, and Ji Zhu. Multivariate sparse group lasso for the multivariate multiple linear regression with an arbitrary group structure. Biometrics, 71(2):354-63, 2015.
Shuheng Lin, Ana Negulescu, Sirisha Bulusu, Benjamin Gibert, Jean-Guy Delcros, Benjamin Ducarouge, Nicolas Rama. Nicolas Gadot, Isabelle Treilleux, Pierre Saintigny, et al. Non-canonical notch3 signalling limits tumour angiogenesis. Nature Communications, 8: 8:16074, 2017.
Zhun Liu, Ying Shen, Varun Bharadhwaj Lakshminarasimhan, Paul Pu Liang, Amir Zadeh, and
Louis Philipe Morency. Efficient low-rank multimodal fusion with pondallty-speciff factors. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, pp. 2247-2256, 2021.
11

---

Under review as a conference paper at ICLR 2023
Ming Y Lu, Drew FK Williamson, Tiffany Y Chen, Richard J Chen, Matteo Barbieri, and Faisal Mahmood. Data-efficient and weakly supervised computational pathology on whole-slide images. Nature Biomedical Engineering, 5:555-570, 2021.
Sijie Mai, Haifeng Hu, and Songlong Xing. Modality to modality translation: An adversarial representation learning and graph fusion network for multimodal fusion. In Proceedings of the AAAI Conference on Artificial Intelligence, pp. 164-172, 2020.
Tejaswini Mallavarapu, Youngsoon Kim, Jung Hun Oh, and Mingon Kang. R-pathcluster: Identifying cancer subtype of glioblastoma multiforme using pathway-bar -wway restricted boltzmann machine. In 2017 IEEE International Conference on Bioinformatics and Biomedicine, pp. 1183-8, 2017.
Pooya Mobadersany, Safoora Yousefi, Mohamed Amgad, David A Gutman, Jill S Barnholtz-Sloan, José E Velázquez. Vega, Daniel J Brat, and Lee AD Cooper. Predicting cancer outcomes from histology and genomics using convolutional networks. Proceedings of the National Academy of Sciences, 115(13):2270--2299) 2018.
Zhenyuan Ning, Weihho Pan, Yuting Chen, Qing Xiao, Xinsen Zhang, Jiaxiu Luo, Jian Wang, and Yu Zhang. Integrative analysis of cross-modal features for the prognosis prediction of clear cell renal cell carcinoma. Bioinformatics, ((9))2888-2899) 2020.
Behnaz Nojavanasghari, Deepak Gopinath, Jayanth Koushik, Tadas Baltrušaitis, and Louis-Philippe Morency. Deep multimodal fusion for persuasiveness prediction. In Proceedings of the 18th ACM International Conference on Multimodal Interaction, pp. 284-288, 2016.
Xipeng Pan, Lingqiao Li, Huihua Yang, Zhenbing Liu, Jinxin Yang, Lingling Zhao, and Yongxian Fan. Accurate segmentation of nuclei in pathological images via sparse reconstruction and deep convolutional networks. Neurocomputinng 15:88-99 99, 2017.
Soujanya Poria, Iti Chaturvedi, Erik Cambria, and Amir Hussain. Convolutional mkl based multi- modal emotion recognition and sentiment analysis. In Proceedings of International Conference on Data Mining, pp. 439 448, 2016.
Guang Shan, Tian Tang, Huijun Qian, and Yue Xia. Expression of tiaml and racl proteins in renal cell carcinoma and its clinical- pathologiaalll features. International Journal of Clinical and Experimental Pathology,
2017.
Vaishnavi Subramanian, Tanveer Saeda-Mahmood, and Minh N Do. Multimodal fusion using sparse cea for breast cancer survival prediction. In Proceedings of IEEE 18th International Symposium on Biomedical Imaging (ISBI), pp. 1429-1432, 2021.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In Proceedings of the 34th International Conference on Machine Learning, volume 70, pp. 3319-3328, 2017.
Joshua B Tenenbaum and William T Freeman. Separating style and content with bilinear models. Neural Computation, 124--1228 2000.
Chunyu Wang. Junling Guo, Ning Zhao, Yang Liu, Xiaoyan Liu, Guojun Liu, and Maozu Guo. can- cer survival prediction method based on graph convolutional network. IEEE Trans Nanobioscience, 19(1):117-126, 2020.
Xiyue Wang, Sen Yang, Jun Zhang, Minghui Wang, Jing Zhang, Junzhou Huang, Wei Yang, and Xiao Han. Transpath: Transformer-based self-supervised learning for histopathological image classification. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 186- 195. Springer, 2021a.
Zhiqin Wang, Ruiqing Li. Minghui Wang, and Ao Li. Gpdbn: deep bilinear network integrating both genomic data and pathological images for breast cancer prognosis prediction. Bioinformatics, 27(18))2963-277 2970, 2021b.
Martin Wöllmer, Felix Weninger, Tobias Knaup, Bjöm Schuller, Congkai Sun, Kenji Sagae, and Louis-Philippe Morency. Youtube movie reviews: Sentiment analysis in an audio-visual context. IEEE Intelligent Systems, 28(3):4-53,, 2013.
12

---

Under review as a conference paper at ICLR 2023
Wing Hung Wong. Theory of partial likelihood. The Annals of Statistics, 14(1):88-123 1986.
Ellery Wulczyn, David F Steiner, Zhaoyang Xu, Apaar Sadhwani, Hongwu Wang, Isabelle Flament- Auvigne, Craig H Mermel, Po-Hsuan Cameron Chen, Yun Liu, and Martin C Stumpe. Deep learning-based survival prediction for multiple cancer types using histopathology images. Plos One, 15(6):e023367, 2020.
Chun-ming Yang, Shan Ji, Yan Li, Li-ye Fu, Tao Jiang, and Fan-dong Meng. B-catenin promotes cell proliferation, migration, and invasion but induces apoptosis in renal cell carcinoma. OncoTargets and Therapy, 10:711-7244 2017.
Zhou Yu, Jun Yu, Jianping Fan, and Dacheng Tao. Multi-modal factorized bilinear pooling with attention learning for visual question answering. In JEEE International Conference on Computer Vision, pp. 1839-1844 2017.
Amir Zadeh, Rowan Zellers, Eli Pincus, and Louis- Phillipp Morency. Multimodal sentiment intensity analysis in vidcos: Facial gestures and verbal messages. IEEE Intelligent Systems, pp. 82-88, 2016.
Amir Zadeh, Minghai Chen, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. Tensor fusion network for multimodal sentiment analysis. In Proceedings the 2017 Conference Empirical Methods in Natural Language Processing, pp. 1103-1114, 2017.
Shekoufeh Gorgi Zadeh and Matthias Schmid. Bias in cross-entropy- training of deep survival networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43()):3126-3137, 2020.
Qiang Zhang, Xiujuan Yin, Zhiwei Pan, Yingying Cao, Shaojie Han, Guojun Gao. Zhiqin Gao, Zhifang Pan, and Weiguo Feng. Identification potential diagnostic and prognostic biomarkers for prostate cancer. Oncology Letters, 18(4):4237-4244 2019.
13

---

Under review as a conference paper at ICLR 2023
Table 4: TCGA Data Feature Alignment Summary
WSI
CNV
MUT
RNA
WSI+CNV+MUT
WSI+MUT+RNA
ALL
Cancer Type
BLCA
454
443
452
450
441
448
437
KIRC
517
509
357
514
352
355
350
KIRP
294
291
286
293
284
285
284
LUAD
528
522
523
522
519
519
515
LUSC
505
502
489
503
486
487
484
PAAD
208
201
187
195
187
180
180
A DATA
Table 3 in Appendix A shows the number of patients with matched different data modalities: WSI (Whole slide image), CNV (Copy number), MUT (Muation), RNA (RNA-Seq gene expression). For each TCGA dataset and each patient we have preprocessed data dimensions dg IR 1x2000 (RNA), de E IR x227 (CNV MUT),, and dp E R x (WSI) which will be used for our multimodal fusion.
B METHODS
B. COX PROPORTIONAL HAZARD MODEL
In survival analysis, we are interested in modeling the continuous time T until some event of interest
(i.e. survival time). The survival function S(t) = P(T ()) = -1 - (s)ds is the probability of individual surviving longer than time lo. where / probability density function survival times. We denote the probability that event occurs in infinitesimal interval after time t, given it has not yct occurred at time as the hazard function X(t):
PP S T T
X() = lim 8-40
(11)
8
This results in the relationship: S(t) ezp- where A(t) J Ass)ds is the cumulative hazard function. The most common semi remi-paramettre approach for estimating the hazard function the Cox proportion hazards model (Cox 1972). which assumes that the hazard function can be parameterized as an linear exponential function A(E X) (t exp (3 xX) where the baseline hazard function (t) describes how the risk of an event changes over time, 3 are model parameters that describe how the hazard varies with features X. The baseline hazard At(t) is unspecified in the original model, making it difficult to estimate 3, however, the Cox partial log- lileeiildoood technique (Wong 1986) can first estimate 3 by maximizing:
3 X; - log Ex,(,,)) o xx)
(8) = A
(12)
j=1
where n is the set of patients, for the i-th subject in the study, T, and C₁ denote, respectively, the event time and the potential censoring time: and X; I2 denotes the observed features. Thus. the observed data from a typical survival study contain independent observations D = {X;, O,, A; J where the observed time O = min (T,, C+) and the event indicator A, 1, if the observed time T,, i.e. T; & C₁, otherwise A, = 0, the subject is censored at C, Once the parameter / has been estimated through the log partial likelihood, the cumulative baseline hazard function A(t) J X(s) can be estimated through the Breslow estimator (Breslow) 1972).
B.2 C-INDEX
Due to the presence of censoring in survival data, traditional performance measures such as mean squared error cannot be used to evaluate the accuracy of predictions. Instead, the concordance-index
14

---

Under review as a conference paper at ICLR 2023
(C-index) (Harrell al., 1982) is one of the most widely used performance measures for survival models. It assesses how good a model is by measuring the concordance between the rankings of the predicted event times and the true event times. Specifically, if the predicted event time of the i-th individual is T. the C-index is defined by C' = P [ T, O < O,, A; = =1). However, it difficult to obtain the predicted event time in most survival models, so the following C-index proposed in Antolini et al. (2005) is often used in practice:
C= PP ($ (0, X,) < $ (O, x2) 50,
(13)
If (X; O;, A.) and $ (t X,) denote observations and predicted conditional probabilities, re- spectively, the C-index in Eq. 13 can be estimated empirically by
Ò Ei Anl (3 (0, x.) <0(0)
E E A (0, <0,)
The range of the C-index is [0, 1], and larger values indicate better performance with a random guess leading to C-index of 0.5.
B.3 WSI REPRESENTATION LEARNING
It has been shown that the WSI visual representations extracted by self-supervised learning methods on histopathological images are more accurate and transferable than the supervised baseline models on domain-irrelevnt datasets such as ImageNet. In this work, a pre-trained Vision Transformer (ViT) model (Wang et al. 20)la) that is trained on a large histopathological image dataset has been utilized for tile feature extraction. The model is composed of two main neural networks that learn from each other, i.c. student and teacher networks. Parameters of the teacher model 0g is updated using the student network with parameter 0, using update rule represented in Eq. (14).
0g < T0e + (1-r)0,
(14)
Two different views of given input H&E image x, uniformly selected from training set I, are gener- ated using random augmentations, i.e. u, Then, student and teacher models generate two different visual representations according to and
1/1 = f", (u) and Ø2 = f (v), respectively. Finally, the generated visual representations are transformed into latent space using linear projection as
P1 = = (g (v1)) and & = nd (jz) for student and teacher networks, respectively. Similarly, feed- ing v and
student and teacher networks leads to y/2 = 8 (v), Vi f (u) P2 = =g* (g (v2)) and E = d (yi)) ). Finally, the symmetric objective function Lioes is optimized through minimizing the f2 - norm distance between student and teacher as Eq. (15)
1 I. (pI. 22) + 2
Lloss
[p1.
(p2, 21)
(15)
2
where T(p,
Tellaa
Talla and || ||2 represents lg norm.
B.4 AGGREGATE PATCH LEVEL FEATURE INTO SLIDE LEVEL FEATURE
Survival outcome information is available at the patient- level instead of for individual slides, we use the attention based strategy in Porpoise (Chen et al. 2022b) which was originally designed in CLAM (Lu al. 202T) to aggregate the patch level feature into slide level representation for our model training. We treat all WSIs corresponding to patient case as single WSI bag during training and evaluation. a patient has WSIs with bag sizes MI.... MM respectively, the WSI bag corresponding the patient is formed by concatenating all N bags, and has dimensions M 2048, where M M M,
There are three components for the model: 1) the projection layer fp: 2) the attention module fattn; 3) the prediction layer fpeed After the VIT feature extraction, each patch-level embeddings of WSI bag, H M are first mapped into more compact, dataset-specific 512 512-dimensional feature
15

---

Under review as a conference paper at ICLR 2023
space by the fulne-connected layer f with weights proj 5122x2048 and bias Duas 12 Subsequently, the attention module fam learns to score each patch for its perceived relevance to patient-level prognostic prediction. Patches receives high attention scores will contribute more the patient-level feature representation than patches assigned low attention scores, all the patches in one patient's WSIs are aggregated based on sitention-pooling (Isse et al. (018). Specifically, fattn has fully-connected layers with weights U. E 2236x512, Va E [R2o6x612 and W E I 12X255 Given patch embedding hm 12522 (the mh row entry of H ), its attention score am can be computed by:
exp (a (tanh (VahT) O sigm (U,h
1==1 exp (Wa (tanh (V (. @ sigm (U.hT.)
Then the patient-level representations ppttiet E R are computed based on the attention attention-pooling operation from the patch-level feature representations by attention scores as weight coefficients, where A E AR is the vector of attention scores:
M
ppatient = Attn pool H
Σ 7/=
ag hm
The last fully-connected layer is used to learn a representation hwst RIx32 which is then used as input to our multimodal fusion.
B.5 SPARSE NETWORK FEATURE INTERPRETATION
We use the Integrated Gradients attribution algorithm to rank the features in all layers. Inspired by PNET (Elmarakeby et al. (021), to reduce the bias introduced by over-annotation of certain nodes (nodes that are member of too many pathways), we adjusted the Integrated Gradients scores using a graph informed function f that considers the connectivity of each node. The importance score of each node i, C divided by the node degree dł if the node degree is larger than the mean of node degrees plus 50 where is the standard deviation of node degrees.
d fan in + fan out!
d 1 50
adjusted C. f(x) =
otherwise
B.6 CO-ATTENTION BASED PATHWAY VISUALIZATION
After we get the ranking of top genes and pathways, we adopted the co-attention survival model (MCAT) (Chen et al. 202T) to show the spatial visualization of genomic features. We trained MACT on all our TCGA datasets, MACT learns how WSI patches attend to genes when predicting patient survival. We define each WSI patch representation and pathway genomic features Hbag and Gbag- The genomic features are the gene list values from the top pathways of each TCGA dataset. The model uses Goag E xd,, guide the feature aggregation of Hbng E Nxd, into clustered set of gene-guided visual concepts Arug E Ndd, dy and d, represents the dimension for the pathway (number of genes involved in the pathway) and patch. Through the following mapping:
QK
CoAung-nn H) softmax
Vap
W,GHTW! ddp
= softmax
W
Acuatn W.H - Å
where W W W. Rd trainable weight matrices multiplied the queries Gbay and key-value pair ( Houg Houg ), and Acaam NxM is the co-attention matrix for computing the weighted average of Hray- Here, M represents the number of patches in one slide and N represents number of pathways (We trained top five pathways, so N = 4 in our study).
16

---

Under review as a conference paper at ICLR 2023
Interpretation: For single genomic pathway embedding 8n. G, the co-attention module scores the pairwise similarity for how much hm attends to gn for all hm E Hbag. written as a row vector [a,1, an2. ... d.ml Acoat.. These attention weights are then applied element-wise to H which constructs a new WSI-level feature embedding h, E R that reflects the biological function of gn. For example, if In is genomic embedding that expresses the underlying biological pathways responsible for tumor formation, Acontt computed by the co-attention layer would saliently localize image patches containing tumor cells as high attention, which then aggregates h, as WSI-level representation primarily containing tumor cells. We describe the set of high attention image patches that attend to a single genomic embedding gn as a "gene-guided visual concept", in which patches that are similar in feature space to g, would share similar phenotypic information. For N pathway embeddings in Gbog, the co-attention module captures up to N different pathway- eathway-guided visual concepts, which we visualizes as attention heatmap in Fig. b.
C EXPERIMENTS
C. NETWORK ARCHITECTURE
Sparse network for gene:
The final gene expression embedding is hg E 1x000
Pathology network: The slide level image feature representation is passed through an image
embedding layer and encodes the embedding as h IR 1x500
CNV + MUT network: Similarly as the pathology network, the patient level CNV + MUT feature representation is passed through an FC embedding layer and encodes the embedding as h, E 1x000
C.2 EXPERIMENTAL DETAILS
PONET. The latent dimensionality of the factorized matrices k is a very important tuning parameter. We tune k [3, 5, 10, 20, 30, 50] based the testing C-index value (Appendix Fig. 55 and the loss of training and testing plot (Appendix Fig. 6) for each dataset. We choose & to maximize the C-index value and also it should have stable convergence in both training and testing loss. For example, we choose k = 10 in TCGA-KIL KIRP for the optimized results. We can see that in Appendix Fig. 5 the testing loss is quite volatile when & is less than 10. Similarly, we choose k [20, 10, 20, 20, 10 for OCGA-BLCA, TCGA KIRC, TCGA-LUAD, TCGA-LUSC, TCGA- respectively.
The learning rate and the regularization hyperparameter for the Cox partial likelihood loss are also tunable parameters. The model is trained with Adam optimizer. For each training/testing pair, we first empirically preset the learning rate to 1.2e-4 as a starting point for a grid search during training, the optimal learning rate is determined through the 5-fold cross-validation on the training set, C-index was used for the performance metric, After that, the model is trained on all the training set and evaluated on the testing set. We use 2e-3 through the experiments for A. The batch size is set to 16, and epoch is 100. During the training process, we carefully observe the training and testing loss for convergence (Figure 4 in Appendix C.2) The server used for experiments is NVIDIA GeForce RTX 2080Ti GPU.
CoxPH. We only include the age and gender for the survival prediction. Using CoxPHFitter from lifelines
DeepSurv We concatenate preprocessed pathological images features, gene expression, and copy number + mutant data in vector to train the DeepSurv model. L.2 reg = 10.0, dropout = 0.4, hidden layers sizes = [25, 25], learning rate le-05, learning rate decay = 0.001, momentum = 0.9.
preprocessed image feature, gene expression and copy number + mutation as model input. k 20, Learning rate is 2e-3, weight decay is 4e-4. Batch size is 16 and epoch is 100. Drop out rate is 0.25.
Pathomic Fusion We use the pathomicSurv model which takes
GPDBN 0 Learning rate is 2c-3, batch size is 16, weight decay is le-6, dropout rate is 0.3, epoch is
"tttp:://github.o. "tttp://gituub.com///faaaa/a/// http:///www....//////////a//oooo
17

---

Under review as a conference paper at ICLR 2023
0.6
C-Index
0.4
0.2
0.0
10
20
50
K
Figure 5: C-Index value under K = 3, 5, 10, 20, 30, 50 for TCGA-KIRP. The mean value and standard deviation for 5-fold cross-validation are plotted.
Fold
Tox
K=5
Fitleri
K-10
K-50
Figure 6: Train and test loss for TCGA-KIRP under K = =, 5, 10, 20, 50 for 5-fold cross- cross-validation.
100.
HFBSury
The learning rate set to 1e-3, batch size 16, À 3e-3, weight decay is 1e-6, epoch is
100.
"hups://githu..com// http:/hhww....mmmimiiiiiuuuuusau
18

---

Under review as a conference paper at ICLR 2023
C.3
ADDITIONAL RESULTS
Gene
Pathways
GNEE
PIEP, PPSA ans IERO Plaga sta BOWKT 5 grall
Tol Liva Fladistor (TURTo) Cuacads
Ca Cesstracchhee rublie inaraciona
PEPOREE
SHC neelededd evonta irggereed by 1GF
Gei dealt sggeill ca NEVCCE, wior NALE
RRVA "ayymrrsee (neeserprao Elergation
NAP2K a M&CK col
viaefon gameno signa knj
IRNA processing it the bieeondiin
Colmrcd in induced events
Mhstie Tyleshore
Regulation of Hnstodd Fartor IHFF oogen
crole gatad Potasilum
Mutarts an aranprrnn Cance
Spi cing
KWH43
auppadi
Godgit natrogaacle tnansport
aggnaligg
jundion degrodation
Bignal ing sease
Mitale Spindlo
HSPMONBT
hydroivdss NUTT protein
Bisyythesls
signaling
TBK1
higraiig
hydrnppii
Figure 7: Inspecting and interpreting PONET on TCGA-BLCA. Sankey diagram visualization of inner layers of PONET shows the estimated relative importance of different nodes in each layer. Nodes in the first layer represent genes; the next layers represent pathways; and the final layer represents the model outcome. Different layers are linked by weights. Nodes with darker colours are more important, while transparent nodes represent the residual importance of undisplayed nodes in each layer.
TCGA-4Z-AATY Survival
PEKKAKT IBRS
SHCCoslaeed
MAPIX and MAPK activation
Calmodulle induced events
TCGA-UY-A78N Survival Month: 86.76
Figure 8: Co-attention visualization of top 4 ranked pathways in two cases of TCGA-BLCA.
19

---

Under review as a conference paper at ICLR 2023
Gene
Pathways
Glacagendipe
Ciaas (Secrain bamily neceplon))
GPCR
Doweregllane lase EREBC ERBES sig slog
C118 Tharstton
01/9 DNA Damage Chockpoers
checkpoitt
Actratto pre-miccaivs
(AA FAR
Uetacaa colectar ratebalaars
PSNC22
1iArndppnndent ENA aage Rospoose
CLECTA (Decth-11 Agraling
MAPKIMMPPKS signaling
MFN
outoore
PLGB1
mosppnss
PaWDr1
CLECPA DSedilo 1) indaoes NFAT ucivation
Delects cobalamin (B12) melabollam
HV Infection
PONF1
RHO GTPassee Activote NADPH coetoaee
HGR Single Stard Anneating
Stonaling ER682
B-C
IDF1R
Actooato af NCDICA and inanciaaaion to relochondts
Adhovtonn of mitl-orly prowin
Mirik: 52-42M plasee
resdual
resiuca
midual
noiccad
Figure 9: Inspecting and interpreting PONET TCGA-KIRC. Sankey diagram visualization of inner layers of PONET shows the estimated relative importance of different nodes in each layer. Nodes in the first layer represent genes; the layers represent pathways; and the final layer represents the model outcome. Different layers are linked by weights. Nodes with darker colours are more important, while transparent nodes represent the residual importance of undisplayed nodes in each layer.
TCGA-A3-3313 Survival Mouth: 24.15
Dnemagriation of RANDERRRN dgwving
MA PIK and MAPK actovatin
Activation on' gyo-pppicativecoom complex
PSI-Independer: DNA (orrage regores
High Am
TOGA-A3-3320 Survival Monch 49.54
LOW A1
Figure 10: Co-attention visualization of top 4 ranked pathways in two cases of TCGA-KIRC.
20

---

Under review as a conference paper at ICLR 2023
Gene
Pathways
PREMT
Prapier dosl vororall suorinere
Keggee peccssing pooppeoeenie
Harpo Paetve sropeecre
COFT
AMl Cantrmiiie weclaion Dych
DCTM
Coll Deccuerr
Hegg a Aaros Pracwerr ser phic sho caein
HB0 digg Seeun Arrweng
Officess
regitao sead gecling
www.aaagggg..... peocurtt
BAIRs/WAn agratin
467 rcadull
rcaidue
redial
Figure 11: Inspecting and interpreting PONET on TCGA-LUAD. Sankey diagram visualization of inner layers of PONET shows the estimated relative importance of different nodes in each layer. Nodes in the first layer represent genes; the next layers represent pathways; and the final layer represents the model outcome. Different layers are linked by weights. Nodes with darker colours are more important, while transparent nodes represent the residual importance of undisplayed nodes in each layer.
TCGA-55.866 Survival Month: 16.92
Procovivv systhoaa lagiing stand
HDR deough Iattolagon recenlination
Phoephade hydndysis by NUDI pretens
CIAINCAICCCS) mediate
inadiiontionn of Cycin B.Clkl complex
TCGA-78-718 Survival Month: 119.42
Figure 12: Co-attention visualization of top ranked pathways in two cases of TCGA-LUAD.
21

---

Under review as a conference paper at ICLR 2023
Gene
Pathways
sallng
D1P0 depeedeen PoaSod
Peetur pholeptade
Me clenlinge
PopoRde
PSR PPSA and IER Repulas norT Siguling
Sepaiin NITRA (Tree))
IRA prosenaig inlocftedtion
NcGWM
aggaiing
BACT
KoTD41
WAFK and MAkK relurion
Tell lin Rhoopior to Redi Canzzon
Ross Piiiskon Repolr AP San Furnatoo
A3AU17
Nagaloo repulatios lultiin pattway
bisin (Be0 metiteltam
Padgeing of
AXB moser citrts aesinaus osstrad
HAV inporadent (19--111 ediaatio
ND 1044
KOOM
SHG ricted ewers rrggored I6F18
Goig to ER manogrode tonopprtt
Nudiotton Necyhhock
residual
residual
Figure 13: Inspecting and interpreting PONET on TCGA-LUSC. Sankey diagram visualization of inner layers of PONET shows the estimated relative importance of different nodes in each layer. Nodes in the first layer represent genes; the next layers represent pathways; and the final layer represents the model outcome. Different layers are linked by weights. Nodes with darker colours are more important, while transparent nodes represent the residual importance of undisplayed nodes in each layer.
TOGA-18-3414 Survival
CDS8 depertknn P13 signalng
PESP IPIA IERS ragulaic PIK/AKT sinniig
Calmockin indacal
NrCAM innmatiie
High
TOGGA-33-4538 Survival
Law
Figure 14: Co-attention visualization of top 4 ranked pathways in two cases of TCGA-LUSC.
22

---

Under review as a conference paper at ICLR 2023
Gene
Pathways
piace Cance
rigiting
orici prase
peaco aed Doiaa Matatt nar
apuarayyrann
Clasi Mils serier
of femvile Auseiu Mel...ioeee (SCH))
Tryn
fume not ylut ape lo cieryge
vero Uncpet wit
Pabin Unisting uffler deriot
Hing
VONG
MCT achvddes SMP" and tac!
Appouden oFAT sas tee on eecie splonnenppatvve
CPCR Bomemmee
PaG ceai
Figure 15: Inspecting and interpreting PONET on TCGA-PAAD. Sankey diagram visualization of inner layers of PONET shows the estimated relative importance of different nodes in each layer. Nodes in the first layer represent genes; the next layers represent pathways; and the final layer represents the model outcome. Different layers are linked by weights. Nodes with darker colours are more important, while transparent nodes represent the residual importance of undisplayed nodes in each layer.
TCGA-2I-ABBO Survival Month: 14.45
Fomnation acacastic aucciaee hstrrddenmaii
Resuiation
Cueszytte bissyelless
High
TCGA-3A-A9U Survival Month: 33 54
LOw
Figure 16: Co-attention visualization of top ranked pathways in cases of TCGA-PAAI
23


Reviews:
[{"review_id": "_1NQ4tUwBR", "reviewer": "Reviewer_x8M7", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Deep Biological Pathway Informed Pathology-Genomic Multimodal Survival Prediction\n\nThe manuscript proposes a multi-model data integration method for survival prediction. The method, called PONET, integrates gene expressions, pathology image features and copy number and mutation data using the multi-model low-rank bilinear pooling (MLB, implemented by SumPooling) which is a variant of the bilinear models.  Specifically, PONET extends the previous work HFBSurv [Li et al., 2022] to include the trimodal fusion (combining two bimodal fusion outputs) and the pathway structures for hidden layers. The experimental results show that the proposed methods work better for survival prediction in several cancer types in TCGA cohort. In addition, the methods provide biological interpretations (e.g., gene-pathway associations or image-pathway associations using co-attention visualization) \n", "summary_of_the_review": "I think that the manuscript still needs improvement to be accepted by the conference (mainly due to its limited novelty and presentation). However, I like the fact that the methods provide biological interpretations, specifically the information presented in Figure 3.  ", "strength_and_weaknesses": "Strengths\n1. PONET shows the supervisor prediction performance over the base-line methods, including the Kronecker product-based fusion methods and the MLB-based method. In addition, the proposed methods, which are based on the MLB, have lower complexities compared to the Kronecker product-based fusion methods\n2. The methods provide biological interpretations which would be useful to identify new biomarkers markers from data. \n\nWeakness\n1. I did not see significant improvement in methodology when comparing the proposed methods to HFBsurv [Li et al., 2022]. It seems that all the main components (the way to fuse multi modalities, attention mechanism, and even all the main equations) are directly borrowed from the previous work.\n2. It is a little bit hard to understand the meanings of several points. I think that the paper needs to be revised. \nFor example, \n2-a) about the equation in the top of page 3, regarding the attention, requires the form of the attention. I don’t think this is the standard attention module which is based on queries, keys and values.\n2-b) In the section 5.2 (comparison with Baselines), “All versions of PONET outperform Pathomic Fusion by ~“. The authors listed two main reasons, but it seems that they need more explanations. For the reason 1, it sounds reasonable that the proposed methods, based on the MLB (low rank), are less computational completed than Pathomic. However, it is not clear why the MLB-based-methods are necessarily outperforms the Kronecker-product-based methods. For the reason 2, are the authors trying to compare both feature extraction methods, VIT and GCN? \n3. In the section 5.3, could the authors give more explanations about why discretization of Survival time can lead better performance for certain cancer types.\n4. Regarding Figure 3-b. It is not clear how co-attention visualization was done. I thought that it could be done by postprocessing the outputs of the trained PONET. However, the appendix just includes a reference method. Could the author elaborate this process in the manuscript? \n5. Figure 1 and the experiments. What kind of the pathway database is used to model the PONET pathway structure? I can see how the layer H1 (genes) and H2 (pathways) are structured (the pathway database indicates the member genes of each pathway), but don’t understand how to build the layers H3 and H4. These kinds of information should be included in the text. \n6. I think the manuscript needs a careful proofreading. Some minor types grammatic errors. For example, in Page 3, “Unlike our proposed PONET can” might be “which“ was missing here?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "NNqdFjVFJe", "reviewer": "Reviewer_LXPk", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents PONET to fuse multi-modal data for making survival predictions. Specifically, a spare biological pathway-informed embedding network (Han et al., 2018; Elmarakeby et al., 2021) is used to deal with genomic data to reveal meaningful biological interpretations. To capture modality-specific information, the authors used bilinear pooling (Yu et al., 2017; Li et al., 2022), which provides richer representations than linear models. For multi-modal fusion, the idea from ARGF (Mai et al., 2020) is borrowed to assign attentions to identify the importance of cross-modal representations. On six cancer datasets from TCGA, PONET achieves performance improves over previous multi-modal survival prediction methods.", "summary_of_the_review": "Based on the limited novelty of ideas and techniques, I'm afraid that the manuscript is not suitable for publish in its current version.", "strength_and_weaknesses": "$Strengths$:\n\nOverall, the descriptions of proposed PONET are clear and easy to understand. Lots of implementation details are included, making the proposed method easy to understand and improving the reproducibility. Pathway-based analysis is an interesting and promising direction for improving both predictive performance and biological interpretability. Besides, I appreciate the efforts made in visualizing the co-attention results.\n\n$Weaknesses$:\n\n1. The novelty is limited from perspectives of ideas and techniques. Multi-modal survival analysis is not a new idea (Mobadersany et al., 2018; Cheerla & Gevaert, 2019; Wang et al., 2020). Biological pathway-informed network has been used in similar problems (Han et al., 2018; Elmarakeby et al., 2021). The idea of attention-based multi-modal fusion methodology is borrowed from ARGF (Mai et al., 2020). The limited technical novelty is also reflected in Table 1. Comparing PONET with PONET-OH, we can draw a conclusion that the performance advantages of PONET mainly orginate from the incorporation of the pathway architecture. However, the pathway architecture has been used in similar problems (Han et al., 2018; Elmarakeby et al., 2021), and the authors did not mention the technical differences between their implementation and previous versions (Han et al., 2018; Elmarakeby et al., 2021), both of which weaken the technical contributions of PONET. Besides, I think the implementation details  of the sparse pathway network should be clarified in the main text instead of the appendix considering it is one of the core contributions. \n\n2. Phrases like \"Unimodal Fusion\" and \"Trimodal Fusion\" are misleading and may exaggerate the contributions of this paper. \"Unimodal Fusion\" actually is equivalent to a feature embedding process, where the factorized multi-modal bilinear pooling (Yu et al., 2017; Li et al., 2022) is used to produce rich representations. Since it contains no interactions among different modalities, the use of \"Fusion\" is incorrect and misleading. A more appropriate name should be like \"Unimodal Embedding\". A similar problem also exists in \"Trimodal Fusion\", where the authors only apply the bimodal fusion strategy again. Personally, I recommend merging sec. 4.3 Bimodal Fusion and sec. 4.4 Trimodal Fusion into a single section as their operations are almost the same.\n\n3. Experimental results are not sufficient. Since this paper heavily borrows the idea (i.e., hierarchical and attentional fusion) from ARGF (Mai et al., 2020) for multi-modal fusion, it is necessary to compare PONET with ARGF in the experiment section. However, I do not see any results of ARGF in the submission.  Besides, as shown in Table 2, bimodal and trimodal fusion are very useful. However, PONET only contains on bimodal fusion layer and one trimodal fusion layer. So, how about stacking more fusion layers? Also, the implementation details of single-fusion methods are not clarified, making the ablation results less convincing.\n\nMinor points.\n\n1. Notations of dimensions. For clarification, $x\\in \\mathbb{R}^m$ should be $x\\in \\mathbb{R}^{m\\times 1}$. $y\\in \\mathbb{R}^n$ should be $y\\in \\mathbb{R}^{n\\times 1}$.\n\n2. In sec. 4.1, $m_1$, $m_2$ $\\in (p, c, g)$ should be $m_1$, $m_2$ $\\in$ {$p, c, g$}.\n\n3. In sec. 4.3, \"If two information vectors are close to each other, then little complementary information lies between them and their information has been well explored in the unimodal fusion.\" The use of \"close\" is inaccurate, how do you define the closeness between vectors? In addition, the unimodal fusion never explores the complementary information between two modalities' vectors regardless of their  similarity. Thus, the above statement should be rephrases.\n\n4. All formulas need to be numbered.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "Ja7hp5yVjij", "reviewer": "Reviewer_JgRd", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose a novel biological pathway-informed hierarchical multimodal fusion model that intnegrates pathology image and genomic profile data for cancer prognosis.\n", "summary_of_the_review": "The authors propose a novel biological pathway-informed hierarchical multimodal fusion model that intnegrates pathology image and genomic profile data for cancer prognosis. For several datasets from TCGA, the results indicated the advantage of the proposed model. Since the authors’ strategy is quite domain specific, it may be difficult to generalize the strategy.\n", "strength_and_weaknesses": "Strength\n\n- Strategy and model building sound.\n- Experiments were conducted extensively.\n- The proposed model showed better results compared to previously reported models.\n\nWeaknesses\n\n- If pathway informed is the key to the advantage of proposed model, how exactly the authors designed and inpremented such part of the model is unclear. How novel from Han et al. 2018 or Elmarakeby et al. 2021?\n- Since the pathway informed strategy is tightly domain specific, it is quite difficult to generalize the authors finding to the other field, thus more specific venue might be suitable for meaningful discussion.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject