Prototypical Information Bottlenecking and Disentangling for Multimodal Cancer Survival Prediction
Authors: Yilan Zhang
Yingxue Xu
Jianqi Chen
Fengying Xie
Hao Chen
Venue: ICLR 2024 spotlight (2024)
Keywords: multimodal survival prediction
computational pathology
Primary area: representation learning for computer vision, audio, language, and other modalities

Abstract: Multimodal learning significantly benefits cancer survival prediction, especially the integration of pathological images and genomic data. Despite advantages of multimodal learning for cancer survival prediction, massive redundancy in multimodal data prevents it from extracting discriminative and compact information: (1) An extensive amount of intra-modal task-unrelated information blurs discriminability, especially for gigapixel whole slide images (WSIs) with many patches in pathology and thousands of pathways in genomic data, leading to an "intra-modal redundancy" issue. (2) Duplicated information among modalities dominates the representation of multimodal data, which makes modality-specific information prone to being ignored, resulting in an "inter-modal redundancy" issue. To address these, we propose a new framework, Prototypical Information Bottlenecking and Disentangling (PIBD), consisting of Prototypical Information Bottleneck (PIB) module for intra-modal redundancy and Prototypical Information Disentanglement (PID) module for inter-modal redundancy. Specifically, a variant of information bottleneck, PIB, is proposed to model prototypes approximating a bunch of instances for different risk levels, which can be used for selection of discriminative instances within modality. PID module decouples entangled multimodal data into compact distinct components: modality-common and modality-specific knowledge, under the guidance of the joint prototypical distribution. Extensive experiments on five cancer benchmark datasets demonstrated our superiority over other methods. The code is released.

Published as a conference paper at ICLR 2024
PROTOTYPICAL INFORMATION BOTTLENECKING AND DISENTANGLING FOR MULTIMODAL CANCER SUR- VIVAL PREDICTION
Yilan Zhang 121 Yingxue Xu 12, Jianqi Chen 2, Fengying Xie *2, Hao Chen x1
'The Hong Kong University of Science and Technology. 'Beihang University yxuebo connect. ust. jhclcse. ust. wust.u... hk Shangyilan, Jigchenjiange
xfy_73)@buae edu.cn
ABSTRACT
Multimodal learning significantly benefits cancer survival prediction, especially the integration of pathological images and genomic data. Despite advantages of multimodal learning for cancer survival prediction, massive redundancy in mul- timodal data prevents it from extracting discriminative and compact information: (1) An extensive amount of intra-modal task-unrelated information blurs discrim- inability, especially for gigapixel whole slide images (WSIs) with many patches in pathology and thousands of pathways in genomic data, leading to an "intra- modal redundancy" issue. (2) Duplicated information among modalities domi- nates the representation of multimodal data, which makes modality-spede in- formation prone being ignored, resulting in an "inter-modal redundancy" is- sue. To address these, we propose a new framework, Prototypical Information Bottlenecking and Disentangling (PIBD), consisting of Prototypical Information Bottleneck (PIB) module for intra-modal redundancy and Prototypical Informa- tion Disentanglement (PID) module for modal redundancy. Specifically, a variant of information bottleneck, PIB, is proposed to model prototypes approx- imating bunch of instances for different risk levels, which can be used for se- lection of discriminative instances within modality. PID module decouples entan- gled multimodal data into compact distinct components: modality comomn and ceddality-specifie knowledge, under the guidance of the joint prototypical distri- bution. Extensive experiments on five cancer benchmark datasets demonstrated superiority over other methods. The code is released
INTRODUCTION
Jenkins 2005 Salerno & Li 2023) aims to estimate the death risk of patients for prognosis, in which multimodal learning by integrating both histological information and genomic molecular profiles can benefit the prognosis of majority of cancer types (Chen al. 2020 2022b 2021; Jaume et al. 2023 Xu & Chen 2023). These modalities offer diverse perspectives for patient stratification and informing therapeutic diciision-akinng (Zuo al. 2022). For example, histological images give visual phenotypic information about tumor mi- croenvironment, e.g., the organization of cells (Jackson et al., 2202)). for different grading of cancer, while genomics data provides global landscapes (Györffy, 2021) for various molecular subtyping of cancer. They collaboratively contribute to different survival outcomes. Nevertheless, a large quantity of redundancy in mulitmodal data poses some significant challenges to effective fusion.
Cancer survival analysis Cox
1975
The primary question at hand is: How modality by eliminating its redundancy, referred as "intra-modal redundancy" issue? The label for WSI consisting of numerous patches is typically provided at the WSI level, leading weak pervision for survival prediction. In the absence precise annotations, such as patch-wise labeling for cancerous regions in WSIs, both task-related and irrelevant information become intermingled in the model's input, resulting in information redundancy (Hosseini et al. (023).. Specifically, the region of interest, e.g., the tumor cells highly related to risk assessment, only occupies small
capture the discriminative information from single
Corresponniig authors, Equal contribution.
httts://gi thub. com/ 2y1 buaauuaaaaa. 110.. /it.git

---

Published as a conference paper at ICLR 2024
portion of gigapixel WSIs with high resolutions of about 100, 000 x 100, 000 pixels (Zuu et al, 2017). For this fin-grrinedt visual recogmition, although certain multiple-inste insttacc learning (MIL) IIse et al. 2018 Li et al. 2021 Yao et al. 2020) have provided some promising solutions, they do not enforce constraints to remove redundant information, thus struggling to obtain discriminative representations. A similar redundancy issue emerges in genomic modality. Research (Jaume iet al., 20233 Chen et al.) (022)) indicates that biological pathway-based gene groups, characterized by known interactions in unique cellular functions, offer more semantic correspondence with pathology features. However, these pathways can yield hundreds to thousands of groups, and only a few specific pathways exhibit a strong correlation with patient prognosis (e.g. immune-related pathways are significant for bladder cancer prognosis prediction (Jiang et al. 2021a
Another concern is: How can we capture compact yet comprehensive knowledge from the dominant overlapping information in multimodal data, referred as "inter-modal redundancy" issue? The redundancy stemming from this duplicated information can complicate the knowledge extraction. Therefore, extracting independent factors by disentangling while discarding superfluous information. The knowledge (Liang et al. 2023) can be split into distinct components: modality-speciff knowledge and modality commoon knowledge. The former contains information unique to single modality, while the latter encapsulates common information and exhibits consistency across modalities. To obtain effective knowledge from multimodal redundancy, existing efforts (Chen et al., 2021; Xu & Chen 2023) focus on integrating common information, emphasizing the inherent consistency through alignment. However, common informa- tion often dominates aligning and integrating multimodal information, leading to the suppression of speecic information, thereby disregarding the wealth of distinctive perspectives.
enhance the feature effectiveness
In this work, we propose a new multimodal survival prediction framework, Prototypical Information Bottlenecking and Disentangling (PIBD), consisting of Prototypical Information Bottleneck (PIB) module for "intra-modal redundancy" and Prototypical Information Disentanglement (PID) module for "inter-modal redundancy". First, Information Bottleneck (IB) provides a promising solution to compress unnecessary redundancy from itself while maximizing discriminative information about task targets. However, IB may suffer from the high-dimensional computational challenges posed by massive patches of a gigapixel WSI and hundreds of pathways. Instead, we propose new IB variant, PIB, that models prototypes approximating a bunch of instances (e.g.. patches of pathology or pathways of genomics) for different risk levels, which can guide selection of discriminative instances within a modality. Secondly, PID removes inter-modal redundancy by comprehensively decomposing entangled multimodal features into ideally independent modality-common and modality-specific knowledge. To do this, we reuse the joint prototypical distributions modeled by aforementioned PIB guide the extraction of common knowledge. Simultaneously, we enforce the model to learn knowledge different from the joint prototypical distribution, which is considered as guidance for capturing modality-speciif knowledge as well.
It is worth noting that the proposed method can be extended into more multimodal problems with modalities of bag structure. The contributions are as follows: (1) Inspired by information theory for mitigating redundancy, we propose a new multimodal cancer survival framework, PIBD, addressing both "odtra- and "inter-modal" redundancy challenges. (2) We design a new IB variant, PIB, that models prototypes for selecting discriminative information to reduce intra-modal redundancy, while PID addresses inter-modal redundancy by decoupling multimodal data into distinct compo- nents with the guidance of joint prototypical distribution. (3) Extensive experiments on five cancer benchmark datasets demonstrate the superiority of our approach over state-of-ther methods.
2
RELATED WORKS
2.1 SURVIVAL PREDICTION FROM SINGLE MODALITY
Predicting survival risk is vital for understanding cancer progression. Recent advances in digital pathology (Evans et al. 2018) and high-throughput sequencing (Christinat & Krek 2015) technolo- gies have led to vibrant research in single-modal survival prediction using WSIs and genomics data, respectively. To handle gigapixel images, multiple-instance learning (MIL) defines a "bag" as a col- lection of multiple instances (i.e., image patches) and provides effective ways to learn global repre- sentations for WSIs. MIL methods focus on aggregations of instance-level predictions (Campanella let al., 2019: Feng & Zhou| 2017: Hou al. 2016) features (llse et al.) 2018). For the former, bag predictions can be simply fused by pooling the probability values of instances. While the latter employs various strategies for getting the global features, e.g., clustering embeddings (Yao al.
2

---

Published as a conference paper at ICLR 2024
2020), modeling patch correlations with graphs (Guan et al. 2022), , assigning attention weights (Isee et al. 2018 Li al. 2021). J. and learning long-range interactions by transformers (Shao et al. 2021) J.
Furthermore, genomics data provides crucial molecular information essential for survival prediction as well. Typically represented as x measurements, genomic features can be extracted using simple neural networks, e.g., MLP (Haykin) 1998) and SNN (Klambauer al.) 2017). Although these single-modality-base based methods achieved remarkable improvements in feature extraction, they do not provide constraints on removing redundant information to capture the discriminative features.
2.2 SURVIVAL PREDICTION FROM MULTIPLE MODALITIES
In clinical practice. patients are usually collected with comprehensive multimodal data such as genomics (Klambauer et al. (017). pathology (Zhu et al.] 2017] Liu et al., [2022] CChen et al.] 2022a), radiology (Jiang et al., 2021b; Yao et al. (021]. etc. for diagnosis and prognosis, thus learning multimodal interactions (Zhang et al. (023) becomes an important motivation for many studies. These methods are broadly categorized into tensor-based and attention-basee fusion tech- niques (Zhang et al., (020). Some bessor-based fusions, like concatenation (Mobadersany et al.) 2018 and weighted sum (Huang et al. 2020), are simple with few parameters. Alternttivlyy other tensor-based fusion uses bilinear pooling to create a joint representation space by computing the outer product of features, e.g., Kroncecker product (Wang et al. 2021), factorized bilinear pooling (Li et al. 2022). However, these methods are typically used in early or late fusion stages, making the inter-modal interactions (Chen et al. 2022b) prone to be neglected. Recently, attention-based fusion methods focus on learning gross-modal correlations through co-attention mechanisms (Chen al. |021] Zhou & Chen, 2023). For instance, MCAT (Chen et all|[202]) proposed a gene-guided co-attention, HMCAT (Li et al. 2023b) designed radiology-guided co-attention, MOTCat (Xu & Chen 2023) introduced the optimal transport (OT) to model the global structure consistency, and SurvPath (Jaume et al., (203)) utilized the cross-attention to model dense interactions between pathways and histologic patches. Although some approaches can partially achieve alleviating redundancy by alignment, they are prone to lose modality-specific information.
2.3 MULTIMODAL LEARNING WITH INFORMATION THEORY.
Recently, information theory has attracted increasing attention within the multimodal learning com- munity due to its ability to provide measures for quantifying information (Dai et al. 2023 Liang
et al. 2023 Hjelm et al., 2018). Specifically, approaches based on the information bottleneck (IB) principle (Tihby et al. 2000; Alemi et al.] 2016) have emerged as effective strategies for com- pressing raw information while retaining task-relevant knowledge, which has found utility across multi-view (Federici et al., [2000] Lee & Van der Schaar (021) and multi-modal learning (Mai al. 2022). Additionally, another kind of method centered on information disentanglement has been harnessed to extract targeted knowledge (Sanchez et al. 2020; Cheng et (0022) Chen et al.) 2023). facilitating the learning of more compact representations, We introduce this direction into multimodal cancer survival analysis for the first time, and inspired by information theory for mit- igating redundancy, we propose a new framework PIBD that provides an information perspective solution to address the massive redundancy issues in multimodal data.
3
METHOD
3.1
OVERALL FRAMEWORK AND PROBLEM FORMULATION
Given the i-th patient multimodal data including pathology data x), and genomic data xg we aim to predict patients' survival outcome by estimating hazard function p(() ftagora(t) that represents the risk probability of death at the time point t. Figure I displays the overall framework of our PIBD.
We start with extracting unimodal representations for pathology and genomics data. Following the common setting for pathological WSIs and genomic pathways in previous works (Chen et al. 2021
Jaume et al. 2023), we formulate x,,
(8) and (+) as the "bag" of instances based on multiple instance M, J==1 and (+) = (1 E IR )) hij E RdMM,
learning (MIL) for the i-th patient, denoted as (5) = {
1==1>
respectively, where M₁ is the patch numbers of a WSI and Mg is the number of biological pathways.
To address "intra-modal redundancy." we propose Prototypical Information Bottleneck (PIB), detailed in Section [.22] to select discriminative instances for each modality. Subsequently, to reduce "inter-modaa redundancy". we propose Prototypical Information Disentanglement (PID) explained in Section 3.3 PID decomposes multimodal data into independent modality-common representation C and modality-specifie representations denoted Sn. and Sg for histological
3

---

Published as a conference paper at ICLR 2024
Unimodal Bag-level level Representation
Prototypical Information Bottleneck
Prototypical Information Disettangemeett
Segmonting Patching
Prsdfian
|
Insiance Sampling
Modallo "Spall"
Sumilarit Meusure
MALLRaO LL Hele Cawaaaaaa
F
Encoder
WSI
Madddyy
W ST Bays
Protatipe
)
Risk
Pathwags
Retalned
Madalby
Similarity
Monsure
Genes
Pathways
Encoder
Figure 1: Framework of PIBD. Patient data from pathology and genomics are initially structured into bags. The Prototypical Information Bottleneck (PIB) selects discriminative features to reduce "intra-modal redundancy". Subsequently, the Prototypical Information Disentanglement (PID) mod- ule decouples the specific and common information to tackle "inter-modal redundancy".
and genomic modalities, respectively. Finally, the decoupled compact representations will be concatenated get the final multimode features H, which are used for survival risk prediction.
Survival prediction estimates the risk probability of an outcome event before a specific time. How- ever, the outcome is not always observed, resulting in right-censored data. We denote e {0, 1} for censorship status (c = O means observed deaths, c c= 1 means unknown outcomes), and discrete survival time E 21,2, ..., ... ] corresponding a specific risk band. For a final multimodal feature ( obtained from the
pairs (x), x(!) z(0, c()) of the i-th patient, we use NLL loss (Zadeh & Schmidl (020) as survival loss function for survival prediction, following previous works (Chen al.] 2021;(Xu [Xu & Chen 2023):
ND
(aum (00 +0,,00, (2 Σ log(()) ((1)
(1) surv (t (()) + (1 (ogg (11H((())1)())
- 1|H())
+ (1 - (1-c
LogT (1)
where ND is the number of samples in the training sets, Shazard (())) = P(T = 1/ 2 t, HO)
is the hazard function representing the death probability, and p( (t (() (k) is the survival function viewed as survival probability up to time point 4. To
IT (1
simplify, assume y represents patient labels (4, c), resulting in 2N,, labels.
3.2 PROTOTYPICAL INFORMATION BOTTLENECK
To tackle the "intra-modality redundancy". we introduce the information bottleneck and propose a new variant called Prototypical Information Bottleneck (PIB).
Preliminary of Information Bottleneck. The IB introduces a new representation variable Z that is maximally expressive about the target Y. while compressing the original information from the input X. Thus, the objective function to be maximized is given in (Tishby et al. (2000) as:
RιB = I[Z, Y) ()
(2)
where I(:-.) represents the mutual information (MT) that measures the dependence between two variables. The hyperparameter 3 > 0 acts as a Lagrange multiplier, controlling the trade-off where higher 3 values lead more compressed representations. However, the computation of MI is in- tractable, VIB (Alemi et al. (016) transformed (q. into maximizing its approximation of variational lower bound. By inverting the objective function of the variational lower bound, it tries to minimize the loss function (Derivation can be found in Appendix B.2.
N
1 N
JIB
E ) (v., [z)] + (p
where N denotes the sample size, (0())2) is variational approximation of the intractable likelihood p(y]z), p(2[z) is the posterior distribution over 2, and r(z) approximates the prior probability p(2). In practice, r(z) is commonly assumed as a spherical Gaussian (Alemi et al. 2016). And the
4

---

Published as a conference paper at ICLR 2024
posterior distribution p(z[x) can be variationally approximated as:
p(2|z) x q0(2|z) (2; f(()),
(4)
where SE is an MLP encoder that predicts both the mean and covariatnce matrix E.
Prototypical Information Bottleneck. IB seems to provide a hopeful solution to reduce intra- modal redundancy. However, in our task, the modality data x is organized as "bag" containing numerous instances. To learn a compact bag via IB, one potential solution is to directly employ the variational approximation (2(22)) of Eq. (4) in VIB to learn representation for each instance in the bag. However, the trawbacks of this solution are two-fold. First, is challenging to derive the overall distribution of the entire bag p(2[x) for bag based such large number of individual instance distributions, leading to a high-dimensional computational challenge. That is, the posterior distribution p(= x) with respect to high-dimensional of the second term in Eq. is intractable. Second, since the distribution of each instance is individually learned, it is difficult to capture bag- level information for representing a compact bag. Therefore, we propose Prototypical Information Bottleneck (PIB) to directly approximate bag-level distribution p(z[x) with parametric distribution p(2) represented by a group of prototypes, denoted as P (=i ply, E,y) | (including scenarios with censored and uncensored data). To capture discriminative information about task target, each prototype is supposed to represent a conditional probability distribution p(2ly) = (2; Mys Mng) for its corresponding risk band y. Then, instances z of a bag are expected to approach & with the same label y. Hence, the objective of variational approximation in Eq. should become:
p(z = =[<2|x, y) = p(gly)
(5)
To achieve this objective, we maximize the similarity between p(=) and spatial distributions of latent features z fx(x) for a bunch of instances, where an MLP is utilized as representation encoder fa(-) to map the input x to latent features z. As a result, we just need to optimize the parametric prototypes & and f-) for a bag x, instead of modeling p(z(x) for each instance of the bag.
In detail, to align the distribution of latent features and parametric prototypes é, we first sample some features from various prototypes via Monte Carlo sampling (to simplify the mathematical notation, we assume sampling once from each prototype). Then, we attempt to maximize the similarities between postive prototypes 2 (with true label) and the most related instances, while minimizing these instances with other negative prototypes é.. For example, given the i-th patient
data, we have the bag features (6) fx(x(() = { J and the features ( = { J sampled from prototypes, where M is the number of instances in bag x(1) 2N, is the number of
prototypes. Then, we measure the similarity between each prototype 2! and bag z as:
M Σ
Sim (2
(6)
M
where d(-) can be any similarity measure and we use cosine in our experiments. To eliminate redundant instances unrelated to risk prediction, we select portion of instances with higher similarity scores in a bag, while the discarded instances do not contribute to the learning process. During training, since we have access to the true label, the objective of approximating p( p(z|x, y) with prototypes p(žly) in (5) can be achieved by gathering these most related instances closer to positive (+) prototypes while pushing them away from negative (-) ones, formulated as:
ND
2N.-
Cpro ND
Sim ( ((0) + 2N,
Σ
Sim
==1
where ((00
(2
A1 j Mirr, d( 2.00 > d(2(() (4)
22+1) represents the retained
instances containing task-related discriminative information with higher similarities. The retained number Mr is determined by the hyperparameter Irr, the information retention rate (Irr), which controls the proportion of redundancy removal achieved by prototypes.
To review the objective of IB, we substitute the prototypes into the IB objective function in Eq. After getting the approximation p(žly) for p(z x, y) or p(z[x) in Eq. (5. we can conduct
similar derivation like from (q) to Eq. 3 (Details can be found in Appendix B.2.2) to obtain the
objective loss function of PIB to be minimized follows:
2N, Ee-p((/pa>>--oooe (vm,))) + 3KLp(())))
JPIB 2N,
(8)
where the first term is a cross- Coooss- loss for learning discriminative features. Since we are dealing with survival prediction task with labels containing survival time and censoring status, we the
5

---

Published as a conference paper at ICLR 2024
Puellive Proteti pe
MI.P
Qx
Pol
Sampling
Attietoo
I(S.
P(r
V.
XL Estimator
Joint Prototypical Distribution
Qh
Albeniam
Muli-bo
Seit.
Concal
Sh
mtn (S
Average
MI F Estimation
Allretker
Mall
Seff.
Weight Sharing
Figure 2: Disentangled Transformer. The self-attention is employed to model the intra-modal interactions while a token sampled from the joint prototypical distribution is used to guide common information extraction through cross-attention.
task-loss NLL in [ as an alternative for the first term. Finally, combining the approximation term Lpro, we obtain the total loss function for PIB to be minimized as follows:
2N.
LPIB 2N. 1=1
3K r()]) + Xrrr fa (2(m), (
where (2; M4n En.) p()) a, 8, 7 are the hyperparameters which control the impact of items. As a result, the modeled PIB can guide the extraction of discriminative features and the removal of redundant information for each modality organized bags.
3.3 PROTOTYPICAL. INFORMATION DISENTANGLEMER
After eliminating redundancy from unimodal sources, we propose a Prototypical Information Disentanglement (PID) module to decouple the shared and specific representations, addressing the "onter-modal redundancy". Suppose the instances selected by PIB are 2 and z we hope to decompose entangled multimodal data into ideally independent modality-common features (00)) and sodality-speccfii features 5/ st To achieve this, we reuse the joint prototypical distributions modeled by PIB for extracting common knowledge. These common features can be further used as guidance for learning modality-speciff knowledge by enforcing specific knowledge independent from these shared features. Thus, we minimize the mutual information (MI) between common and specific factors to preserve modality-specific information. Consequently, our objective is to ensure the independence of specific representations within each modality and also the independence between common and specific features. The loss function of PID can be formally expressed as:
LPID [S, C) + I(S,.S,) S,)), where S (Sp. where, S denotes all specific representations obtained by concatenating Cal(-) the features Sr.S, from each modality. As MI intractable, we introduce an upper bound CLUB (Cheng et al.) (0200) to accomplish MI minimization in Eq. 10 (Details about CLUB can be found in Appendix B.3).
(10)
To implement the above loss, we design a disentangled layer called disentangled transformer shown in Figure? 2 This transformer models various interactions within the inputs thereby obtaining the features Sy and C required in TO). We initially extract the common information guided by the joint prototypical distribution, denoted the joint posterior distribution p (z(xh.xg) which is defined by the product-of-expers (PoE) (Cao & Fleet) 2014), an idea of combining several distributions ("experts")) by multiplying them. Since have previously obtained the positive prototype in PIB, which approximates the distribution p(2[x x) of the patient's 's risk band, the p- (z) Xh, xg) can be formulated into:
p[=|X,, Xx, X,) (x)))((((x)((////
(11)
where p(2)xx) NN 14 ) p(2|x,) = NN Et
where p(z) is the prior distribution and p(=|x) approximately equals to the distributions of the positive prototypes N/ jit, E+). We assume the prior distribution p(z) is spherical Gaussian N( po. 2o), thus it can be shown that the product of Gaussian distributions is also a Gaussian p(z|x,, xg) = N22 per )
Σe (20))
Σ
- pr (1O + Σ 1ー
(12)
BE (h.g))
ie (E(h.g

---

Published as a conference paper at ICLR 2024
Hence, we sample from p(2|x,, Xg) to obtain a guiding token for shared information extraction. The modality-common representations C' are then extracted by the cross- within the disentangled transformer. Moreover, for the mopality-specific information, self-attention encodes pathway-to-pathwar and patch-to-patch interactions, and their mean representation becomes Sh. Sy. Thus, under the constraint of Eq. (TO), can simultaneously extract compact features that contain both specific and common information.
Overall Loss. The final loss of PIBD is as follows, where L'P1 and Ciprn represent the PIB loss formulated in Eq. for pathology and genomics modalities, respectively:
(13) where À is the weight factor that controls the impact of loss item, as well as a, 3, 7 in Eq. Note that the proposed method can be extended to more multimodal data of the bag structure.
L = L + C + LPiB 1 + ALPID
Inference. The inference process differs from the training mainly in how we find the positive proto- types. During training. with known labels, we can directly obtain the joint prototypical distribution for PID. However, in inference, we need to identify the positive one from the set of prototypes. To achieve this, we first select instances with higher similarity scores calculated with all prototypes in Eq. (7). These selected instances are considered relevant instances. Among them, the prototype with the highest proportion of relevant instances is considered positive. Hyperparameters such as the number of samples and information retention rate remain consistent with the training process.
4 EXPERIMENT
4.1 DATASET AND IMPLEMENTATION DETAILS
We conduct extensive experiments over five public cancer datasets from TCGA Breast Invasive Carcinoma (BRCA), Bladder Urothelial Carcinoma (BLCA), Colon and Rectum Adenocarcinoma (COADREAD), Stomach Adenocarcinoma (STAD), and Head and Neck Squamous Cell Carcinoma (HNSC). We follow the work (Jaume et al. 2023) to collect the biological pathways as genomics data. 5-fold cross-validation for each dataset is employed. The models are evaluated using the concordance index (C-index) (Harrell Jr et al. 1996) and its standard deviation (std) to quantify the performance of correctly ranking the predicted patient risk sores. We also visualize the Kaplan- Meier (KM) Kaplan & Meier (1958) curves that can show the survival probability of different risk groups. The details of the dataset and experimental implementation can be found in Appendix C.1
4.2 COMPARISONS WITH STATE-OF-THE THE-ARTS
We compare our method with three groups of SOTA methods: (1) Unimodal methods. For pathways data, we adopt MLP (Haykin 1998). SNN (Klambauer et al.) 2017), and SNNTrans Klambauer et al., 2017 Shao et al. 2021) 1) as the genomic baselines. For histology, we compare with SOTA MIL. methods ABMIL (Isee et al. 2018). AMISI (Yao et al. 2020). TransMIl. (Shao et al. 2021) and CLAM (Lu et al. 2021). ). (2) Multimodal methods. Four SOTA methods compared in this group: Porpoise (Chen et al.) (022b)), MCAT (Chen et al. 2021), MOTCat (Xu & Chen 2023), and SurvPath (Jaume et al. 2023). where we adopt two
approaches including concatenation (Cat) and Kronecker product (KP) for both Porpoise and MCAT. Besides, a prediction-level combination using a CoxPH (Cox) 1972) model of risk scores from the best- performing methods of genomics and histology is also conducted.
(3) Information theory-based methods. As our work provides an information theory perspective on multimodal cancer survival prediction, we also compare it with information theory-based methods in multi-view, multi-modal and task-specific fine-tuning domains, including CLAM-SB-FT (Li et al. 2023a), MIB (Federici al., 2020), DeepIMV (Lee & Van der Schaar (021), and L-MIB (Mai et al., 2022). Note that although CLAM-SB-FT is an IB-based method for WSIs, it is designed within fine-tuning framework and not be studied in multimodal survival prediction.
Comparison. From the results in Table [ we can observe that PIBD achieves the best overall performance across five cancer datasets. Compared with unimodal methods most multimodal methods* including ours show higher overall C-index, indicating that the information from both modalities gives great perspectives and contributions to survival prediction. Note that among multimodal methods, the proposed PIBD achieves superior performance in 4 out of 5 benchmarks and outperforms the second-best method by 1.6% in overall C-index, revealing the importance of addressing intra-modal and inter-modal redundancy. Then, from the comparison between IB-based
hhttss://portaa gde. sancer.gov

---

Published as a conference paper at ICLR 2024
Table 1: C-index (mean + std) over five cancer datasets. g. and h. refer to genomic modality and histological modality, respectively. The best results and the second-bes best results are highlighted in bold and in underline. A method marked with the subseript i falls into the unimodal group, & into the multimodal group, and into the information theory-based group.
BRCA
BLCA
COADREAD
HNSC
STAD
Model
Modality
Overall
(N=869)
(N=359)
(N=296)
(N=392)
(N=317)
'MLP
g.
0.622 = 0.079
0.530 0.077
0.712 0.1114
0.520 0.064
0.497 : 0.031
0.576
'SNN
0.621 = 0.073
0.521 = 0.070
0.711 = 0,162
0,514 = 0.076
0.485 :: 0.047
0,570
'SNNTrans
0.679 = 0.053
0.583 0.060
0.739 =: 0.124
0.570 = 0.035
0.547 + 0.041
0.622
'ABMIL
0.672 = 0.051
0.624 = 0.059
0.151
0.624 0.042
0.636 0.043
0.657
1AMISL
b.
0.681 = 0.036
0.627 = 0.032
0.710 0.091
0.607 = 0.048
0,553 + 0.012
0.636
'TranMIL
0.663 = 0.053
0.617 0.045
0.747 = 0.151
0.619 = 0.062
0.660 +: 0,072
0.661
CCLAM-SB
0.675 0.074
0.643 0.044
0.048
0.616 0.078
1CLAM-MB
0.696 + 0.098
0.623 + 0.045
0.721
0.159
0.620 0.034
0.648 + 0.050
0.662
SSNNTans+CLAMMMM
g.
0.699 0.064
0.625 0.060
0.716 0,160
0.638 0.066
0.629 + 0,065
0.661
Porppoise(Cat)
g.
0.668 0.070
0.617 0.056
0.738
0.151
0.660
0.106
0.660
PPr)poise(KP)
g.
0.691 + 0.038
0.619 + 0.055
0.721 + 0.157
0.630 0.040
0.661
0.085
0.664
'MCAT(Cat)
g.
0.685 0.109
0.640 0.076
0.724 0.137
0,564 :: 0.840
0.625 0,118
0.647
*MCAT(KP)
g.+h.
0.727 0.027
0.644 0.062
0.709 0.162
0.093
0.643 0.075
0.668
*MOTCat
g.
0.727 = 0.027
0.659 = 0.069
0.742 0.124
0.656 0.041
0.621 + 0.065
0.681
'SuryPath
g.
0.724 + 0.094
0.660 0.054
0.758 + 0,143
0.606 0.080
0.667 + 0.035
0.683
"CLAM-SB-FT
0.606 0.110
0.633 0.065
0.725 0.150
0.620 0.084
0.654 0.051
0.648
"MIB
g.
0.602 = 0.112
0.573 = 0.036
0.711
0.182
0.555 0.055
0.588 0.057
0.606
"DeepIMV
g.
0.659 + 0.089
0.638
0.054
0.749 + 0,145
0.604 + 0.061
0,597 + 0.047
0.649
*I--MBB
0.687 0.071
0.662
0.093
0.720
0.167
0.615
0.085
0.634 0.060
0.664
*PPBBD
g.th.
0.736 = 0.072
0.667 0.061
0.768
0.124
0.640 #: 0.039
0.684
0.035
0.699
BLCA
BRCA
COADREAD
HNSC
STAD
12.210.5544188 (8
SursPath
15331.91102246777
25.1313 84K.39)
32.771.95j24 499224
18.700.772339911.0
1502Q135/286.211.0
OHTS
20.150200251166999
13
Figure 3: Kaplan- Meinrr curves of predicted high-risk (red) and low-risk (green) groups. A P-value < 0.05 indicates statistical significance, and the shaded regions represent the confident intervals. The median survival months are reported in the format of "high-risk: mean(std)/low-risk mean(std)"
methods, our method achieves superior performance performance gains. PIBD, which fully considers the characteristics of bag structure under weak supervision and is designed for multimodal cancer survival prediction, demonstrates its superiority.
all cancer datasets, with 0.5%-4.9%
Kaplan-Meier analysis We further evaluate our method using statistical analysis, and the Kaplan-Meier curves are presented in Figure 3 Patients are separated into high-risk and low-risk groups based on predicted risk scores, with the median value of each validation set serving as the off- Subsequently, we utilize the log-rank test to compute p-values, which assess the statistical significance of differences between these groups, and the median survival months are also reported for each group. Our approach demonstrates significantly improved discrimination between the two groups when compared to the second-best method, SurvPath. This effect is particularly pronounced in the BRCA, COADREAD, and HNSC datasets, with substantial margins of magnitude.
4.3 ABLATION STUDY
Component validation. In Table we ablate the designs mentioned in Sections 3.2 and 3.3 which
proposed for "inter-modal redundancy" and "intra-modal redundancy".. For ablating PIB, we
8

---

Published as a conference paper at ICLR 2024
Table 2: Ablation study assessing C-index (mean + std).
Variants
PIB
PID
BRCA
BLCA
COADREAD
HNSC
STAD
Overall
AP
0.684 + 0.044
0.619
0.090
0.713 + 0,161
0.567 + 0.073
0.609 + 0.048
0.638
PIR(AP)
0.705 0.108
0.593
0.038
0.753
0.143
0.623 0.107
0.613 0.071
0.657
TransMIL
0.672 0.088
0.636
0,059
0.750
0.133
0.591
+: 0.080
0.662 + 0.090
0.662
PIB(TrunMMIL)
0.696 0.069
0.648
0.074
0.757
0.176
0.615
0.062
0.643 0.074
0.672
PIBD
0.736 0.072
0.667 0.061
0.768 0.124
0.640
0.039
0.684 0.035
0.699
Table 3: Interventions in PIB. We conduct interven- tions by either removing the positive prototype or ran- domly deleting one of the negative prototypes.
Intervention
BLCA
COADREAD
STAD
Positive
0.401 + 0.086
11.471 + 0.196
0.384 + 0,110
Histological Protodypes
Genomic Protutypes
Negative
0.645
0.067
0.731
0.106
0.672 + 0.055
w/o Intervention
0.667 0.061
0.768 0.124 0.684 + 0.035
Figure 4: Visualization of prototypes.
tablished two baselines: one involves direct average pooling (AP) on original features, and the other employs a non-disenin -ouddisettdglee TransMIL encoder as a strong baseline. We incorporate PIB into both baselines to assess the effectiveness of the prototypical features selected by PIB. As shown in the first four rows of Table the addition of PIB outperforms the baselines in terms of higher C-index. This suggests that learning multiple distinctive prototypes in PIB and employing them to filter task-related features can effectively mitigate redundant features within each modality. For ablating PID, we con- duct comparison between our PIBD and the baseline using the non-disentangled TransMIL with PIB. The last two rows demonstrate that disentangling shared and specific information from multi- modal data effectively eliminates inter-modal redundancy, preventing the loss of information during the fusion process and significantly enhancing the model's performance. More- over, we conduct more quantitative studies about parameter settings presented in Appendix C.2
Interpretability of PIB. To validate that the learned prototypes in PIB have modeled discriminative underlying distributions for different risk bands, we conduct random sampling on each prototype with a frequency of 2000. Subsequently, we reduce the obtained high-dimensional vectors to two-dimensional plane using 1-SNE (Van der Maaten Hinton 2008). As illustrated Figure 4 the distributions exhibit excellent separability. Furthermore, inspired by the intervention in (Sarkar cet al. 2002), we conduct interventions during the inference process shown in Table and the results demonstrated significant disparity. It can be seen that interventions in positive prototypes led to a dramatic decrease in the C-Index (all below 0.5), signifying a complete loss of predictive ability. Intervention in positive prototypes further results in passing wrong guided signal to the following disentanglement module PID with an incorrect prototypical distribution as well, leading to worse performance. Conversely, when randomly removing a negative prototype, there was only a slight decline in the C-Index, which further underscores the effective modeling of discriminative risk- distributions in PIB. Visualization of similarity scores for both modalities are presented in Appendix D
CONCLUSION
In this work, we explore multimodal cancer survival prediction inspired by information theory and propose a new framework called PIBD aimed at addressing both "intra-model redundancy" and "inter-model redundancy" challenges. First, we propose Prototypical Information Bottleneck (PIB) that reduces redundancy while preserving task-related information. PIB models prototypes of various risk bands, allowing us to select discriminative features from massive instances and alle- viating "intra-model redundancy". Furthermore, to address "odter-modal redundancy",. we propose a Prototypical Information Disentanglement (PID) to decouple independent modality-common and modality features with the guidance of the joint prototypical distribution. These compact features offer distinct perspectives and knowledge, effectively enhancing the network's performance. Moreover, to handle the high-dimensional computational challenges inherent in our task, the PIB models prototypes approximating a bunch of instances by maximizing the cosine similarities within true labels. During this approximation, the choice of an appropriate similarity metric can contribute to better aligning spatial distributions, which warrants further investigation in future research endeavors.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
This work was supported by National Natural Science Foundation of China (No. 62202403), Shenzhen Science and Technology Innovation Committee Funding (Project No. SGDX20210823 10320100),), the Research Grants Council of the Hong Kong Special Administra- tive Region, China (Project No. R6003-22 and C4024-22GF).
REFERENCES
Alexander A Alemi, Ian Fischer, Joshua V Dillon, and Kevin Murphy. Deep variational information bottleneck. In International Conference on Learning Representations, 2016.
Gabriele Campanella, Matthew G Hanna, Luke Geneslaw, Allen Miraflor, Vitor Werneck Krauss Silva, Klaus J Busam, Edi Brogi, Victor E Reuter, David S Klimstra, and Thomas Fuchs. Clinical-grade computational pathology using weakly supervised deep learning on whole slide images. Nature medicine, 25()):1301-1309 2019.
Yanshuai Cao and David Fleet. Generalized product of experts for automatic and principled fusion of gaussian process predictions. arXiv preprint arXiv:1410.7827. 2014.
Richard J Chen, Ming Y Lu, Jingwen Wang, Drew FK Williamson, Scott J Rodig, Neal Lindeman, and Faisal Mahmood. Pathomic fusion: integrated framework for fusing histopathology and genomic features for cancer diagnosis and prognosis. IEEE Transactions on Medical Imaging, 41 (4):757-770, 2020.
Richard J Chen, Ming Lu. Wei-Hung Weng. Tiffany Y Chen, Drew FK Williamson, Trevor Manz, Maha Shady, and Faisal Mahmood. Multimodal co-attention transformer for survival prediction in gigapixel whole slide images. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 0015-4025,, 2021.
Richard Chen, Chengkuan Chen, Yicong Li, Tiffany Y Chen, Andrew D Trister, Rahul G Kr- ishnan, and Faisal Mahmood. Scaling vision transformers to gigapixel images via hierarchical self-supervised learning. In Proceedings of the IEEF/CVF Conference Computer Vision and Puttern Recognition, pp. 161 16144-16155, 2022a.
Richard Chen, Ming Y Lu, Drew FK Williamson, Tiffany Y Chen, Jana Lipkova, Zahra Noor, Muhammad Shaban, Maha Shady, Mane Williams, Bumjin Joo, et al. Pan-cancer integrative histology-genen analysis via multimodal deep learning. Cancer Cell, 40(8):865-88)) 878, 2022b.
Yuanyuan Chen, Yongsheng Pan, Yong Xia, and Yixuan Yuan. Disentangle first, then distill: A unified framework for missing modality imputation and alzheimer's disease diagnosis. 1EEE Transactions on Medical Imaging, 2023.
Mingyuan Cheng, Xinru Liao, Quan Liu, Bin Ma, Jian Xu, and Bo Zheng. Learning disentangled representations for counterfactual regression via mutual information minimization. In Proceed- ings of the 45th International ACM SIGIR Conference on Research and Development in Informa- fion Retrieval, pp. 1802-1806 2022.
Pengyu Cheng. Weituo Hao, Shuyang Dai. Jiachang Liu. Zhe Gan, and Lawrence Carin. Club: A contrastive log-ratio upper bound of mutual information. In International conference on machine learning, pp. 1779-1788. PMLR, 2020.
Yann Christinat and Wilhelm Krek. Integrated genomic analysis identifies subclasses and prognosis signatures of kidney cancer. Oncotarget, 6, 03 2015. doi: 10. 18632/omcotargee.322..4
David R Cox. Regression models and life-tables. Journal of the Royal Statistical Society: Series (Methodological), 34(2):187-202, 1972.
David R Cox. Partial likelihood. Biometrika, 62(2):269-276 1975.
Yinglong Dai, Zheng Yan, Jiangchang Cheng, Xiaojun Duan, and Guojun Wang. Analysis of mul- timodal data fusion from an information theory perspective. Information Sciences, 623:164-188 2023.
10

---

Published as a conference paper at ICLR 2024
J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. ImageNet: A Large-Scale Hierarchical Image Database. In CVPR09, 2009.
Andrew Evans, Thomas Bauer, Marilyn Bui, Toby Cornish, Helena Duncan, Eric Glassy, Jason Hipp, Robert McGee, Doug Murphy, Charles Myers, Dennis O' O'Neill, Anil Parwani, Rampy, Mohamed Salama, and Liron Pantanowitz. Us food and drug administration approval of whole slide imaging for primary diagnosis: A key milestone reached and new questions are raised. Archives of pathology laboratory medicine, 142, 04 2018. doi: 1.58%%arpa22017-4496--C0
Marco Federici, Anjan Dutta, Patrick Forré, Nate Kushman, and Zeynep Akata. Learning robust rep- resentations via multi-view information bottleneck. In 8th International Conference on Learning Representations. OpenReview. net, 2020.
Feng and Zhi-Hua Zhou. Deep miml network. Proceedings of the AAAI conference on artificial intelligence, volume 31, 2017.
Mare Gillespie, Bijay Jassal, Ralf Stephan, Marija Milacic, Karen Rothfels, Andrea Senff-Ribeirg Johannes Griss, Cristoffer Sevilla, Lisa Matthews, Chuqiao Gong. et al. The reactome pathway knowledgebase 2022. Nucleic acidy research, 50(D1):D687-D692, 2022.
Yonghang Guan, Jun Zhang, Kuan Tian, Sen Yang. Pei Dong, Jinxi Xiang, Wei Yang, Junzhou Huang, Yuyao Zhang, and Xiao Han. Node-aligned graph convolutional network for whole- slide image representation and classification. In Proceedings of the IEEE/CVE Conference on Computer Vision and Puttern Recognition, pp. 18813-18823, 2022.
Chuan Guo, Florian Bordes, Pascal Vincent, and Kamalika Chaudhuri. Do ssl models have d' aejia vu? a case of unintended memorization in self-supervised learning. arXiv preprint arXiv:2304. 13850, 2023.
Balázs Györffy. Survival analysis across the entire transcriptome identifies biomarkers with the highest prognostic power in breast cancer. Computational and structural biotechnology journal, 19:4101-4109, 2021.
Wei Han, Hui Chen, and Soujanya Poria. Improving multimodal fusion with hierarchical mutual information maximization for multimodal sentiment analysis. arXiv preprint arXiv:2109.00411 2021.
Frank E Harrell Jr, Kerry L Lee, and Daniel B Mark. Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors. Statistics medicine, 15(4)33613377 1996.
Simon Haykin. Neural networks: a comprehensive foundation. Prentice Hall PTR, 1998.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.
R Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon. Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. arXiv preprint arXiv: 1808.06670, 2018.
Mahdi S Hosseini, Babak Ehteshami Bejnordi, Vincent Quoc-Huy Trinh, Danial Hasan, Xing- wen Li, Tachyo Kim, Haochen Zhang, Theodore Wu, Kajanan Chinniah, Sina Maghsoud- lou, et al. Computational pathology: A survey review and the way forward. arXiv preprint 2rX.v:2304.05482, 2023.
Le Hou, Dimitris Samaras, Tahsin M Kure, Yi Gao, James E Davis, and Joel H Saltz. Patch-base convolutional neural network for whole slide tissue image classification. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1424-2443,, 2016.
Shih- Shih-Cheng Huang, Anuj Pareek, Saeed Seyyedi, Imon Banerjee, and Matthew P Lungren. Fusion of medical imaging and electronic health records using deep learning: a systematic review and implementation guidelines. NPJ digital medicine, 3(1):136, 2020.
11

---

Published as a conference paper at ICLR 2024
Maximilian llse, Jakub Tomezak, and Max Welling. Attention- based deep multiple instance learn- ing. In International conference on machine learning, pp. 2127-2136. PMLR, 2018.
Hartland W Jackson, Jana Fischer, Vito RT Zanotelli, H Raza Ali, Robert Mechera, Savas D Soysal, Holger Moch, Simone Muenst, Zsuzsanna Varga, Walter P Weber, et al. The single-cell pathology landscape of breast cancer. Nature, 578(7796)6115-620 2020.
Alon Jacovi, Avi Caciularu, Omer Goldman, and Yoav Goldberg. Stop uploading test data in plain text: Practical strategies for mitigating data contamination by evaluation benchmarks. arXiv preprint arXiv:2305. 10160, 2023.
Guillaume Jaume, Anurag Vaidya, Richard Chen, Drew Williamson, Paul Liang, and Faisal Mah- mood. Modeling dense multimodal interactions between biological pathways and histology for survival prediction. arXiv preprint arXiv:2304.068// 2023.
Stephen P Jenkins. Survival analysis. Unpublished manuscripi, Institute for Social and Economic Research, University of Essex, Colchester, UK, 42:54-56, 2005.
Xuewen Jiang, Yangyang Xia, Hui Meng, Yaxiao Liu, Jianfeng Cui, Huangwei Huang, Gang Yin, and Benkang Shi. Identification of nuclear mitochondrial-eelttee multi-genes signature to pre- dict the prognosis of bladder cancer. Frontiers in Oncology, 11:746029, 2021a.
Yuming Jiang, Cheng Jin, Heng Yu, Jia Wu, Chuanli Chen, Qingyu Yuan, Weicai Huang, Yanfeng Hu, Yikai Xu, Zhiwei Zhou, al. Development and validation of deep learning signature to predict survival and chemotherapy benefit in gastric cancer: a multicenter, retrospective study. 2021b.
Annals of surgery,
Edward L Kaplan and Paul Meier. Nonparametric estimation from incomplete observations. Journal of the American statistical association, 53(282):457-481 1958.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:14122 44/2.6980. 2014.
Günter Klambauer, Thomas Unterthiner, Andreas Mayr, and Sepp Hochreiter. Self-normalizing neural networks. Advances in neural information processing systems, 30, 2017.
Changhee Lee and Mihaela Van der Schaar. A variational information bottleneck approach multi- omics data integration. International Conference on Artificial Intelligence and Statistics, pp. 1513-1521. PMLR, 2021.
Bin Li, Yin Li, and Kevin W Eliceiri. Dual-stream multiple instance learning network for whole slide image classification with self-supe contrastive learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 14318-14328, 2021.
Honglin Li, Chenglu Zhu, Yunlong Zhang, Yuxuan Sun, Zhongyi Shui, Wenwei Kuang, Sunyi Zheng, and Lin Yang. Task-specific fine-tuning via variational information bottleneck for weakly- supervised pathology whole slide image classification. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pp. 7454-7463, 2023a.
Ruiqing Li, Xingqi Wu, Ao Li, and Minghui Wang. Hfbsurv: hierarchical multimodal fusion with factorized bilinear models for cancer survival prediction. Bioinformatics, 38(9):2587-259, 2022.
Zhe Li, Yuming Jiang, Mengkang Lu, Ruijiang Li, and Yong Xia. Survival prediction via hierar- chical multimodal co-attention transformer: A computational histology-radiology solution. IEEE Transactions on Medical Imaging, 2023b.
Paul Pu Liang, Yun Cheng, Xiang Fan, Chun Kai Ling, Suzanne Nie, Richard Chen, Zihao Deng, Faisal Mahmood, Ruslan Salakhutdinov, and Louis-Philippe Morency. Quantifying & modeling feature interactions: An information decomposition framework. arXiv preprint 1222::2302..2247, 2023.
Arthur Liberzon, Chet Birger, Helga Thorvaldsdóttir, Mahmoud Ghandi. Jill P Mesirov, and Pablo Tamayo. The molecular signatures database hallmark gene set collection. Cell systems, 1(6): 417-425, 2015.
12

---

Published as a conference paper at ICLR 2024
Pei Liu, Luping Ji, Feng Ye, and Bo Fu. Advmil: Adversarial multiple instance learning for the survival analysis on whole-slide images. arXiv preprint arXiv:2212.06515, 2022.
Ming Y Lu, Drew FK Williamson, Tiffany Y Chen, Richard J Chen, Matteo Barbieri, and Faisal Mahmood. Data-efficient and weakly supervised computational pathology on whole-slide images. Nature biomedical engineering, 5(6):555-5700 2021.
Sijie Mai, Ying Zeng, and Haifeng Hu. Multimodal information bottleneck: Learning minimal sufficient unimodal and multimodal representations. IEEE Transactions Multimedia, 2022.
Nathan Mantel et al. Evaluation survival data and two new rank order statistics arising in its consideration. Cancer Chemother Rep, 50(3):163-170, 1966.
Pooya Mobadersany, Safoora Yousefi, Mohamed Amgad, David A Gutman, Jill Barnholtz-Sloan, José E Velázquez Vega, Daniel J Brat, and Lee AD Cooper. Predicting cancer outcomes from histology and genomics using convolutional networks. Proceedings of the National Academy of Sciences, 115(13)=2970-E2) E2979, 2018.
Hui Qu, Mu Zhou, Zhennan Yan, He Wang, Vinod K Rustgi, Shaoting Zhang, Olivier Gevaert, and Dimitris N Metaxas. Genetic mutation and biological pathway prediction based on whole slide images in breast carcinoma using deep learning. NPJ precision oncology, 5(1):87, 2021.
Stephen Salerno and Yi Li. High-dimensional survival analysis: Methods and applications. Annual review of statistics and its application, 10:25-49, 2023.
Eduardo Hugo Sanchez, Mathieu Serrurier, and Mathias Ortner. Learning disentangled representa- tions via mutual information estimation. Computer ference, Glasgow, UK, August 23-28 2020, Proceedings, Part XXII 16, pp. 205-221. Springer, 2020.
2020: 16th European Con-
Anirban Sarkar, Deepak Vijaykeerthy, Anindya Sarkar, and Vineeth N Balasubramanian. A frame- work for learning ante-hoc explainable models via concepts. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 10286-10293 2022.
Zhuchen Shao, Hao Bian, Yang Chen, Yifeng Wang, Jian Zhang, Xiangyang Ji, et al. Transmil: Transformer based correlated multiple instance learning for whole slide image classification. Ad- vances in neural information processing systems,
2021.
Aravind Subramanian, Pablo Tamayo. Vamsi K Mootha, Sayan Mukherjee, Benjamin L Ebert, Michael Gillette, Amanda Paulovich, Scott Pomeroy, Todd R Golub, Eric S Lander, et al. Gene set enrichment analysis: knowledge besee- approach for interpreting genome-wide expres- sion profiles. Proceedings of the National Academy of Sciences, 102(43):15545-1555 2005.
Naftali Tishby, Fernando C Pereira, and William Bialek. The information bottleneck method. arXiv preprint physics/0004057, 2000.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 911)), 2008
Xiyue Wang, Sen Yang. Jun Zhang, Minghui Wang, Jing Zhang, Wei Yang. Junzhou Huang, and Xiao Han. Transformer-based unsupervised contrastive learning for histopathological image classification. Medical Image Analysis, 81:102559, 2022. ISSN 1361-8415. doi: 10255,,,
102559. [RLl:ttpes//wwssceeeedeererr www.ncedirecctte..eeeeeeeeeeeeee com/ secience/artiicle/p//////// TTt.///3311111122002022
Zhiqin Wang, Ruiqing Li, Minghui Wang, and Ao Li. Gpdbn: deep bilinear network integrating both genomic data and pathological images for breast cancer prognosis prediction. Bioinformatics, 37 (18):2963-2970, 2021.
Yingxue Xu and Hao Chen. Multimodal optimal transport-basee co-attention transformer with global structure consistency for survival prediction. Proceedings of the IEEE/CVF International Conference Computer Vision, 2023.
13

---

Published as a conference paper at ICLR 2024
Jiawen Yao, Xinliang Zhu, Jitendra Jonnagaddala, Nicholas Hawkins, and Junzhou Huang. Whole slide images based cancer survival prediction using attention guided deep multiple instance learn- ing networks. Medical Image Analysis, 65:101789, 2020.
Jiawen Yao, Yu Shi, Kai Cao. Le Lu, Jianping Lu, Qike Song, Gang Jin, Jing Xiao, Yang Hou, and Ling Zhang. Deepprognosis: Preoperative prediction of pancreatic cancer survival and surgical margin via comprehensive understanding of dynamic contrast- -ehaanet- ct imaging and tumor- vascular contact parsing, Medical image analysis, 73: 13:10105, 2021.
Shekoufeh Gorgi Zadeh and Matthias Schmid. Bias in cross-entropy- -baed training of deep survival networks. IEEE transactions on pattern analysis and machine intelligence, 2020.
Chao Zhang, Zichao Yang, Xiaodong He, and Li Deng. Multimodal intelligence: Representation learning, information fusion, and applications. IEEE Journal of Selected Topics in Signal Pro- cessing, 14(3):478-493, 2020.
Yilan Zhang. Fengying Xie, and Jianqi Chen. Tformer: A throughout fusion transformer for multi- modal skin lesion diagnosis. Computers in Biology and Medicine, 157:106712, 2023.
Fengtao Zhou and Hao Chen. Cross-modal translation and alignment for survival analysis. Pro- ceedings of the IEEE/CVF International Conference on Computer Vision, 2023.
Xinliang Zhu, Jiawen Yao, Feiyun Zhu, and Junzhou Huang. Wsisa: Making survival prediction from whole slide histopathological images. In Proceedings the IEEE conference on computer vision and pattern recognition, pp. 7234-7242, 2017.
Yingli Zuo, Yawen Wu, Zixiao Lu, Qi Zhu, Kun Huang, Daoqiang Zhang, and Wei Shao. Identify consistent imaging genomic biomarkers for characterizing the survival-associated interactions be- tween tumor-infiltratin lymphocytes and tumors. In International Conference on Medical Image Computing and Computer-Assian Intervention, pp. 222-231. Springer, 2022.
14

---

Published as a conference paper at ICLR 2024
A OVERVIEW
In this supplement, will first provide detailed formulations of the proposed PIBD in Appendix Then, we provide implementation details, more quantitive studies about parameter settings, and additional comparisons in Appendix C Finally, we display the visualization results of similarity scores for pathology and genomics modalities to further showcase our method's interpretability in Appendix D
B DETAILED FORMULATIONS
B. BAG FORMULATION
The bag construction process of histology and genomic data are formulated into a weakly supervised MIL task. Given WSI, we slice it into non-overleap Md patches and pre-trained visual encoder to extract the features, aiming to get the patch-level d-dimensional embedding of each instance denoted into x (=()) E 1= For genomic data, the bag formulation process is similar. Following previous works (Jaume et al. 2023) that tokenize genes into pathways involved in particular biological processes, we use multiple separated encoders to learn the pathway-level (5) embeddings. Then bag of genomic data can be built as x9 {x E { where M g is the total number of pathways.
B.2 DETAILED FORMULATION OF PROTOTYPICAL INFORMATION BOTTLENECK
B.2.1 VARIATIONAL INFORMATION BOTTLENECK (ALEMI ET AL., 2016)
The information bottleneck (IB) (Tishby et al. 2000) principle aims to find a better representation Z by maximizing the mutual information (MI) between the latent representation Z and label Y, and minimizing the MI between the Z and original input X. The objective function to be maximized in IB can be formulated as follows:
RIB= = ((2,) ()
(14)
where ((.-) represents the mutual information, and 3 is the Lagrange multiplier. The IB principle defines better representation, in terms of the tradeoff between a concise representation and pre- dictive power. However, the computation of MI is intractable, variational information bottleneck (VIB) (Alemi et al. 2016) is proposed to give a variational approximation solution of MI estimation, which allows the use of the deep neural network for efficient training.
Given the joint distribution p (X, Y, Z) as follows: P,X,Y, Y, Z) p(Z]X, ()p(Y = ((Xx))==((XX))(Xx//))0)))
(15)
where assuming p =((2),,, corresponding to the Markov chain Y ++ 2
Then, the first part / /(2 Y) can be derieved into:
I(Z,Y) / dydzp(z, p(u,z)
p())p(z)
(16)
/ duddyy/spcchploo))eettlllo
plulz) ply)
Since p(y|z) above is intractable, decoder qo (v]zz) is used variational approximate the z).
Using the fact that the Kullback-Leibler divergence is always positive, we have:
K KLpp(yl2), go(1))) dyp(v)2)\og p(y|z)
qo (v)
(17)
- (oplul=))ogaooelkk 0
15

---

Published as a conference paper at ICLR 2024
Hence, the lower bound of I(Z is as follows:
I(Z,Y) dadgydaafee/bow////ttea/oog p(ylz)
p(y)
duply))ogp(y)
(18)
drccyyaapp(hhhhlll/nnseetttoooo H(Y)
H(Y)
where (Y) is the entropy of label Y, which is independent of the optimization procedure and can therefore be ignored.
Similarly, the second part / (Z, X) can be formulated into its upper bound:
p(=[2r)
I(Z,X) = dadæp(æ, 200
p(z)
(ond=(())p(=]=)(og p ((2))))rz)
r(=)p(z)
(19)
dzd=p(z)p(zlz)\og p(=(20)
KL(p(2),r(())
r(=)
/
(rd=p(=))((1())log p(=|z)
r(z)
where p(z[x) is the posterior distribution over and r(2) is a variational approximation of p(z). as
the computation of p(2) f dxp(z|x)p(x)) might be difficult. To sum up, the IB objective function can be seen to minimize:
daddydapfccppulls)pplll//// o 3 dxd=p(=))(s/a))a p(z|z)
JIB
r(z)
(20)
1 N
Σ
E E (KKN::/co_....kkk.kKKK.Kk....drr
where N denotes the number of samples. r(z) can be assumed as spherical Gaussian described in Alemi al. (2016) practice. And the posterior distribution p(=[ær variationally approximated by an encoder:
p(2|z) x (2(xx) = Nix;
(21)
where fE is an MLP that predicts both the mean and covariance matrix L.
B.2.2 PROTOTYPICAL INFORMATION BOTTLENECK
As mentioned in Section 3.2 we propose an IB variant called Prototypical Information Bottleneck (PIB), which models the prototypes approximating bunch of instances for different risk bands.
In our task, given that the input x is "bag" structure containing numerous instances, the posterior distribution p(z(x) is intractable. Therefore, we choose to approximate x(2|x) with latent space distribution p(2) represented by group of prototypes P (2 My: Eny) 2y) where each proto- type represents conditional probability distribution p(žly) y)) (2; May Eg) under the label As result, we just need to optimize the parametric prototypes & and fø( for a bag x, instead directly
employing the modeling variational approximation (2(22)) of Fqq 21 in VIB to learn compact representation for each instance a bag.
Thus get the loss function for the approximation:
2N.-1
Cpro
-Sim (1) 2(0) + 2N,
Sim(
(1)
(0)
(22)
ND
Afterward, we substitute the prototypes into the objective function in Eq. and examine each item in turn. First, in our settings, since we aim approximate p(=|x) p(2|x, y) p(ll))
16

---

Published as a conference paper at ICLR 2024
by introducing the variable 2, actually add new chain Y + Z to the original Markov chain Y X Z. Then, the objective function to be maximized of PIB can formulated into: ReIB = Y) - -
(23)
where 2 is enforced approach Z. For the first item T(2, Y), we got the formulation based on Eq
18
/ dydēp(ā, y)ogg p
I(2,Y)
p(y)
(24)
dydēp( (anpn22yyylogll y)loggo(y13) + HYY)
In our setting, hence p(8) doesn't 't directly depend on z. In this case, p(2, y) can be formulated into: p(2,y) p(1ly)p(y)
(25)
and that we will take p(v) = 1
can bond the first term into:
2N.
1(2,8) Y) 2 / vlloggoyy)
f
(26)
2N.
1logoo [2)]
2N,
Then for the second item I(Z, X), we approximate p(z[x) with the latent space distribution repre- sented by IB-based prototypes:
1 2N.
p()) dyp(y)p(2lu)
Σ p()))n)
(27)
2N,
Hence, we can directly replace p(z|x) with p(2) to the second item of 20). expressed as:
(xdep(=))(=/z))log p(2|>)
T(Z, <
r(z)
(xddydpp<))p()))(auu p(v)p(āl))
r(z)
/ dyply) dydūp(y))(ály))l p(āly)
/
r(2)
(28)
2N
= -H(Y) + KT.pp((laa,,(())
2N
2N:
kL
2N.
So we obtain the objective loss function of PIB to be minimized as follows:
2N.
JPIB = 2N,
Σ
(logo (y,, + +3KKLp(()))))
(29)
where the first term is the task loss to learn discriminative features, we use the negative likeliieddooo (NLL) loss in Eq. as an alternative for the first term. Finally, after combining the approximation term Cpros we obtain the total loss function for PIB as follows:
2N, CaLsuro (m)) (()) ,,) BKLLW((;; pn, En),r(z)]} + Cpro
CPTB 2N,
Σ
(30)
n=1
17

---

Published as a conference paper at ICLR 2024
where (2; Jm: En) p(2()n), a, B.y the hyperparameters which control the impact of items.
B.3 DETAILED FORMULATION OF PROTOTYPICAL INFORMATION DISENTANGLEMENT
Since in (O). the computation of mutual information (MI) is intractable, we introduce an upper bound called contrastive log-ratio upper bound (CLUB) (Cheng et al. 2020) as an MI estimator to accomplish MI minimization, which can be used in more general scenarios. Given two variables and b, the ICLUB(a,b b) is calculated as follows:
ICLUB(a, b) = b) [logp(bla)] Bp(a) [oogpaog)(
logp(b.la;)
logp(byla:)
N
N2
(31)
[logp(bsla)) logp(b, |a:)]
N
where (ogp(b:la,) denotes the conditional log-likelihood of positive sample pair (b,, a:) while logp(byla:) is the conditional log-likelihood of negative sample pair.
However, in our work, both the modality-specifie features S₁ and Sg, as well as the modality- common features C. simultaneously generated by disentangled transformer, the conditional distribution p(bla) in (3) is unknown. We employ an MLP go to provide a variational approximation of p(bla), thus the variational CLUB (vCLUB), CLUB variant, is defined:
LCLUB(a, b) = Ep(a,)) logge (bla)] - Ep(a) Epc) (logge(b)))
N
N
loggo (be aa))
loggo (bylaz)
N
N2
(32)
N
N
loggo - loggo (oygoobble)) (byla:)
N
The variational approximation (b(a) can be optimized by maximizing the log- likkliihhood:
N
1
Lastimator (b,
Σ N
loggo (b:(a))
(33)
VCLUB still holds an upper bound on MI when the variational approximation go (bla) is reliable. Therefore, to train good estimator for the conditional distribution g(bla) is critical. this context, higher dimensional distribu- we choose to predict lower dimensional distribution conditioned
tion (Han et al. 2021) to avoid mode collapse. Following this, our work predicted (c(ss)) instead of (3(3)c for estimating mutual information in Eq. (10) where is modality-speciif distribution integrating two modalities with higher dimension, while e modality-common distribution with lower dimension. In conclusion, the proposed disentangle loss in Eq.(IO) be further defined as:
LPID LICUB(S, C) LOLUB(S, Sy)) + Lestimator (8, C) + Lestimator (Sh.
(34)
where S (St, Sg).
18

---

Published as a conference paper at ICLR 2024
C MORE EXPERIMENTS
C.I IMPLEMENTATION DETAILS
Dataset. We conduct extensive experiments over five public cancer datasets from TCGA Breast Invasive Carcinoma (BRCA), Bladder Urothelial Carcinoma (BLCA), Colon and Rectum Adeno- carcinoma (COADREAD), Stomach Adenocarcinoma (STAD), and Head and Neck Squamous Cell Carcinoma (HNSC). We follow the work (Jaume et al. 2023) to predict disease-speciffe survival (DSS), which is a more accurate representation of the patient's ' disease status than overall survival. For histological data, we collect all diagnosis WSIs used for primary diagnosis. For genomic data, we get the raw transcriptomics from the Xena database along with DSS labels. The human biologi- cal pathways QQu al. (02T), represented transcriptomics sets with specific interactions among molecules in cells, are collected from two resources by selecting where at least 90% of transcrip- tomic accessible: the Human Molecular Signatures Database (MSigDB)- Hallmarks (Subramanian
et al. Liberzon et al.) 2015) (50 pathways from 4,241 genes) and the Reactome (Gillespie
2005
ct al.
2022
(281 pathways from 1577 genes).
Evalution. We employ f-lold cross-validation for each dataset. The models are evaluated using the concordance index (C-index) (Harrell Jr et al., 1996) and its standard deviation (std) to quantify the performance of correctly ranking the predicted patient risk sores. We also visualize the Kaplan- Meier (KM) Kaplan & Meier (1958) curves that can show the survival probability of different risk groups. The log-rank statistical significance test Mantel et al. (9966) is performed to determine if the separation between these groups is statistically significant.
Bag constraction. We first segment the tissue from the images and extract non-overlapping 224 x 224 patches at the 20 magnification. Then a Swin Transformer (CTransPath) (Wang et al. 2022). which is pre-trained
more than 14 million pan-cancer histopathology patches via self-supervised contrastive learning, is used as the feature extractor to get 768-dimensional embeddings. Meanwhile, the feature extractors of pathways SNNs following the settings in works (Jaume et al. 2023 Xu
& Chen 2023: Chen et al. 2021).
Implementation. The proposed algorithm is implemented in Python with Pytorch library and runs on a PC equipped with an NVIDIA A 100 GPU. For the survival prediction task, we divide the overall survival time into four intervals, resulting in eight different risk bands when considering censorship status. Therefore, we set the number of prototypes in PIB to 8. We use an MLP with a 512-d hidden layer as the latent vector encoder (-) to embed the bag features into a fixed dimension of 256. The hyper-paramette a, 8, 7, and À are set to 1. 0.01, 1, and 0.1 respectively. We take the top 50% and 80% of the samples with the highest similarity to prototype as the retained features for histological data and genomics data, respectively, to remove redundancy within the modalities. To increase the variability during training, 4096 patches are randomly sampled from the WSI. We follow the idea
in (Xu & Chen 2023) and split the WSI bag into sub-bags, each bag has 512 instances. We use Adam (Kingma & Ba 2014) as the optimizer with the learning rate of 5 x 10 4 All the networks including compared methods are trained for 30 epochs and the batch size is set to 32. We report the 5-fold averaged C-index on validation sets.
C.2 MORE QUANTITIVE STUDIES
In this part, we reveal more experimental results about the parameter settings on three cancer datasets, shown in Figure 5 We adopt careful approach to address the potential risk of selection bias in hyperparameter choices. To begin with, we defined the hyperparameter search space based on the commonly used ranges in information bottleneck methods like VIB (Alemi et al. 2016) and MIB (Federici et al., 2020). During the selection, we implement a grid search strategy. For each hyperparameter, while keeping others fixed, we conducted a five-fold cross-validation, selecting the hyperparameters that exhibited the best average performance across all validation sets. Furthermore, to further mitigate selection bias, these hyperparameters were selected on multiple datasets.
Settings of Weight Factor for Loss Function. We conduct quantitive studies about the weight factor of loss items in Eq. [3), shown in Figure (a)-(d) Among these parameters, a, 3, 7 are
httos://portal.cdd.com cod..cerr.
https::/xeenbbroser..eeee. het.:////eeeer.tttteaaaaaaagg//
19

---

Published as a conference paper at ICLR 2024
Parameter Setting Weight factor arn Ayr
Parameter Setting Weight Fector 2 Last
Parometer Setting of Weight facer y Gas
775
0.7
5TAD
HRSC
1.24
PRSC:
STAD
STAD
H966
COADHE40
0.14
CONDREAD
4.2
3.40
3.4
150
Agha
Beta
Gamma
(a)
(b)
(C)
Parymater Setting Weight factor in Egg
Parameter Sctt ig of Irr to Hasslogy
Peroncter Setting frr for Genomies
0.75
0.75
3.5
COADRE4D
0.50
0.50
CORDREAD
0.4
0.75
0.5
6:50
0.00
3.70
0.71
0.79
1.50
0.55
0450
1.4
0:25
04.0
0.00
0.30
20
03
03
61
6.
6.
180
20
(44
(d)
(e)
(1)
Figure 5: The effect of different parameter settings. We conduct a quantitative study on the weight factors of loss items (d) and the information retention rate of PIB (c)-(f).
for CPIB. and À is for LPID- In LPIB. higher implics that more label-related information is retained, a higher 3 indicates more information is compressed original inputs, and a higher ? means larger constraint as these parameters increase, LPID effectively removes redundant information by modeling better prototypes for various risk bands, thus enhancing the model's performance. However, when these parameters are further increased, excessive information compression within modalities may discard useful knowledge. Especially, when y becomes larger, the over-separability of the prototypes may reduce the generalization ability for samples deviating from the prototype's central. Then in LPTD, the increase exhibits different trends across datasets. This variation could be attributed to differ- ences in multimodal redundancy levels, which in turn affects the optimal weighting for information disentanglement. the end, we set the a, B, 7, and À to 0.1, 0.01, 1, and 0.1, respectively with better performance.
the approximation of the prototypes distributions. It can be seen that
Settings of Redundancy Removal in PIB. Irr controls the proportion of redundancy removal of uni- mni-modalities by prototypes in PIB. A lower Irr indicates that more irrelevant instances dropped by prototypes. We can see from Figure (e))(() when Irr gradually decreases from 100% (i.e., retaining all instances), the model's performance improves, suggesting that removing intra- modal redundancy can effectively extract discriminative information. On the contrary, when Irr is too low, the model's performance deteriorates, which could indicate that task-related instances are also being discarded, resulting in the loss of valuable information. For histological data, we achieve comparable performance using only approximately 25% to 40% of the instances compared to utilizing the entire bag, resulting in reduction in data usage of approximately 60% to 75%. Similarly, for genomic data, performance remains equivalent when utilizing approximately 55% to 70% of the dataset compared to employing all the pathways. In the end, here we set the value of Irr for the pathology and genomics modalities to 50% and 80%, respectively. This allows PIB to remove redundancy while retaining effective instances, ultimately improving the model's performance.
Settings of Sampling. In our work, we adopt the commonly used Monte Carlo sampling method following the information bottleneck-based approaches (Alemi et al.) 2016 Federici et al. 2020 Lee & Van der Schaar 2021). leveraging the reparameterization trick and randomly sampling from standard Gaussian distribution. Moreover, the choice of the number of samples affects the models' performances. Therefore, we gradually increase the number of samples from a small to a larger number, as illustrated in the table below. The experiments demonstrate that the number increases,
20

---

Published as a conference paper at ICLR 2024
Table 4: Settings of sample number. C-index (mean + std).
Sample Number
BLCA
COADREAD
STAD
0.614 + 0.032
0.763 = 0.129
0.634 + 0.059
10
0.659 + 0.074
0.804 = 0.093
0.673 0.082
0.667 + 0.061
0.768 = 0.124
0.684 0.035
100
0.689 + 0.064
0.811 0.128
0.678 0.060
the model's performance gradually improves. Increasing the number of samples effectively aids in learning prototypes. In the paper, we strive balance performance and sampling complexity and thus set the sample number uniformly to 50.
C.3 MORE COMPARISONS WITH STATE-OF-TIIB-ARTS
To mitigate the impact potential biases that emerge from data contamination of self-supervised models pretrained on TCGA (Guo al.) 2023 Jacovi et al. 2023). we also conduct additional experiments with a ResNet50 (He et al.) 2016) encoder pretrained only on ImageNet (Deng et al.) 2009) for histological modality. Here select the top perfoorer methods in the unimodal group and the multimodal group in Table respectively. The results are shown in Table [5] It is clearly demonstrated that even when employing great improvement compared to the comparison methods, which further underscores the superiority of our method.
VVCGA-pretrained encoder, PIBD exhibits
Table 5: C-index (mean + std) over five cancer datasets by using ResNet50 encoder (ImagaeNet transfer). g. and h. refer to genomic modality and histological modality, respectively. The best results and the second-bes best results are highlighted in bold and in underline. A method marked with the subscript falls into the unimodal group, into the multimodal group
BRCA
BLCA
COADREAD
HNSC
STAD
Model
Modality
Overall
(N=869)
(N=359)
(N=296)
(N=392)
(N=317)
TransMIL
h.
0.617 + 0.127
0.588 + 0.072
0.699 + 0.170
0.633
0.023
0.592 = 0.095
0.626
CLAM-MB
h.
0.626 0.109
0.584
0.058
+ 0.170
0.606
0.031
+:
0.603
MOTCat
g.+h.
0.639 + 0.090
0.591 + 0.028
0.705 + 0.168
0.589
0.067
0.598 + 0.113
0.624
SurvPath
g.+h.
0.685 +: 0.067
0.570 +: 0.052
0.749 +: 0.100
0.640
0.059
0.624 + 0.057
0.653
PIBD
g.+h.
0.697 0.092
0.595 0.061
0.784 + 0.124
0.637 0.047
0.644
0.063
0.671
D VISUALIZATION OF SIMILARITY SCORES
To show the interpretability, have visualized similarity scores of instances each prototype of risk level and presented top-3 relevant patches and biological pathways, as well as top-3 irrelevant biological pathways for the local interpretability of individual patients. Results from high-risk case and one low-risk case are shown in Figure and Figure 7 respectively. The visualizations clearly demonstrate that prototypes representing different risk intervals focus on distinct regions within the WSIs, enabling the extraction of discriminative instances. Conversely, instances with lower attention are considered redundant and discarded. This further elucidates how the model ben- eñits from the redundancy removal mechanism introduced in our proposed PIB. Through analyzing these instances, we may discover biomarkers of various risk levels. This is because prototypes are designed to be discriminative, which enforces selecting instances that are most distinctive for each risk level, potentially corresponding to biomarkers for different risk levels.
21

---

Published as a conference paper at ICLR 2024
High
Similarity Scores of Histology
Low
Slide Image
Prototype (F-0,c00)
Prototype (t=1,e=0)
TCGA-AR-AIAR
Survival Time: 17.47 months
Label: t=0,em0
High Risk
Prototype (1-2,e-0)
Prototype (I-3,e-0)
Prototype (t-0,e-1)
Netiin
Prototype (F1.e=1)
Prototype (+2,c=1)
Prototype (t=3,e-))
Figure 6: Visualizations of similarity scores for high-risk case in BRCA, along with the patches displaying the highest similarities and the biological pathways showing both the lowest and highest similarities.
22

---

Published as a conference paper at ICLR 2024
High
Similarity Scores Histology
Low
Whole Slide Image
Prototype (t=0,e=0)
Prototype (t-1,e-0)
TCGA-CQ-6223
Survival Time: 47.60 months
Label: 1=3,c-1
Low Risk
Prototype (F=2,=0)
Prototype (t=,,==)
Prototype (t=0,e=1)
Prototype (F1.e=1)
Prototype ((-2,c-1)
Prototype (t=3,e-1)
Figure 7: Visualizations of similarity scores for low-risk case in HNSC, along with the patches displaying the highest similarities and the biological pathways showing both the lowest and highest similarities.
23


Reviews:
[{"review_id": "JC1CbRQgVs", "reviewer": "Reviewer_bKh7", "summary": "This paper introduces a novel approach to predicting survival outcome by combining histology and genomics data. The model incorporates two novel components based on information bottlenecks in information theory: prototypical information bottleneck and prototypical information disentanglement.", "questions": "Kaplan-Meier analysis: please state how cut-offs were selected in the manuscript. \n\nPlease see comment about combining individual modality predictions using CoxPH as a “naive” baseline. \n\nThe appendix gives some discussion of hyperparameter choices, although it is not clear how these were selected with the dataset. As cross validation is used to generate the evaluation results, to what extent were the hyperparameters selected using the evaluation datasets and is there risk of selection bias as a result?", "limitations": "", "rating": 10, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "The problem is well defined and the introduced method is appropriate with detailed explanation and evaluation. Results represent a meaningful improvement on prior state of the art approaches. The analysis is thorough and builds upon established practices in the literature. Although the advances over the state of the art are modest, it does represent a consistent improvement compared to other methods across all datasets.", "weaknesses": "There are very few weaknesses with this paper. \n\nThe evaluation could be improved by adding a “naive” combination of the risk scores from genomics and histology. This could be performed by taking the best performing models which look at an individual modality – i.e. SNNTrans and CLAM-MB – and then combining the predictions of these models using a Cox proportional hazards model against the original survival outcome data. This would act as a suitable baseline to see to what extent learning the combined risk prediction model as described in this work outperforms a naive approach to combining the risk prediction across the two modalities. This would allow the reader to fully understand the added benefit of the model described over the other approaches.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WfCmCSanuI", "reviewer": "Reviewer_HNZs", "summary": "The authors introduce a novel framework, Prototypical Information Bottlenecking and Disentangling (PIBD), comprising a Prototypical Information Bottleneck (PIB) module for handling intra-modal redundancy and a Prototypical Information Disentanglement (PID) module for addressing inter-modal redundancy. Extensive experiments conducted on five cancer benchmark datasets establish their method's superiority over other approaches.", "questions": "(1)What is the significance of the red arrows in Figure 1, particularly within the context of the PID module?\n(2)How does PIB save the computation? Please make quantitative analysis.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "(1) The concept of Information Bottleneck (IB) is intriguing, as it offers a promising solution to eliminate unnecessary redundancy, and the PIB module effectively addresses computational challenges.\n(2) The technical details are presented with clarity and precision.\n(3) The thoroughness of the experiments conducted to validate the effectiveness of each component is commendable.\n(4) The quality of the visual presentations is noteworthy.", "weaknesses": "(1) The Prototypical Information Disentanglement (PID) module could benefit from improved clarity in its description.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "riO3ETnVCk", "reviewer": "Reviewer_ottr", "summary": "This work presents PIBD (Prototypical Information Bottlenecking and Disentangling) for multimodal survival analysis using pathology and genomics, which extends previous progress thus far on co-attention-based early-based fusion [1] and learning information bottleneck [2] in computational pathology via two mechanisms: 1) a Prototypical Information Bottleneck (PIB) module for intra-modal redundancy and 2) a Prototypical Information Disentanglement (PID) module for inter-modal redundancy. Experimentation is performed on the same splits for disease-specific survival analysis in Jaume et al. [1], with comparisons against other relevant works (both unimodal and multimodal) and ablation experiments that assess PIB and PID independently.\n\n\nReferences\n1. Jaume, G., Vaidya, A., Chen, R., Williamson, D., Liang, P. and Mahmood, F., 2023. Modeling Dense Multimodal Interactions Between Biological Pathways and Histology for Survival Prediction. arXiv preprint arXiv:2304.06819.\n2. Li, H., Zhu, C., Zhang, Y., Sun, Y., Shui, Z., Kuang, W., Zheng, S. and Yang, L., 2023. Task-specific fine-tuning via variational information bottleneck for weakly-supervised pathology whole slide image classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 7454-7463).", "questions": "Primary Questions and Suggestions\n- Current application and findings of PIBD overlap with other multimodal survival analysis works wr.t. to addressing patch feature redundancy, and having similar interpretability experiments. How does the contributions and findings of this work advance the field further?\n- Ablation experiments for PID/PIB can have stronger baselines.\n- Evaluating PIBD with a ResNet-50 encoder (ImageNet transfer) or a non-TCGA-pretrained encoder would strength the study and its findings.\n\nMinor Questions\n- How are CLAM-SB / CLAM-MB adapted for survival analysis? To the reviewer's knowledge, the implementation of the CLAM framework is mostly situated for slide classification (not survival), as the clustering constraints in CLAM are most appropriate for subtyping problems.", "limitations": "", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- As summarized above, extensive experimentation regarding comparisons, ablation studies, and assessment of multiple cancer types is performed. In particular, this work compares with many state-of-the-art multimodal methods such as MCAT (both concatenation and Kronecker Product), SurvPath, MOTCat, and other relevant information bottleneck-related works. Beyond ABMIL and TransMIL, other strong unimodal methods such as the task-specific finetuning variant of CLAM is also considered. Background information and literature is also extensive, with many important studies related to PIBD cited in a comprehensive manner.\n- The proposed PID and PIB methodology, though largely adapted and inspired by other IB applications in computational pathology such as that of Li et al. 2023, is (on balance) appropriate for its extension into survival analysis w.r.t. addressing sparsity of patch features in MIL via prototypes. Figure 1 is informative and communciates how PID/PIB are used.", "weaknesses": "- **Limited findings w.r.t. clinical application / multimodal interpretability**: PID/PIB is intuitive for solving multimodal integration problems of WSIs and gene sets, but the demonstrated findings of this work appears limited to only improvement of c-Index performance. Methodologies such as MOTCat and SurvPath, though not approaching survival analysis from an information bottleneck perspective, similarly resolve issues regarding redundancy of patch features in MIL and reach similar findings already established in prior works. The interpretability experiments, shown as attention heatmaps in the supplement, are difficult to interpret and draw conclusions from. As the improvement in c-Index is minor, it would be valuable to investigate the interpretability of PIBD, for example: (1) performing local and global interpretability to assess individual and cohort-level image-omic drivers of disease survival, or (2) experiment+showcase unique clinical applications that would arise from PIBD that cannot be performed in other methods, which would produce new scientific findings and expand this study's significance. In particular, I found the prototypical aspect of this work to be intriguing, with concurrent works also realizing (and finding new applications) of using prototypical patterns for pathology [5].\n- **Ablation Experiments for PIB/PID**: Some baselines do not seem fair in the PID/PIB ablation experiments.\n- - In ablating PID, the baseline model considered was average pooling the prototypical features - a more appropriate baseline would be to use a non-disentangled Transformer like TransMIL on top of the prototypical features. \n- - In ablating PIB, the reviewer is not certain what the baseline looks like (using PID without PIB), but would think that this comparison would be similar to having each modality being processed by a TransMIL-like encoder + PID (or SurvPath + PID). \n- **Using CTransPath for TCGA evaluation**: A limitation of this study is the usage of CTransPath (pretrained on TCGA) for training and evaluating MIL models for survival analysis in TCGA. Understandably, access to powerful pretrained encoders for pathology is challenging, due to: (1) lack of computing power, and (2) lack of diverse and independent data outside of TCGA, with other studies in computational pathology such as Jaume et al. [1] and Filliot et al. [2] having also conducted survival analyses via TCGA-pretrained encoders. This limitation could remain as \"a limitation of this study\" (fitted in the conclusion) as the novelties of this work are with respect to the MIL encoder (all comparisons are performed using the same features and evaluated in the same way), in combination with the aforementioned challenges in computational pathology. However, as the broader pathology and machine learning community is becoming more aware of potential biases that emerge from data contamination of self-supervised models that are also trained using the evaluation data (as seen in LLMs and other studies [3-7]), it is important for advances going forward to recognize and resolve this issue and to develop better standardization for developing and evaluating MIL methods. \n\nReferences\n1. Jaume, G., Vaidya, A., Chen, R., Williamson, D., Liang, P. and Mahmood, F., 2023. Modeling Dense Multimodal Interactions Between Biological Pathways and Histology for Survival Prediction. arXiv preprint arXiv:2304.06819.\n2. Filiot, A., Ghermi, R., Olivier, A., Jacob, P., Fidon, L., Mac Kain, A., Saillard, C. and Schiratti, J.B., 2023. Scaling Self-Supervised Learning for Histopathology with Masked Image Modeling. medRxiv, pp.2023-07.\n3. Guo, C., Bordes, F., Vincent, P. and Chaudhuri, K., 2023. Do SSL Models Have D\\'ej\\a Vu? A Case of Unintended Memorization in Self-supervised Learning. arXiv preprint arXiv:2304.13850.\n4. Xiang, J. and Zhang, J., 2022, September. Exploring low-rank property in multiple instance learning for whole slide image classification. In The Eleventh International Conference on Learning Representations.\n5. Chen, R.J., Ding, T., Lu, M.Y., Williamson, D.F., Jaume, G., Chen, B., Zhang, A., Shao, D., Song, A.H., Shaban, M. and Williams, M., 2023. A General-Purpose Self-Supervised Model for Computational Pathology. arXiv preprint arXiv:2308.15474.\n6. Jacovi, A., Caciularu, A., Goldman, O. and Goldberg, Y., 2023. Stop uploading test data in plain text: Practical strategies for mitigating data contamination by evaluation benchmarks. arXiv preprint arXiv:2305.10160.\n7. Kapoor, S. and Narayanan, A., 2023. Leakage and the reproducibility crisis in machine-learning-based science. Patterns, 4(9).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "I1AW1ILGf0", "reviewer": "Reviewer_G86X", "summary": "This work introduces two different information-theory-based modules for multimodal (histology and genomics) survival prediction problem. Based on information bottleneck theory, these modules aim to remove redundant information, which are prevalent in WSIs, within each modality and across different modalities. In the process, the concept of prototypes are introduced to guide the learning of the multimodal prognosis framework, while only retaining necessary information.", "questions": "- Motivation leading up to Eq. (5) is confusing. Why is large bag size an issue, if you can learn parametric mapping (via neural network) between z and x, similar to Eq. (4)? Wouldn't this simply be a matter of backpropagation in the training, which is scalable to numerous instances?\n- Variational approaches involving NNs are known to be prone to mode collapse - What are the tricks/methods used in the work to prevent it?\n- I think the introduction of Markov chains confuses the paper more - Perhaps better to remove it.\n- Eq. (6) - Can't we simply use KL divergence to assess the similiarity (distance), if both p(z|x) and p(\\hat{z}|y) are normal distributions?\n- Usually the use of \"Top-k\" refers to small number of instances being selected. However, from implementation section, it seems the authors are using 50~80% of the entire bag? Am I misunderstanding something?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- While the use of information bottleneck theory is not new in computational pathology, the use of it for 1) survival prediction setting and 2) reduction of intra-modal redundant information is quite novel.\n- The authors conduct extensive baseline comparisons to demonstrate that PIBD framework indeed is better/highly-competitive compared to even the most recent baselines (MOTCat and SurvPath). This is to be appreciated by any fellow researchers looking to get understanding of the available survival prediction methods in CPath.\n- Probabilistic modeling of the bag of patch features (although it leads to several variational approximations), opens up new avenues for probabilistic approaches for CPath, which have been under-explored thus far.", "weaknesses": "Clarity\n- The explanation around PIB and PID need to be much more straightforward for the work to be appreciated. In its current form, it is really hard to understand and Section 3 lacks clarity. The authors should understand that typical readership of this article will either be pathologist or ML practitioner who would be less-versed in probabilistic and variational language. This, in my opinion, significantly reduces the impact of the paper as any other researcher hoping to reuse the framework for their own experiments. Authors should try to inject more  pathology-related intuition throughout the work.\n\nDiscriminative assumption\n-  I am not sure if \"discriminative\" approach (despite its good performance as shown in the paper) is the right approach for survival. Different from diagnosis problems where subtype/grades are indeed decided with discriminative instances (patches), survival prediction problems are not as clear-cut. Certain morphological characteristics will overlap between different risk groups (or bands), which the current approach, centered around sampling positive/negative prototypes for maximum discrimination, is not well suited for.\n- Related to the comment above, this approach seems only applicable to NLL loss, which treats the problem as a classification problem and thus have \"different\" bins (i.e., discretized timeline). What if more common loss functions, such as Cox PH or rank losses, were to be used that do not rely on such discriminative hypothesis?\n\nKaplan-Meier analysis\n- In Kaplan-Meier analysis section, authors state that \"our approach demonstrates significantly improved discrimination.... as indicated by lower p-values\". While it is tempting to compare between p-values as a metric for separation of the risk groups, it is statistically not correct. Once these are below significance threshold, it shouldn't be over-interpreted. Maybe, use median survival days for each risk group?\n\nSampling operation\n- Since both PID and PIB rely on sampling from distribution, it does seem that the performance will indeed by affected by which samples are chosen or how many of them are sampled. The discussion around this point needs to be made explicit.\n\nInference\n- While there are detailed information about training procedure, not much is written about the actual inference step. For instance, how many samples for each prototype are required for reliable performance?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)