Interpretable Vision-Language Survival Analysis with Ordinal Inductive Bias for Computational Pathology
Authors: Pei Liu
Luping Ji
Jiaxiang Gou
Bo Fu
Mao Ye
Venue: ICLR 2025 Poster (2025)
Keywords: Computation Pathology
Survival Analysis
Multi-Instance Learning
Whole-Slide Images
Vision-Language Modes
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: This paper first introduces Vision-Language Survival Analysis (VLSA) paradigm for computational pathology.

Abstract: Histopathology Whole-Slide Images (WSIs) provide an important tool to assess cancer prognosis in computational pathology (CPATH). While existing survival analysis (SA) approaches have made exciting progress, they are generally limited to adopting highly-expressive network architectures and only coarse-grained patient-level labels to learn visual prognostic representations from gigapixel WSIs. Such learning paradigm suffers from critical performance bottlenecks, when facing present scarce training data and standard multi-instance learning (MIL) framework in CPATH. To overcome it, this paper, for the first time, proposes a new Vision-Language-based SA (**VLSA**) paradigm. Concretely, (1) VLSA is driven by pathology VL foundation models. It no longer relies on high-capability networks and shows the advantage of *data efficiency*. (2) In vision-end, VLSA encodes textual prognostic prior and then employs it as *auxiliary signals* to guide the aggregating of visual prognostic features at instance level, thereby compensating for the weak supervision in MIL. Moreover, given the characteristics of SA, we propose i) *ordinal survival prompt learning* to transform continuous survival labels into textual prompts; and ii) *ordinal incidence function* as prediction target to make SA compatible with VL-based prediction. Notably, VLSA's predictions can be interpreted intuitively by our Shapley values-based method. The extensive experiments on five datasets confirm the effectiveness of our scheme. Our VLSA could pave a new way for SA in CPATH by offering weakly-supervised MIL an effective means to learn valuable prognostic clues from gigapixel WSIs. Our source code is available at https://github.com/liupei101/VLSA.

Published as a conference paper at ICLR 2025
INTERPRETABLE VISION-LANGUAGE ANALYSIS WITH ORDINAL INDUCTIVE BIAS FOR COMPUTATIONAL PATHOLOGY
SURVIVAL
Pei Liu, Luping Ji Jiaxiang Gou, Bo Fu, Mao Ye School of Computer Science and Engineering, University of Electronic Science and Technology of China
ABSTRACT
Histopathology Whole-Slide Images (WSIs) provide an important tool to as- sess cancer prognosis in computational pathology (CPATH). While existing sur- vival analysis (SA) approaches have made exciting progress, they are gener- ally limited to adopting highly-expressive network architectures and only coarse- grained patient-level labels to learn visual prognostic representations from gi- gapixel WSIs. Such learning paradigm suffers from critical performance bot- tlenecks, when facing present scarce training data and standard multi-instance learning (MIL) framework in CPATH. To overcome it, this paper, for the first time, proposes
Vision-Language- based SA (VLSA) paradigm. Concretely, (1) VLSA is driven by pathology VL foundation models. It no longer relies high-capability networks and shows the advantage of data efficiency. (2) In vision-end, VLSA encodes textual prognostic prior and then employs it as auxiliary signals to guide the aggregating of visual prognostic features in- stance level, thereby compensating for the weak supervision in MIL. Moreover, given the characteristics of SA, we propose i) ordinal survival prompt learn- ing transform continuous survival labels into textual prompts; and ii) ordinal incidence function as prediction target to make SA compatible with VL-based prediction. Notably, VLSA's predictions can be interpreted intuitively by our Shapley values-based method. The extensive experiments on five datasets con- firm the effectiveness of our scheme. Our VLSA could pave SA in CPATH by offering weakly-supervised MII. an effective means to learn valuable prognostic clues from gigapixel WSIs. Our source code is available https: //g/ tuuuthhubb... wtwmmiipeei10///SS...
way for
INTRODUCTION
Histopathology whole-slide image (WSI) plays a vital role in cancer diagnosis and treatment (Zarella al., 2018). It usually covers rich and holistic microscopic information from cellular morphology to tumor micro-environment and then to tissue phenotype (Pati et al., 2022; Chen et al., 2022). As this information can directly reflect tumor progression, digital WSIs are often used in computational pathology (CPATH) to assess cancer patients' prognosis (or survival) (Kather et al., 2019; Song et al., 2023; Jaume et al., 2024). An accurate prognosis assessment of great significance for enhancing patient management and disease outcomes (Skrede et al., 2020).
However, the survival analysis (SA) of WSI data has always faced two critical challenges. While existing approaches have made exciting progress in overcoming these challenges, they still suffer from performunce bottlenecks due to the inherent limitations current SA paradigm. (1) Searce training data. Owing to many real-world factors, e.g., the difficulties in long-term patient follow-up and the concerns about patient privacy, the scale of WSI data for SA has always been limited (Lu et al., 2022; Liu et al., 2024a; Song et al., 2024a), typically the order of 1,000. Most existing SA models often overlook this and generally seek for network-level solutions to improve the per- formance, such adopting highly-expressive modern networks like GNNs (Chen et al., 2021a; Liu
Corresponding author: Luping Ji celuping@uestc.eduu.

---

Published as a conference paper at ICLR 2025
et al., 2023b; Shao et al., 2024; Wang et al., 2024) or Transformers (Huang et al., 2021; Hou al., 2022: Liu et al., 2023a). However, when facing present small WSI data, this way is more likely to cause overfitting in deep learning models (Srivastava et al., 2014), leading to suboptimal prediction performance. (2) Learning from gigapixel images under weak supervision. Digital WSIs have extremely-high resolution, e.g., 40,000x 40,000 pixels, so each one is usually processed into bag of multi-instances for training. With only WSI-level labels, bag-level representations are derived via defacto weakly-supervised multi-instance learning (MIL) framework (Ilse et al., 2018; Liu al., 2024b). It first i) learns task-specific embeddings from single instances and ii) then aggregates nu- merous instances (typically 10,000) into one single vector. Current SA schemes follow this way, yet the whole learning process is driven entirely by patient-level labels. We argue that such paradigm could lead to inefficient representations, since SA models are only provided with overall patient- level labels while they are not only required to i) learn prognostic embeddings at a fine-grained instance level; but also to ii) select key instances from numerous candidates (Li et al., 2023).
To overcome the limitations of current practices, this paper studies new SA paradigm for CPATH, called Vision-Language Survival Analysis (VLSA). Concretely, we find that recent VL foundation models (VLMs) in CPATH, e.g., CONCH (Lu et al., 2024), offer potential means mitigate the challenges above. First, these VLMs are pretrained on large-scale image-text pairs by task-agnostic objectives. They show surprisingly-gode performance in terms of data efficiency, especially in zero- shot transfer, as highlighted in Lu et al. (2024); Javed et al. (2024). This is promising for alleviating the challenge of scarce training data. Second, VL contrastive pretraining aligns image and language in latent embedding space (Radford et al.. 2021), which enables language to act as "prompt" for vision tasks. This implies that, with prior knowledge, language is likely to provide auxiliary signals to boost learning efficiency. Such additional signals could be particularly helpful to improve the weak supervision in MIL. Despite all these appealing merits, VLM-driven SA has not yet been studied. We believe there are two main reasons: i) the powerful VLMs for CPATH are developed just recently; ii) different from classification, there remains a gap for SA to be adapted to VLMs.
Based on these insights, this paper first proposes a VL.A framework for CPATH. Different from existing VL-based schemes, VLSA comprises four core designs as follows. (1) Vision-end:
leverages language-encoded prognostic priors to guide multi-instance aggregation, producing multi- level visual presentations.
(2) Language-end: considering the intrinsic ordinality in survival risks, propose ordinal survival prompt learning to encode continuous time-to-event labels into textual prompts. (3) Prediction and optimization: To make SA compatible with VL-based prediction, we take incidence function as prediction target and introduce an ordinal inductive bias into it for regular- ization in optimization.
(4) Prediction interpretation: with the classic Shapley values from game theory, individual prognostic risk could be interpreted from an intuitive pesseetive-deecrriiive language. The extensive experiments on five datasets verify the effectiveness of our scheme. No- tably, our comparative experiments and analyses suggest that VLSA could pave a new way for SA in CPATH by offering weakly-supervisee MIL an effective means to learn valuable prognostic clues from gigapixel WSIs. We summarize the main contributions of this paper as follows:
An interpretable Vision-Language Survival Analysis framework (VLSA) is proposed for compu- tational pathology. To our knowledge, it is the first work that studies VI.-based SA in CPATH.
Language-encoded prognostic prior is proposed to boost learning. In view of the characteristics of SA, two ordinal inductive bias terms, i.e., ordinal survival prompts and ordinal incidence function, are introduced into VLSA to enhance the performance.
WSI representation
To assess model performance more rigorously, this paper conducts extensive experiments and adopts multiple metrics for discrimination and calibration evaluation. Empirical results show that VLSA could often obtain new ofate-of-the-art performances with less computation costs.
2 RELATED WORK
Survival Analysis on WSIs One key challenge of WSI survival analysis is effectively learning global prognostic representations from multi-instances with only patient-level survival labels. To this end, various data-driven approaches gories based on the structure assumed for instances: i) cluster-based (Yao et al., 2020; Shao et al., 2021a), la), ii) graph-based (Chen et al., 2021a; Liu et al., 2023b; Shao et al., 2024; Wang al., 2024),
studied. They can be roughly grouped into three cate-
2

---

Published as a conference paper at ICLR 2025
and iii) sequence-based (Huang et al., 2021; Hou et al., 2022; Liu et al., 2023a). These structures are employed to learn non-local prognostic embeddings at cluster, node, or global token level. How- ever, most studies in this field currently focus learning (Zhou & Chen, 2023; Jaume et al., 2024). The vision-language learning paradigm, which has witnessed remarkable successes in recent years (Zhang al., 2024a), remains under-studied.
pure vision vision-gent representation
Vision-Language Models for Computation Pathology Since the inception of CLIP (Radford et al., 2021), Vision-Language models (VLMs) have attracted considerable attention and applied wide range of fields. CPATH is one of them. Related studies or applications cover two main direc- tions. (1) Foundational VLMs for pathology, e.g., PLIP (Huang et al., 2023). QUILTNET (Ike- zogwo et al., 2023), PathAlign (Ahmed et al., 2024), and CONCH (Lu et al., 2024). These models are usually pretrained on large-scale pathology image-text pairs by VI. contrastive learning. Their pretrained encoders often perform significantly better than CLIP in pathology-relatee tasks, laying a solid foundation for CPATH. (2) VI.-based WSI classification. Recent efforts involving VLMMs mainly seen in WSI classification (Qu et al., 2023; Li et al., 2024; Shi et al., 2024). Based on pretrained VL encoders, they further improve the performance by large margins through fine- grained text guidance multi-scale image features. Given the good foundation and strong potential of pathology VLMs, a further study on VL-based survival analysis sss is strongly anticipated.
(b)
Survival prompey
promot
Yeett Moeti
40
B
imaage
AsIL
B
Class prompt
query
interr
Incidence function
Xo-ax
What roaal baauna Will
rilaied LUAD progaso??
x
x
x
Objective
A Progrostic viuaa featurea sunce booder apportatio..
Exeri
min Caax
Leoos
event timw 220 montha
Survival label
Figure 1: Overview of Vision-Language Survival Analysis (VLSA). (a) WSI representation learn- ing with language-encoded prognostic priors (Section 3.1). (b) Ordinal survival prompt learning
(Section 3.2). (c) Prediction of ordinal incidence function (Section 3.3). The survival prediction of VLSA can be interpreted by quantifying each prognostic prior's contribution to risk (Section 3.5).
METHOD
This section presents our VLSA, a vision-Language-based survival analysis framework (Figure 1) for CPATH. We first introduce its three key parts: i) WSI representation learning with language- encoded prognostic priors (Figure ((a), Section 3.1); ii) Ordinal survival prompt learning (Figure 1(b), Section 3.2); and iii) ordinal incidence function prediction (Figure 1(c), Section 3.3). Then, we give our overall training objectives Section 3.4 and explain how interpret the survival prediction of VL.SA in Section 3.5. The preliminaries can be found in Appendix A.
3.1 WSI REPRESENTATINN LEARNING WITH LANGUAGE- PROGNOSTIC PRIORS
Given the WSIs of one patient, we denote their processed instances by a bag X ...... xx RKDD, where 23 E D represents the k-th instance feature vector. Instance features are extracted by the image encoder of VLM, written Eimage- To derive WSI representations, language-encoded prognostic priors to encourage MIL models to distill valuable prognostic clues from numerous instances under weak supervision, shown in Figure ((a).
introduce
Language-Eneode Prognostic Priors Concretely, for a specific cancer disease, we first obtain the textual descriptions about its critical prognostic features visible in histopathology WSIs. written as Tprog = (T T }, using a LLM (large language model), GPT-4o. Then, we encode Tpog into textual features with the text encoder of the VLMM, denoted by Freu- We further fine-tune these
3

---

Published as a conference paper at ICLR 2025
textual features using group of learnable parameters, T prog E ERMxD We write the result as P = Eiex Eiex: (Tprog + Tpog
(I)
P = [p...... PM] E RMDD represents language-eneoder prognostic priors, where pm E R is the m-th prior item and M is the total number of prior items.
WSI Representation
the MIL aggregator AMm., any prior pm is utilized as a query to aggregate key instances by similarity matching and weighted averaging, producing one-level WSI representa- tion (fm E P) corresponding to the m-th prognostic description (Trig). Finally, multi-level visual representations, F = ff.....ffe fM] E IR M are averaged and passed through a linear neural network layer to obtain the final WSI representation Simuge E RD, In summary, there are
K
exp (a cos cos(pm; xxa)
fm
TK
K ΣE exp (a cos(pm: 2::))
(2)
k=
finage = Linear (mean( F)) = Linear Linear(mean(( (f fm m=m)))
(3)
where a is a fixed temperature nyper-parameter cos(,,) is an operator that calculates cosine simi- larity, and Linear(-) indicates a simple linear layer with learnable weights and biases.
Justification (1) Prognostic prior knowledge: from Eq. (2), can find that, the instances whose visual features are well-aligned with the prognostic description, would be selected and aggregated into one visual representation. This implies that, with the well-aligned VL embedding space pro- vided by VLMs, prognostic prior could offer additional helpful signals and compensate for the inher- ent weak supervision in MII.. (2) Multi-level visual representations: they are designed to provide sufficient prognostic clues because there are usually more than one prognostic markers in WSIs, e.g. nuclear atypia, perineural invasion, mitotic activity, etc. Moreover, multi-level prognostic clues could enable us decouple and interpret individual survival predictions from intuitive perspectives (i.e., descriptive language). Our interpretation method is elaborated in Section 3.5. (3) Difference from multi-modal representation learning: textual description of prognostic prior is utilized to provide the weight of each instance for MIL aggregation and mainly plays a guidance role. The final representation in vision-end comes from WSI features, not multi-modal representation fusing text and vision features. This is different from current multi-modal SA models (Chen et al., 2021b; Xu & Chen, 2023; Zhou & Chen, 2023; Xiong al., 2024; Song et al., 2024b).
3.2 ORDINAL SURVIVAL PROMPT LEARNING
Different from classification, SA provides continuous time-to-evee eventt labels, i..., = 0t. where t the last follow-up time and 8 E {0, 1} is an event indicator Due this intrinsic difference, obtaining SA prompts would be more challenging. compared with VL-based classification. To ad- dress it, we propose ordinal survival prompt learning that encodes time-to-event labels into textual survival prompts for VL-based prediction, as depicted in Figure (bb).
Time Discretization Following the convention of discrete- time SA (Haider et al., 2020), we first uniformly discretize time into set of non-overlapping bins, [To, Ti), [Ti, ... TTT Tc). For any E [T--1, 1, Te), we assign its corresponding time-discrete label c, where e E [1,C]. These time bins are equal in length; C is determined by VN., where N₂ the number of patients with 8 1. Refer to Appendix C.5 for more details and Appendix D.6 for experiments on these settings.
Ordinal Survival Prompt Textual prompt usually consists of context and class label in VLM- driven prediction. Thus, (1) for context prompt, we follow CoOp (Zhou et al., 2022) to employ learnable parameters, Vax E IR Ro. Damn, to optimize the context of survival prompts, where Lax is the length of context tokens and Demb is the dimension of token embedding. (2) For class prompts, maintain B learnable base class prompts, {Vr22 ..... VA ). where V E h [L, B)) is the token embedding of the b-th base class prompt and Lets is the maximum length of class tokens. These base class prompts
(b e
initialized using common prognosis risk descriptions, e.g., ("very poor", "moderate", "very good"). Then, considering the ordinality between survival classes (1,2, ... C). we obtain the remaining class prompts by interpolating virtual points between base class prompts, inspired by Li et al. (2022). The e-th class prompt is written as follows:
B
W(D..)
Σ b=1
(4)
cls
(D...)
1=
4

---

Published as a conference paper at ICLR 2025
where De,b is element of matrix D RC representing the ordering distance between the c-th class prompt and the b-th base class prompt, and ((.) is a function determining the interpolation weight based on distance. A closer distance usually indicates greater interpolation weight. Dr.b could be simply set by De.6 = ( - 1) (b 1) (C - () - 1)1- For linear interpolation, Da.
W(Deb) 1 c-i- (3) For survival prompts, each class prompt is concatenated with Vax the token level, followed by passing through frozen Evext to produce the final ordinal survival prompts, denoted as FixI = [ieui f Feeil" E IRCxD, Namely, there is
fiest = Exex ([Ve Val) VC [1,C].
(5)
Justification (1) Ordinal inductive bias: in common sense, as the length of survival time decreases, the corresponding death risk becomes increasingly high. This implies the ordinality between survival classes; such inductive bias could be considered when designing prompts for SA. More explanations and discussions could be found in Appendix B. (2) Class prompt interpolation: on one hand, most the sensitivity to numbers' ordinality (Thawani et al., 2021; Paiss et al., 2023), so it may be unsuitable to directly adopt numerical classes as textual prompts. On the other hand, when the number of classes becomes large, e.g., C = 10, it would be intractable to manually design fine-grained prognosis risk descriptions at C' different levels. In view of these obstacles, we adopt a few base prompts (A < 4) and then utilize an interpolation- et al., 2022) to preserve the ordinality between survival prompts.
textual encoders are lacking
strategy (Li
3.3 ORDINAL INCIDENCE FUNCTION PREDICTION
The prediction target of VL-based models is usually class probability, calculated based on the sim- ilarity measures between image features and a group of textual class prompts. To make survival prediction compatible with this way, we propose to take individual incidence function as prediction target and introduce another ordinal inductive bias term into it, as shown in Figure 1(c).
Survival Prediction with Incidence Function Similar to the manner of VL-based classification, we calculate SA prediction, denoted by ý = [,,,... .CC, as follows:
Ue exp (T cos( imige f(x))
ACE [1.C],
(6)
1= exp (r cos( fiex))
where r is a temperature parameter optimized in training. From a conventional classification per- spective, de given by Eq. (6) can be cast the probability that an event first occurs at time c; ý is thus a probability distribution on the first hitting time. Such prediction is closely associated with the concept of incidence function (IF) (Fine& & Gray, 1999) and the first hitting time model (Lee & Whit- more, 2006; Lee et al., 2018) in traditional SA. Therefore, we call Eq. (6) the prediction of individual IF and leverage IF tools for the interested task. By definition, cumulative IF is CIF(() Σi= Do-
Survival function funvitoo
(surviving past time c) can be written as S(c) = 1 CIF(c) = - Dc.
Ordinal Inductive Bias in Incidence Function As is probability distributed over ordered sur- vival classes, we consider an ordinality constraint for it (see Appendix for further elucidations). Concretely, given that is the survival class with the largest probability, we assume there is a secutive decline in probability for those classes away from c, as shown in Figure 1(c). Note that our assumption is concerned with the event that first occurs, as ÿ is defined as the probability distribution on the first hitting time. To impose the ordinality constraint on ÿ. at first we adopt Earth Mover's ' Distance (EMD) (Levina & Bickel, 2001) as the measure quantify the distance between ÿ and y (ground truth distribution), written as
EMD y) + |[CDF()) CDF(
where CDF(.) means cumulative distribution function. EMD is a measure aware of distribution ordinality as it considers the geometry property of distribution in distance measurement and it is smaller when the geometry (shape) of two distributions closer. Please refer to Appendix A.3 for detailed explanations. Furthermore, we adopt squared EMD objective (Hou et al., 2017) to regularize ý in model optimization:
LEMD = [[CDF()) CDF(y(c
(8)
5

---

Published as a conference paper at ICLR 2025
((c, 6) is defined follows. For the patient with event, y(c, 1) Softmax (2 Li=e 1))- For the patient censored at time c, we only know that its actual time-to-event is not less than c, so y(c, 5 0) Softmax (7' (2 12.- 1)). L {0, is an indicator with element '1' at index i. The value of r' is from but not involved in optimization. We further analysis and discuss the ordinal inductive biases introduced into Fiext and ÿł the details can be found in Appendix B.
3.4 OVERALL TRAINING OBJECTIVES
For any patient with survival label V {t,5), we denote its time-discrete label by V. (c, (Section 3.2). To optimize the prediction of individual incidence function, maximum likelihood estimation (MLE)-based objective (Tutz& & Schmid, 2016) is utilized in training:
MMEE -----log(
log((e) + (1 5) log(
Σ 90)
For censored patients (6 0), LMLE minimizes - Di. i.e., the probability that the event of interest first occurs at discrete bins I, 2, ..., or e 1, according to the definition of ÿ; in Eq. (6). Furthermore, with the LEMD given in Eq. (8), our overall training objective is minimizing
L
MLE + 3 LEMD.
(10)
where B 2 is hyper-parameter that modulates the weight of LEMD-
3.5 PREDICTION INTERPRETATION
Understanding how model makes predictions is crucial for reliable decision- decision-making, especially in medical domains. We thereby study the prediction behavior of VLSA and propose an interpretation method based on the classic Shapley values (Appendix A.4) from cooperative game theory.
Concretely, we interpret the survival prediction of VLSA through language-encoded prognostic priors, i.e., P (p...... PM]] I described in Section 3.1, because i) each prior intuitively describes a certain prognostic visual feature in WSIs and ii) these language-encoded priors play an important role in learning valuable prognostic clues and making final survival predictions. However, each prior item's 's contribution to risk prediction cannot be computed directly since it is not linearly correlated with ý. Thus, based on principled framework for model interpretation Shapley value (Shapley, 1953; Lundberg, 2017), we calculate the contribution of each prior item by
Z
E M!
1)!
Om (Fris) =
Σ ZCP
[risk (Z, X) - -risk (Z (Pm)x)),
(11)
where P is of prognostic priors ( PMM. ). The function fris (Z,, X) outputs the risk pre- diction given the prognostic prior subset ž and the WSI features X as inputs. Specifically, we calculate finage using subset F corresponding through Eq. (6), and iii) finally derive the risk via the summation of CIF: f Σ CIF(1).
Z, ii) use it to obtain the IF prediction ý
4 EXPERIMENTS
4.1 EXPERIMENTAL SETUP
Datasets Following Chen et al. (2021a), five publicly-available datasets from TCGA (The Cancer
Genome Atlas) are used in experiments: BLCA (bladder urothelial carcinoma) (N = 373), BRCA (breast invasive carcinoma) (N = 956), GBMLGG (glioblastoma & lower grade glioma) (N = 569), LUAD (lung adenocarcinomaa) (N = 453), and UCEC (uterine corpus endometrial carcinoma) (N =88)). These datasets cover five different cancer types, a total number of 2,831 patients, and 3,530 diagnostic gigapixel WSIs. Overall Survival (OS), which only occurs once, is set the event of interest in this paper, following Chen et al. (2021a). We adopt a standard CLAM tool (Lu et al., bags for training. CONCH (Lu al.. 2024),
2021) to preprocess all WSIs into multi-instane state-of-thh-aan VLM recently developed for CPATH. is utilized to provide powerful encoders Einage and Frest. More details datasets can be found in Appendix C.

---

Published as a conference paper at ICLR 2025
Baselines (1) Vision-only (V) methods contain ABMIL (Ilse al., 2018), TransMIL (Shao et al., 2021b), ILRA (Xiang & Zhang, 2023). R²T-MIL (Tang et al., 2024). DeepAttMMISL (Yao et al., 2020), and Patch-GCN (Chen et al., 2021a). (2) Vision-language (VL) methods are MI-Zero (Lu et al., 2023), ABMIl- Prompt: CoOp (Zhou et al., 2022), and OrdinalCIIIP (Li et al., 2022). MI-Zero is VI.-hased zero-shot approach for classification; we adapt it for SA to assess the lower bound performance of VLM on new SA tasks. ABMILprompt is another ABMIL baseline with naive (not learnable) textual prompts for prediction
language-end. As CoOp and OrdinalCLIP are not proposed for gigapixel WSIs, we adapt them for comparison: i) in vision-end, the classic attention- based MIL, i.e., ABMIL. is used for instance aggregation: ii) in language-end CoOp optimizes both context and class prompts without ordinal inductive bias, while OrdinalCLIP improves CoOp using ordinal class prompts. As a result, OrdinalCLIP and our VLSA share the approach of generating or- dinal survival prompts. Note that all methods use the same Fiinage from CONCH; VI.-based methods involve an additional VL projection layer. More details are provided in Appendix C.2 and C.3.
Evaluation Metries To evaluate models more comprehensively, we follow traditional SA (Qi et al., 2023a;b) to adopt multiple metries. (1) Concordance index (CI), as a prevalent metric in SA, as- sesses models' discrimination power. (2) Mean absolute error (MAE) gives the absolute error of model's ' prediction on timeett-event. (3) Distribution calibration (D-cal) (Haider et al., 2020) is a statistical test to evaluate models' ability in calibrating survival distribution prediction; we mainly check if there is no significant difference in distribution between ground truth and prediction (p > 0.05). For any method, we adopt 5-fold cross-validation evaluate its performance and report the average on 5 folds. Please refer to Appendix C.4 for more details on evaluation metrics.
4.2 COMPARISON WITH BASELINES
As shown in Table 1, there are three key observations. (1) Our VLSA achieves new state-of-the-ar performances and it could often perform better than most baselines by large margin. In terms of average CI, our VLSA (average CI 0.695) leads runner-up by 2.3%. Moreover, it often predicts well-calibrated survival distributions, reflected by D-cal Count. (2) For VL-based methods, CoOp and OrdinalCLIP are competitive with other baselines, although they adopt the ABMIL network much simpler than others. (3) Original CONCH encoders (with MI-Zerosur)) often cannot dis- criminate risks correctly in SA (average CI = 0.54) without supervised fine-tuning (SFT), especially on GBML.G. This suggests the necessity of adapting CONCH to SA through SFT.
Table I: Main comparative results. D-cal count the number of datasets with p > 0.05 in D-cal test. MI-Zero is adapted for SA. Best performance in bold, second best underlined.
TCGA GBMLGG
Average
D-cal
Method
BLCA
BRCA
LUAD
UCEC
CI
MAE
Count
0.5581
0.5825
0.7935
0.6121
0.6667
ABMIL
0.6426
29.83
4
( 0.031)
(= 0.035)
(# 0.032)
(+ 0.050)
(# 0.033)
0,5885
0.6140
0.7956
0.5708
0.6380
TransMIL
0.6414
30.43
5
(1 0.055)
(= 0.060)
(1 0.015)
(1 0.050)
(1 0.067)
0.5549
0.5705
0.7742
0.5179
0.6503
ILRA
0.6136
32.59
4
(# 0.053)
(= 0.067)
(= 0.014)
(# 0.081)
(+ 0.064)
RºT-MIL
0.5775
0.5473
0.7757
0.5711
0.6510
0.6245
32.54
4
( 0.024)
0.095)
(1 0.024)
(1 0.076)
(1 0.087)
0.5646
0.5346
0.6750
0.4678
0.62599
DeePAttMMSSL
0.5736
52.10
(= 0.035)
(= 0.036)
(= 0.048)
(+ 0.039)
(+ 0.085)
0.6124
0.6375
0.7999
0.5922
0.7212
Patch-GCN
0.6726
26.70
(1 0.031)
(= 0.033)
( 0.021)
(# 0.053)
(# 0.025)
0.5541
0,5788
0.3842
0.5209
0.6623
MI-Zerogire
0.5400
25.63
(= 0.034)
(= 0.028)
(# 0.063)
(+ 0.049)
(+ 0.059)
0,5717
0.6215
0.7825
0.5984
0.6762
ABMIL Prompt
0.6500
25.68
4
( 0.035)
(= 0.084)
(= 0.020)
(+ 0.052)
(+ 0.063)
0.5971
0.5994
0.7853
0.5750
0.6840
CoOp
0.6482
28.70
5
(+ 0.033)
0.086)
0.015)
0.064)
(+ 0.070)
0.6037
0.6202
0.7893
0.6053
0.6836
OrdinalCLIP
0.6604
28.01
5
(= 0.043)
(= 0.046)
(= 0.018)
(# 0.065)
(+ 0.036)
0.6176
0.6652
0.8002
0.6370
0.7571
VLSS (ours)
0.6954
25.
(+ 0.025)
(- 0.057)
(+ 0.010)
(+ 0.027)
(+ 0.045)
7

---

Published as a conference paper at ICLR 2025
4.3 ANALYSIS OF BAAGUAGGEENNCDER PROGNOSTIC PRIORS
Ablation Study To verify the effectiveness of our instance aggregation method, we compare three baselines: i) attention, classical attention-based aggregation; ii) learnable protolypes. using the same number of learnable vectors without prior knowledge encoding; iii) prognostic lexts, i.e,, only en- coding textual priors, without Tpreg. Their results are shown in Table 2. Our main findings are as follows. (1) Language-encoded prognostic priors play a particularly important role in the performance improvement of VLSA. After incorporating prognostic priors, VLSA obtains im- provements ranging from 1.3% to 6.5% in terms of CI, and an improvement of 3.5% % in average CI. (2) Fine-tuning with Trog boosts the performance with a slight overall improvement (0.4%).
Table 2: Ablation study on our method for instance aggregation. FT is fine-tuning with T prog-
Aggregation method
TCGA
Average
D-cal
BLCA
BRCA
GBMLGG
LUAD
UCEC
MAE
Count
0.6083
0,6180
0.7908
0.6048
0.6908
Attention
0.6625
26.78
5
0.047)
(1: 0.046)
(1: 0.017)
(d: 0.063)
(:: 0.035)
0.5872
0.6201
0.7853
0.6061
0.6845
Learnable prototypes
0.6566
26.76
4
(# 0.048)
(# 0,050)
(# 0.013)
(#: 0,053)
(# 0.052)
0.6159
0.6614
0.7985
0.6314
0.7491
Prognustic texts
0.6912
25.05
4
(#: 0.025)
(+ 0.047)
(# 0.009)
(+ 0.028)
(+ 0.049)
0.6176
0.6652
0.8002
0.6370
0.7571
Prognostic texts FT
0.6954
25.15
5
(1 0.025)
(1t 0.057)
(1: 0.010)
(1: 0.027)
(:: 0.045)
Texts
and
Lake wid
lar poill
Yog Searyy
Regresentbre patches
Centrittiion to dikk
1.4
1,5
0.8
1.0
0.4
4.1
-1.1
P1 Po P4 Ps. Pe Po Pe Pin Pi2 Language enooded prognoadi prior
PL. Px Ps Pa Ps Ps Pr Do Po Pio
P2
D3
P4
P5
P5
D7
Longuageenoodde prognoetic prior
Language onooded prognostic prior
Figure 2: Interpreting the survival prediction of VLSA via language-encoded prognostic pri- ors. Top row shows language descriptions (simplified for better view) about prognostic visual fea- tures in WSIs. Detailed texts are provided in Appendix C.3. Middle row gives the most repre- sentative patches corresponding to each prognostic text. Last row presents each prognostic prior's contribution to risk. We mainly examine the top three language priors in terms of contribution. The three examples are from the first three datasets. More results are shown in Appendix D.2.
Visualization We conduct a case study to further examine the role of language-encoded prognostic priors in our VLSA. As shown in Figure 2, we observe that valuable visual features can be high- lighted by prognostic texts in most cases. These visual features, e.g., "differentiation of tumor cells" and "high cellular density"", could often provide important clues for cancer prognosis. We note that, such desirable behavior is largely attributed to the intrinsic property of VLMs, i.e., VLMs offer well-aligned vision-languag embedding space to enable to manipulate visual features with lan- guage prior. We believe that this property would be particularly important to the MIL in which a model is usually required to aggregate numerous instances under weak supervision.
4.4 ANALYSIS OF ORDINAL INDUCTIVE BIASES
A Abltion Study We conduct ablation studies to verify the effectiveness of our two ordinal inductive bias terms when the language-encoded prognostic priors P are used not used in VI.SA. Its
8

---

Published as a conference paper at ICLR 2025
results are shown in Table 3. We have two main observations from these results. (1) Ordinal survival prompt: when P is not used in VLSA, the improvements by imposing ordinality on survival prompts are 1.2% and 1.3% in average CI; when P is used, the improvements large (3.4% and 3.6%) on BRCA and are often marginal on the other four datasets.
relatively
(2) Ordinal probability: also called ordinal IF, it always shows consistent improvements in terms of average MAE, ranging from 1.2 to 1.7, when P is used in VLSA or not. More discussions on the two inductive bias terms are provided in Appendix B.
Table 3: Ablation study on the two ordinal inductive bias terms in VLSA.
Ordinality
TCGA GBMLGG
Average CI
D-cal Count
Prompt Proha.
BLCA
BRCA
LUAD
UCEC
MAE
w/o language-encoded prognostic priors
0.5971
0.5994
0.7853
0.5750
0.6840
0.6482
28.70
0.033)
(# 0.086)
0.015)
(# 0.064)
0.070)
0.6037
0.6202
0.7893
0.6053
0.6836
0.6604
28.01
(= 0.043)
(= 0.046)
(# 0.018)
(# 0.065)
(# 0.036)
0.5997
0.6049
0.7846
0,5818
0.6769
0.6496
26.99
(- 0.033)
(+ 0.104)
(+ 0.016)
(+ 0.056)
(+ 0.065)
0.6083
0.6180
0,7908
0.6048
0.6908
0.6625
26.78
(= 0.047)
(1 0.046)
(1 0.017)
(1 0.063)
(1 0.035)
w/ language-encolle prognostic priors
0.6128
0.6304
0.7927
0.6351
0.7606
0.6863
26.76
(= 0.028)
(# 0.065)
(# 0.015)
(# 0.041)
(# 0.037)
0.6145
0.6643
0,7973
0,6368
0.7478
0.6921
26.53
(= 0.024)
(+ 0.055)
(+ 0.009)
(+ 0.034)
(+ 0.060)
/
0.6138
0.6293
0.7975
0.6361
0.7592
0.6872
25.23
5
(= 0.022)
(# 0.067)
(# 0.013)
(# 0.036)
(# 0.036)
0.6176
0.6652
0.8002
0.6370
0.7571
0.6954
25.15
5
0.025)
(+ 0.057)
0.010)
0.027)
(+ 0.045)
Discussion We note that there are two possible reasons for the marginal CI gain of two ordinal inductive biases when P is used in VLSA. (1) Ordinal incidence function, often helps improve MAE, not CI, observed from the results of Table 3. (2) The effect of the ordinal inductive bias imposed on survival prompts may be pared down, when the learned visual features become more discriminative (due to the use of P). This is because be tuned better in optimization to align with the discriminative visual features. As result, the effect of the ordinal bias term for prompts could be alleviated. We note that, although there is marginal gain in CI when P is present, the effect ordinal inductive biases on VLSA is often notable the other terms, such MAE and the ordinal Acc in Figure 3(a).
survival prompts are learnable and they can
VUSA Ordinal Frobe.
Freckited Probas
weo Ordinal Prompt
AEMD
0.1
83.84%
0.0
Predlched Prost
0.3
AEMD U.115
AEMO a OLTO4
AEMO OLOBY
VLSA
0.0
ACc
Time Interval
Time interval
Time Intenval
Figure 3: Ordinality visualization. (a) Hcatmap of the similarity between any two learned survival prompts. Its horizontal axis places the first prompt to the last prompt from left to right; its vertical one does from top to bottom. Acc is the accuracy of prompt ranking. The results are from the first three datasets. More results are shown Appendix D.2. (b) Predictive probability (Proba.) w/o and w/ ordinality. The patients from test set used for prediction. AEMD is equal to EMD(y. Uwo)) EMD(y, y, (w)). Vertical dashed line indicates individual time-to-event.
Ordinality Visualization (1) Survival prompts: we calculate the similarity between any two sur- vival prompts learned by the model and show the similarity in heatmap. By comparing the heatmaps

---

Published as a conference paper at ICLR 2025
from the VLSA model with and without ordinal prompts, the effect of ordinal prompts can be exam- ined qualitatively. As shown in Figure 3(a), we can see that survival prompts are better in ordinality when considering
ordinal inductive bias for them. (2) Incidence function: the case study shown in Figure 3(b) suggest that the ordinality constraint imposed on IF could help to decrease the EMI) between y and the ground truth. This often leads the gains in MAE, as seen from Table 3.
4.5 ANALYSIS OF DATA EFFICIENCY (FEW-SHOT SURVIVAL ANALYSIS)
Here focus few-shot learning (FSL) scenario in which only few samples available for training. We use this scenario to simulate the case of scarce training data and assess the performance of different SA methods in terms of data efficiency. Since existing works on SA rarely study FSL, this experiment follows common few-shot settings. Concretely, we randomly sample s patients from each come-discrete class. For censored patients, their true class labels are partially given, so we adopt the KM method (Kaplan & Meier, 1958) to estimate them in advance for few-shot sampling. More- over, due to small patient numbers, we run each single experiment times and report the median metries, following CONCH (Lu et al., 2024). MI-FewShot (Xu et al., 2024) is a non-parametric method designed for FSL, so we adopt it as an additional baseline for comparisons.
Table 4: Few-shot survival analysis for data efficiency evaluation.
# Shots
Method
Average
Full-shots
2
4
8
16
ABMIL
0,5061
0.5136
0,5339
0.5541
0.5798
0,5375
0.6426
TransMIL
0.5607
0.5856
0.6145
0.6236
0.6388
0.6046
0.6414
ILRA
0.5503
0.5658
0.5891
0.6023
0.6033
0.5822
0,6136
RºT-MI.
0.5725
0.5835
0.5924
0.5927
0.6072
0.5897
0.6245
DeepAttnMISL
0,5182
0,5252
0.5522
0.5682
0,5877
0,5503
0.5736
Patch-GCN
0,5627
0.5497
0,5872
0.6022
0.6186
0,5841
0.6726
MI-FewShot
0.5478
0.5439
0.5499
0.5539
0.5873
0.5567
0.5692
ABMIL
0.5314
0,5588
0.5737
0.5958
0.6068
0.5733
0.6500
CoOp
0.5846
0.5870
0.6081
0.6056
0.6237
0,6018
0.6482
OrdInalCLIP
0,5710
0.5983
0.6118
0.6106
0,6246
0,6033
0,6604
VLSA
0.5787
0.6068
0.6271
0.6465
0.6592
0.6237
0.6954
The results are shown in Table 4. (1) From these results, we find that our VLSA still often obtains the best performance in FSL scenarios. This indicates the data efficiency of our VLSA models. (2) When using only 16 shots, our VLSA can obtain comparable performance with nearly-all full- shot baseline models. (3) Furthermore, we observe that, in standard full-shot setting, two VL- based baselines, i.e., CoOp and OrdinalCIIIP with ABMIL as their MIL. aggregator, only show an improvement of 0.6% and 1.8% over visl ABMIL, respectively. By contrast, the average improvements seen in few-shot settings are around 6.5%, much larger than those in full-shot. This result also suggests the advantage of VL.-based schemes over vision-only ones in data efficiency.
5 CONCLUSION
This paper presents a new Vision-Language Survival Analysis (VLSA) paradigm. It designs the first VL-based SA framework for histopathology WSIs. Different from current SA paradigm in CPATH, VLSA proposes to leverage language-encoded prognostic priors as auxiliary signals to improve weakly-supervised WSI representation learning. Moreover, considering the intrinsic characteristics of SA, VLSA proposes ordinal survival prompt learning and ordinal incidence function prediction. Owing to the introduce of language-encoder prognostic priors, our VLSA enjoys a good inter- pretability, supported by its Shapley values-based interpretation method. The extensive experiments on five datasets verify our scheme's effectiveness: VLSA obtains new state-of-the-ar performances in SA tasks and it could often show clear superiority over existing schemes in terms of data and computation efficiency. Notably, our empirical results suggest that VL-based learning paradigm can offer weakly-supervised MIL an effective means to learn valuable prognostic clues from gigapixel WSIs. This finding is likely to provide an alternative solution for improving the weak supervision in MII,, thereby paving a new way for survival analysis in computational pathology.
10

---

Published as a conference paper at ICLR 2025
6 ACKNOWLEDGMENT
This work is supported by the National Natural Science Foundation of China (NSFC) under Grant No. 62476049, partly by the Fundamental Research Funds for the Central Universities under Grant No. ZYGX2022YGRII GGR015 and the project of Sichuan Provincial Department of Science and Tech- nology under Grant No. 2023YFQ0011. The authors would like thank Mahmood Lab for making CONCH public to facilitate the research, and anonymous reviewers for their constructive comments to improve this work.
REFERENCES
Faruk Ahmed, Andrew Sellergren, Lin Yang, Shawn Xu, Boris Babenko, Abbi Ward, Niels Olson, Arash Mohtashamian, Yossi Matias, Greg Corrado, et al. Pathalign: A vision-language model for whole slide images in histopathology. arXiv preprint arXiv:2406. 2024.
Richard Chen, Ming Y Lu, Muhammad Shaban, Chengkuan Chen, Tiffany Y Chen, Drew FK Williamson, and Faisal Mahmood, Whole slide images are 2d point clouds: Context-aware sur- vival prediction using patch- graph convolutional networks. In Medical Image Computing and Computer Assisted Intervention, pp. 339-344 239-349. Springer, 2021a.
Richard J Chen, Ming Lu, Wei-Hung Weng, Tiffany Y Chen, Drew FK Williamson, Trevor Manz, Maha Shady, and Faisal Mahmood. Multimodal co-attention transformer for survival prediction in gigapixel whole slide images. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4015-4025, 2021b.
Richard Chen, Chengkuan Chen, Yicong Li, Tiffany Chen, Andrew D. Trister, Rahul G. Kr- ishman, and Faisal Mahmood. Scaling vision transformers to gigapixel images via hierarchical self-supervised learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 16144-16155, June 2022.
D. R. Cox. Partial likelihood. Biometrika, 62(2):269-277 276, 1975.
Thomas G Dietterich, Richard H Lathrop, and Tomás Lozano-Pérez Solving the multiple instance problem with axis- axis-parallel rectangles. Artificial intelligence, 89(1-2):31-71, 1997.
Jason P Fine and Robert J Gray. A proportional hazards model for the subdistribution of a competing risk. Journal of the American statistical association, 94(446):496-509, 1999.
Olga Fourkioti, Matt De Vries, and Chris Bakal. CAMIL: Context-aware multiple instance learning for cancer detection and subtyping in whole slide images. In The Twelfth International Confer- ence on Learning Representations, 2024. URL https:::///opeereee /wwp//oveeeeeweeeee/ee net htt//formmmm//m did= rzBskAEmoc.
Humza Haider, Bret Hoehn, Sarah Davis, and Russell Greiner. Effective ways to build and evaluate individual survival distributions. Journal of Machine Learning Research, 21(85):1-63, 2020.
Frank E Harrell Jr, Kerry L Lee, and Daniel B Mark. Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors. Statistics in medicine, 15(4):361-377, 1996.
Le Hou, Chen- then-Ping Yu, and Dimitris Samaras. Squared earth mover's distance-based loss for train- www.orr//aabb.oeee.eeeeeeeeebbee /tt.///111...91
ing deep neural networks, 2017. URL https::t////
Wentai Hou, Lequan Yu, Chengxuan Lin, Helong Huang, Rongshan Yu, Jing Qin, and Liansheng Wang. H2-mil: Exploring hierarchical representation with heterogeneous multiple instance learn- ing for whole slide image analysis. Proceedings the AAAI Conference on Artificial Intelligence, 36( (6(1):933-941 2022.
Zhi Huang, Federico Bianchi, Mert Yuksekgonul, Thomas J Montine, and James Zou. A visual- language foundation model for pathology image analysis using medical twitter. Nature medicine, 29(9):2377-2316 2023. 2023.
11

---

Published as a conference paper at ICLR 2025
Ziwang Huang, Hua Chai, Ruoqi Wang, Haitao Wang, Yuedong Yang, and Hejun Wu. Integration of patch features through self-supervised learning and transformer for survival analysis on whole slide images. In International Conference Medical Image Computing and Computer-Assisted Intervention, pp. 561-570. Springer, 2021.
Wisdom Ikezogwo, Saygin Seyfioglu, Fatemeh Ghezloo, Dylan Geva, Fatwir Sheikh Mohammed, Pavan Kumar Anand, Ranjay Krishna, and Linda Shapiro. Quilt- Im: One million image-text pairs for histopathology. Advances in neural information processing systems. 36, 2023.
Maximilian llse, Jakub Tomezak, and Max Welling. Attention- deep multiple instance learn- ing. In International Conference on Machine Learning, pp. 2127-2136. PMLR, 2018.
Guillaume Jaume, Lukas Oldenburg, Anurag Vaidya, Richard J Chen, Drew FK Williamson, Thomas Peeters, Andrew H Song, and Faisal Mahmood. Transeriptomics-guiee slide repre- sentation learning in computational pathology. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 1632-9644, 2024.
Sajid Javed, Arif Mahmood, lyyakutti lyappan Ganapathi, Fayaz Ali Dharejo, Naoufel Werghi, and Mohammed Bennamoun. Cplip: Zero-shot learning for histopathology with comprehensive vision-language alignment. In Proceedings the the IEFE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1 11450-11459, June 2024.
Edward L Kaplan and Paul Meier. Nonparametric estimation from incomplete observations. Journal of the American statistical ussociation, 53(282):457-481, 1958.
Jakob Nikolas Kather, Johannes Krisam, Pornpimol Charoentong, Tom Luedde, Esther Herpel, Cleo-Aron Weis, Timo Gaiser, Alexander Marx, Nektarios A. Valous, Dyke Ferber, Lina Jansen, Constantino Carlos Reyes-Aldasoro, Inka Zörnig, Dirk Jäger, Hermann Brenner, Jenny Chang- Claude, Michael Hoffmeister, and Niels Halama. Predicting survival from colorectal cancer his- tology slides using deep learning: retrospective multicenter study. PLOS Medicine, 16(1): e1002730, jan 2019. ISSN 1549-1676. doi: 10. (0.1371/jooul 1/1.117//ouraallpmee.l. 1002730.
Soheil Kolouri, Se Rim Park, Matthew Thorpe, Dejan Slepcev, and Gustavo K Rohde. Optimal mass transport: Signal processing and machine-leaming applications. IEEE signal processing magazine, 34(4):43-59, 2017.
Changhee Lee, William Zame, Jinsung Yoon, and Mihaela Van Der Schaar. Deephit: A deep learn- ing approach to survival analysis with competing risks. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018
Mei-Ling Ting Lee and GA Whitmore. Threshold regression for survival analysis: Modeling event times by a stochastic process reaching boundary. Statistical Science, 21(4):501-513, 2006.
Elizaveta Levina and Peter Bickel. The earth mover's distance is the mallows distance: Some in- sights from statistics. In Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001, volume 2. pp. 251-256. IEEE, 2001.
Hao Li, Ying Chen, Yifei Chen, Rongshan Yu, Wenxian Yang, Liansheng Wang, Bowen Ding, and Yuchen Han. Generalizable whole slide image classification with fine-grained visual-semantic interaction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, pp. 11398-11400 11477, 2024.
Honglin Li, Chenglu Zhu, Yunlong Zhang, Yuxuan Sun, Zhongyi Shui, Wenwei Kuang, Sunyi Zheng, and Lin Yang. Task-specific fine-tuning via variational information bottleneck for weakly- supervised pathology whole slide image classification. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pp. 7454-7463, 2023.
Wanhua Li, Xiaoke Huang, Zheng Zhu, Yansong Tang, Xiu Li, Jie Zhou, and Jiwen Lu. Ordi- nalclip: Learning rank prompts for language-guidel ordinal regression. In Advances in Neural Information Processing Systems, volume 35, pp. 35313-35325, 2022.
Pei Liu and Luping Ji. estimation. In Proceedings of the 41st International Conference on Machine Learning, volume 235, pp. 31262-3121111 31222 PMLR, 2024.
residual evidential learning for multi- instance uncertainty
12

---

Published as a conference paper at ICLR 2025
Pei Liu, Bo Fu, Feng Ye, Rui Yang, and Luping Ji. DSCA: A dual-stream network with cross- attention on whole-slide image pyramids for cancer prognosis. Expert Systems with Applications, 227:120280 2023a. ISSN 0957-4174.
Pei Liu, Luping Ji, Feng Ye, and Bo Fu. Graphtsurv: A scalable survival prediction network with adaptive and sparse structure learning for histopathological whole-slide images. Computer Meth- ods and Programs in Biomedicine, 231:107433, 2023b.
Pei Liu, Luping Ji, Feng Ye, and Bo Fu. Advmil: Adversarial multiple instance learning for the survival analysis whole-slide images. Medical Image Analysis, 91:103020, 2024a.
Pei Liu, Luping Ji, Xinyu Zhang, and Feng Ye. Pseudo-bag mixup augmentation for multiple in- stance baaming-based whole slide image classification. IEEE Transactions on Medical Imaging, 43(5):1841-1852 2024b. doi: 10. 10.10099TM1.22243551211
Ming Y Lu, Drew FK Williamson, Tiffany Y Chen, Richard J Chen, Matteo Barbieri, and Faisal Mahmood. Data-efficient and weakly supervised computational pathology on whole-slide images. Nature biomedical engineering, 5(6):555-570, 2021.
Ming Y. Lu, Richard J. Chen, Dehan Kong, Jana Lipkova, Rajendra Singh, Drew F.K. Williamson, Tiffany Chen, and Faisal Mahmood. Federated learning for computational pathology on gi- gapixel whole slide images. Medical Image Analysis, 76: 76:102298, 2022.
Ming Y Lu. Bowen Chen, Andrew Zhang. Drew FK Williamson, Richard Chen, Tong Ding, Long Phi Le, Yung-Sung Chuang, and Faisal Mahmood. Visual language pretrained multiple stance zero-shot transfer for histopathology images. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 19764-19777 2023.
Ming Y Lu, Bowen Chen, Drew FK Williamson, Richard Chen, Ivy Liang, Tong Ding, Guillaume Jaume, Igor Odintsov, Long Phi Le, Georg Gerber, et al. A visual- visual-language foundation model for computational pathology. Nature Medicine, 30:863-874 2024.
Scott Lundberg. A unified approach to interpreting model predictions. arXiv preprint
arXiv:1705.072 077744 2017.
Roni Paiss, Ariel Ephrat, Omer Tov, Shiran Zada, Inbar Mosseri, Michal Irani, and Tali Dekel. Teaching clip to count to ten. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 31 3170-3180, 2023.
Pushpak Pati, Guillaume Jaume, Antonio Mmncubierta-Rodrigunn Florinda Feroce, Anna Maria An- niciello, Giosue Scognamiglio, Nadia Brancati, Maryse Fiche, Estelle Dubruc, Daniel Riccio, Maurizio Di Bonito, Giuseppe De Pietro, Gerardo Botti, Jean-Philippe Thiran, Maria Frucci, Or- cun Goksel, and Maria Gabrani. Hierarchical graph representations in digital pathology. Medical Image Analysis, 75:102264, 2022. doi: 10.
Shi-Ang Qi, Neeraj Kumar, Mahtab Farrokh, Weijie Sun, Li-Hao Kuan, Rajesh Ranganath, Ricardo Henao, and Russell Greiner. An effective meaningful way to evaluate survival models. In Inter- national Conference on Machine Learning, pp. 28244-28276. PMLR, 2023a.
Shi-ang Qi, Weijie Sun, and Russell Greiner. Survivaleval: A comprehensive open-source python package for evaluating individual survival distributions. Proceedings of the AAAI Symposium Series, volume 2, pp. 453-457, 2023b.
Linhao Qu, xiaoyuan luo, Kexue Fu, Manning Wang, and Zhijian Song. The rise of ai language pathologists: Exploring two-level prompt learning for few-shot weakly-superviset whole slide image classification. In Advances in Neural Information Processing Systems, volume 36, pp.
67551-67564, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML., 2021.
13

---

Published as a conference paper at ICLR 2025
Wei Shao, Tongxin Wang, Zhi Huang, Zhi Han, Jie Zhang, and Kun Huang. Weakly supervised deep ordinal cox model for survival prediction from whole-slide pathological images. IEEE Transac- tions Medical Imaging, 40(12):3739-377 -347, 2021a.
Wei Shao, Yang Yang Shi, Daoqiang Zhang, JunJie Zhou, and Peng Wan. Tumor micro-environment interactions guided graph learning for survival analysis of human cancers from whole-slide patho- logical images. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 11694-11703, June 2024.
Zhuchen Shao, Hao Bian, Yang Chen, Yifeng Wang, Jian Zhang, Xiangyang Ji, and Yongbing Zhang. Transmil: Transformer based correlated multiple instance learning for whole slide image classification. Advances in Neural Information Processing Systems, volume 34, pp. 2136-2147, 2021b.
Lloyd S Shapley. A value for n-person games. Contributions to the Theory of Games, 2(28):307- 317. 1953.
Jiangbo Shi, Chen Li, Tieliang Gong, Yefeng Zheng, and Huazhu Fu. Vila-mil: Dual-scale vision- language multiple instance learning for whole slide image classification. In Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition, pp. 11248-11258 2024.
Ole-Johan Skrede, Sepp De Raedt, Andreas Kleppe, Tarjei Hveem, Knut Liestøl, John Maddison, Hanne A Askautrud, Manohar Pradhan, John Ame Nesheim, Fritz Albregtsen, Inger Nina Farstad, Enric Domingo, David N Church, Arild Nesbakken, Neil A Shepherd, Ian Tomlinson, Rachel Kerr, Marco Novelli, David J Kerr, and Håvard E Danielsen. Deep learning for prediction of colorectal cancer outcome: a discovery and validation study. The Lancet, 395(102211:350-366, feb 2020. ISSN 01406736. doi: 10.1016/50140-60 103144466-1191339998888
Andrew H Song, Guillaume Jaume, Drew FK Williamson, Ming Y Lu, Anurag Vaidya, Tiffany R Miller, and Faisal Mahmood. Artificial intelligence for digital and computational pathology. Na- ture Reviews Bioengineering, 1(12):930-949, 2023.
Andrew H. Song, Richard J. Chen, Tong Ding, Drew F.K. Williamson, Guillaume Jaume, and Faisal Mahmood. Morphological prototyping for unsupervised slide representation learning in compu- tational pathology. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern June 2024a.
Recognition (CVPR), pp. 11566-11578,
Andrew H. Song, Richard Chen, Guillaume Jaume, Anurag Jayant Vaidya, Alexander Baras, and Faisal Mahmood. Multimodal prototyping for cancer survival prediction. In Proceedings of the 41st International Conference on Machine Learning, volume 235, pp. 46050-46073, PMLR, 2024b.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, (111):1929-19555 2014.
Wenhao Tang, Sheng Huang, Xiaoxian Zhang, Fengtao Zhou, Yi Zhang, and Bo Liu. Multiple instance learning framework with masked hard instance mining for whole slide image classifica- tion. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), pp- 4078 4087, October 2023.
Wenhao Tang, Fengtao Zhou, Sheng Huang. Xiang Zhu, Yi Zhang, and Bo Liu. Feature re- embedding: Towards foundation model-level performance in computational pathology. In Pro- ceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp- 11343-11355 June 2024.
Avijit Thawani, Jay Pujara, Pedro A Szekely, and Filip llievski. Representing numbers in nlp: survey and a vision. arXiv preprint 11Xi:22003.131366 2021.
Gerhard Tutz and Matthias Schmid. Modeling discrete time-to-event data. Springer, 2016.
Zhikang Wang. Jiani Ma, Qian Gao, Chris Bain, Seiya Imoto, Pietro Liò, Hongmin Cai, Hao Chen, and Jiangning Song. Dual-stream multi--eependency graph neural network enables precise cancer survival analysis. Medical Image Analysis, 10:252 2024.
14

---

Published as a conference paper at ICLR 2025
Jinxi Xiang and Jun Zhang. Exploring low-rank property in multiple instance learning for whole slide image classification. In International Conference on Learning Representations, 2023.
Conghao Xiong, Hao Chen, Joseph Y Sung, and Irwin King. Diagnose Like a Pathologist: Transformer--nabbled Hierarchical Attention-Guided Multiple Instance Learning for Whole Slide Image Classification. In Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, IJCAI-23, pp. 1587-1595, 2023.
Conghao Xiong, Hao Chen, Hao Zheng, Dong Wei, Yefeng Zheng. Joseph J Y Sung, and Irwin King. MoME: Mixture of Multimodal Experts for Cancer Survival Prediction. In Medical Image Computing and Computer Assisted Intervention MICCAI 2024, pp. 318-328, 2024.
Yingxue Xu and Hao Chen. Multimodal optimal transport-based co-attention transformer with global structure consistency for survival prediction. In Proceedings the IEEE/CVF Interna- tional Conference on Computer Vision (ICCV), pp. 21241-21251, October 2023.
Yingxue Xu, Yihui Wang, Fengtao Zhou, Jiabo Ma, Shu Yang, Huangjing Lin, Xin Wang, Jiguang Wang, Li Liang, Anjia Han, Ronald Cheong Kin Chan, and Hao Chen. A multimodal knowledge- enhanced whole-slide pathology foundation model, 2024. URL https: /awxiv.vvvv..////i///a/////aavva orgg abs/
Jiawen Yao, Xinliang Zhu, Jitendra Jonnagaddala, Nicholas Hawkins, and Junzhou Huang. Whole slide images based cancer survival prediction using attention guided deep multiple instance learn- ing networks. Medical Image Analysis, 65:101789, 2020.
Chun-Nam Yu, Russell Greiner, Hsiu-Chin Lin, and Vickie E. Baracos. Learning patient-specific cancer survival distributions sequence of dependent regressors. In Neural Information Pro- cessing Systems, 2011.
Shekoufeh Gorgi Zadeh and Matthias Schmid. Bias in cross-entropy- training of deep survival networks. IEEE transactions on pattern analysis and machine intelligence, 4369):3126-3137, 2020.
Mark D) Zarella, Douglas Bowman; Famke Aeffner, Navid Farahani, Albert Xthona;, Syeda Fatima Absar, Anil Parwani, Marilyn Bui, and Douglas J. Hartman. Practical Guide to Whole Slide Imaging: White Paper From the Digital Pathology Association. Archives of Pathology & Labo- ratory Medicine, 143(2):222-234, 10 2018. ISSN 0003-9985,
doi: 10.885/aaa.2008844333AAA
Hongrun Zhang, Yanda Meng. Yitian Zhao, Yihong Qiao, Xiaoyun Yang, Sarah E. Coupland, and Yalin Zheng. Dtfd-mil: Double-tier feature distillation multiple instance learning for histopathol- ogy whole slide image classification. In 2022 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 18780-18790, 2022.
Jingyi Zhang, Jiaxing Huang, Sheng Jin, and Shijian Lu. Vision-language models for vision tasks: A survey. IEEE Transactions Pattern Analysis and Machine Intelligence, 2024a.
Yilan Zhang, Yingxue Xu, Jianqi Chen, Fengying Xie, and Hao Chen. Prototypical information bottlenecking and disentangling for multimodal cancer survival prediction. In The Twelfth Inter- national Conference on Learning Representations, 2024b.
Fengtao Zhou and Hao Chen. Cross-modal translation and alignment for survival analysis. In Proceedings of the IEEE/CVF Internutional Conference Computer Vision, pp. 21485-21494, 2023.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision- language models. International Journal of Computer Vision, 13019):2337-2348, 2022.
15

---

Published as a conference paper at ICLR 2025
APPENDIX
A Preliminaries
17
A.I Survival Analysis
17
A.2 Weakly--uuprrvsed MIL
17
A.3 Earth Mover's Distance
18
A.4 Shapley Value
18
More Discussions on Ordinal Inductive Bias
19
More Experimental Details
20
C.1 Datasets
20
C.2 Baselines
20
C,3 VLSA
C,4 Evaluation Metrics
23
C,5 Time Discretization Settings
24
Additional Experimental Results
24
D.I Risk Grouping and Kaplan-Meier Analysis
24
D.2 Additional Visualization Results
................................
................................
25
D.3 Analysis of Computation Efficiency
25
D.4 Comparison with Hazard-Based Survival Modeling
26
................................
D.5 Comparison with Multi-Modal Survival Analysis Methods
26
D.6 Different Time Discretization Settings
27
D.7 Adopting PLIP as VLM for Survival Analysis
27
Limitation and Future Work
28
16

---

Published as a conference paper at ICLR 2025
A PRELIMINARIES
A.1 SURVIVAL ANALYSIS
Survival analysis (SA) Also known as time-to-event analysis, SA is one of the primary statistical approaches for analyzing data time to event (Kaplan & Meier, 1958; Cox, 1975). Time-to-event data is usually denoted by D = {X,, ts, 6, P, where X; represents individual characteristics and 5, is the individual status of event of interest (e.g. death and machine failure) observed time t,- Particularly, 8 O, called censorship in SA, indicate that event does not occur during follow-up observation. They are also considered in analysis and modeling. One common task in SA to predict the risk of event occurrence. It covers wide range of applications in healthcare and finance. Next, we give some key concepts in SA for reference.
Survival Function It quantifies the probability one will survive past given time often written as S(t = Pr(T > t) where T is a random variable indicating survival time. This function is non-increasing, i.e. the probability of survival will not increase as the observation process con- tinues. Given individual survival function, the corresponding time-to-event can be estimated by i == [,((()) = Jo S(t)di.
Hazard Function It quantifies the probability that one will experience event at time given no event occurrence before L. By definition, it is written as h(() lim Pr(S55HAA(P>4) At
Thus, survival
Ar-+
function can be derived from it by S(t) = exp(- fo h(=)dz).
Incidence Function (IF) It aims to capture the probability distribution of the first hitting time, p(t), often seen in SA under competing risks (Fine & Gray, 1999; Lee & Whitmore, 2006; Lee et al., 2018). Cumulative IF (CIF) gives the probability that one experiences event at or before time t, namely, CIF(t) = fo p(z)d2. Thus, survival function can be calculated by S(t) - CIF(tt.
Modeling Approach To estimate individual risks, most survival modeling approaches take the fol- lowing functions as prediction target.
Proportional hazard (Cox, 1975). As the most popular one (CoxPH) in SA, it assumes that individual hazard function is proportional to a cohort-level hazard function (computable); it predicts scalar indicating individual proportional hazard.
Hazard function (Tutz& & Schmid, 2016). Traditional SA models usually assume a specific form for the distribution of individual hazard function, and then optimize the parameter of this specified distribution. Modeling hazard function without any explicit assumption is also studied in SA. It often performs better than traditional ones in real-world datasets.
Incidence function. In DeepHit (Lee et al., 2018), it is first taken as prediction target for deep searning-basee SA. Its optimization are similar to classification with partial labels (refer to Eq. (9)). One biggest difference is that modeling incidence function requires to handle the censored individuals (whose true time-to-event is partially known) in D.
Additionally, survival time is also directly adopted as prediction target in some SA models (Liu et al., 2024a). Readers could refer to Tutz & Schmid (2016) for more details about SA.
When prediction target is incidence function or hazard function, survival time is usually prepro- cessed as discrete time labels for analysis. We compare these two prediction targets and try different time discretization settings for our V..AA. The results can be found in Appendix D.6.
A.2 WEAKLY-SUPERVISHR MIL
Multi- Insti-Instance Learning (MIL) is a fundamental machine learning problem that has been studied for decades (Dietterich et al., 1997; Ilse et al., 2018; Liu & Ji, 2024). Different from conventional learning settings, a sample in MIL is bag of multi-instances, i.e. X {k æg is the MIL, bag-level labels are given while instance-level labels unknown. A MIL model is often required to leam from size-varied bags and make bag-leve or instance-leve predictions. Its applications cover pathology image analysis, video anomaly detection, point could analysis, etc.
k-th instance. In weakly
17

---

Published as a conference paper at ICLR 2025
Embedding-level approach (Ilse et al., 2018) is often adopted for bag-level prediction in the era of deep learning. This approach provides a principled and general three-step framework for
MIL. (1) Instance-level embedding: E3 is transformed into a low-dimensional embedding hg- (2)
1 permutation-ivvarian pooling: (h& k== are combined into a bag-level representation hag via
crrmutation-invarian pooling. (3) Bag classification: bag-level representation is passed through a classifier for prediction. Common pooling strategies contain mean, max, and attention. Particu- larly, attention-based pooling is a classic method proposed by Ilse et al. (2018), frequently adopted in weakly- supervsup MIL. It can be written as follows:
exp (sp(hx)
bag
hgg ag Softmax (f((t))) ag
(12)
=== exp (v²(h;)
k=1
where (p-) is a transformation function in the attention network, parameterized by a multi-layer perceptron. Embedding-level approach usually assumes #1, 22, ... &x are i.... Subsequent works (Shao et al., 2021b) improve by assuming and handling the relation between instances. They often perform better in bag classification tasks.
A.3 EARTH MOVER'S S DISTANCE
Earth Mover's Distance (EMD), as known the Wasserstein distance (p 1), is utilized to the measure the distance between two distributions (Kolouri et al., 2017). Intuitively, considering two simple one-dimensional discrete probability distributions, P(x) and Q(x), EMD captures the opti- mal total transportation costs in moving the probability mass of P from any point to to make P equal Q in probability density. In every moving process, the cost is the product of moving mass and moving distance (la - bl). From this intuitive example, we could find that EMD considers the geometry property of distribution in distance measurement. This makes EMD different from other metrics like Kullback-Leibler divergence and Euclidean distance.
Formally, EMD is defined as infimum over joint probabilities:
inf EMD(P,Q) 7El((P.)) [(,,))-d(zz,))
(13)
II(P,Q) Q) is the set of all joint distributions whose marginal distributions are P and Q. d(x, y) the distance between and y. As shown in Levina & Bickel (2001), for two one-oonentt one-dimensional discrete distributions. and Q EMD can be calculated by
EMD(P, Q) (+) |ICD((P) CDF(Q)]l.
(14)
where CDF means cumulative density function and C' is the number of discrete values.
A.4 SHAPLEY VALUE
Shapley value is classic solution concept in cooperative game theory (Shapley, 1953). It assigns a unique contribution to each player according to the payoff earned by the coalition of players. In machine learning, Shapley value is often adopted to interpret model predictions (Lundberg, 2017),
e.g., the importance of input features. This application enables us to interpret black-box, complex nonlinear models, e.g., deep networks, particularly popular in the medical domain.
Formally, to calculate the importance of each feature for an original model f(.), there is f(x) = g(x") = (x'())))) for corresponding explanation model g(-), where x' is simplified input that maps to the original input x through a function x ha (x'). Additive feature attribution meth- ods provide such explanation models. Concretely, g(-) formulated as a linear function of binary variables. It is expected to have property of local accuracy, expressed as
f(x) g(x') 6
(15)
where o, E R indicates the importance of the i-th feature, x' E {0, 1)A,, and is the number of sim- plified input features. Moreover, g(.) should satisfy two other desirable properties: i) missingness,
18

---

Published as a conference paper at ICLR 2025
missing features (x} 0) have no attributed impact; and ii) consistency, some features' attribu- tion should not decrease if these features' contribution (i.e., marginal reward) does not decrease. Theorem I in Lundberg (2017) states that only one possible g()) follows those three properties:
|=
|='| - 1)! [fa(z) fm ()),
o1(f, = Σ
(16)
A!
where |a'| is the number of non-zero entries in z' and x' C x' represents all z' vectors where the non-zero entries are subset of the non-zero entries in x'. This theorem follows from combined cooperative game theory results. Φ; is known as Shapley value (Shapley, 1953). Readers could refer Lundberg (2017) for proofs and related details.
B MORE DISCUSSIONS ON ORDINAL INDUCTIVE BIAS
Our VLSA considers two ordinal inductive biases: ordinal survival prompts and ii) ordinal in- cidence function. For better understanding, in this section we first explain them in detail and then discuss the connection and difference between them.
Further Explanation (1) Ordinal survival prompts. For C survival classes after time discretization, there prompts that encode the information of survival labels. Concretely, from the first class to the last class, i.e., for time-to-event ranging from [To, Ti to [I Tc), the respective class prompt describes a risk from high to low. Therefore, if two classes are closer in ordering, egg., the e-th and (c++ 1- classes, their class prompts would be more similar in encoded risk information. Formally,
Sim(Vá. Vá.. >> Sim(Vass Vál) if |c [c--l] < lc-jl.
(17)
where -im function measuring two inputs' similarity. (2) Ordinal incidence function. It captures the probability distribution of discrete first-hitting time, i.e., = yr. ycl. Since this distribution also characterize ordered survival classes like the class prompt aforementioned, there is similar ordinality in incidence function,
We yil We Vıl if |c < c jl.
(18)
as depicted in Figure 1(b). Our experiments (Table 3) confirm the effectiveness of these two ordinal inductive biases: i) ordinal survival prompts tend to enhance the model's ability in risk discrim- ination; and ii) ordinal incidence function could often reduce the bias of predicted survival time (MAE).
Lee
Let1,0
fest
J
co+1 text
Dee
f f
f
fimage
Figure 4: An example to illustrate the case that incidence function is not ordinal whereas prompts are. Le,o+1 is the bisector of the angle between f and pet1 When fimage falls into the gray area,
(c-1) > cos( fmage f(-2) does not hold.
Connection and Difference between Two Ordinal Inductive Biases Taking two inductive biases into one formula, prediction individual incidence function can be written as
ý Softmax (7 [cos(finage, fiva) cos cos(fimage images fex).... c.oss f(a))).
(19)
19

---

Published as a conference paper at ICLR 2025
So, the first inductive bias, introduced into (ffe could be cast model-level ordinal con- straint, because Fiex is part model parameters and is used all individual predictions. The second inductive bias, imposed on individual prediction y. is subject-level ordinal constraint, We claim that only considering model-level ordinal constraint cannot always lead to ordinal incidence function, i.e., Eq. (18) does not hold in some
when Eq. (17) holds for all c, E [],
We show example in Figure 4. In this example, ficu: fist fistt are ordinal and there cingfe fixa) cos(fe fiett) and cos( cos(ffimges flea) >cos<ccssfff comgeeaager fiett). However, there ists an area (colored gray) so that cos(fimage- fiett)) cos(ffmage- fistt) does not hold. Namely, there exists the case in which incidence function is not ordinal even if all survival prompts satisfy the ordinality. Therefore, both model-level and subject-level ordinal constraint should be considered in VLSA for better survival prediction. This is also implied by our empirical results in Table 3.
C MORE EXPERIMENTAL DETAILS
C.I DATASETS
We provide the statistics on five TCGA datasets (BLCA, BRCA, GBMLGG, LUAD, and UCEC) used in this study. Please see them in Table 5.
The preprocessing of histopathology WSIs contains the following key steps: i) tissue region seg- mentation, ii) image patching, and iii) patch feature extraction. Following CONCH (Lu al., 2024), each WSI is divided into patches with size of 448 x 448 pixels at 20x magnification. In the last step, CONCH's image encoder used to obtain patch features. We utilize toolkit provided by CLAM (Lu et al., 2021) for these steps. The statistical details on processed instances (patches) are provided in Table 5.
There are three notable numbers in Table 5. (1) The number of patients in each dataset is smaller than 1,000. This scale too small to compare with current standard vision datasets. This poses great challenges to survival analysis on WSIs. (2) The number of instances for one patient can reach up to 134,016 in UCEC. Thus, it could be challenging for MIL models to learn one effective representation from numerous patches. (3) The number of time bins is usually around 10. This means that manually designing fine-grained risk descriptions at all levels is often intractable.
Table 5: Dataset details. The last rows are calculated patient level.
TCGA GBMLGG
Statistic
BLCA
BRCA
LUAD
UCEC
# Patients
373
956
569
453
480
# Patients w/ event (N.)
169
130
189
158
75
Time bins (vN.)
Event ratio
45.31% %
13.60%
33.22%
34.88%
15.63%
Maximum (month)
163.17
282.69
211.01
238.11
225.33
# WSIs
437
1,022
1,016
516
539
# WSIs patient
1.17
1.07
1.79
1.14
1.12
# Instances (sum)
2,463,402
4,113.285
3,508.119
2,302,933
3,229,606
# Instances (mean)
6,604.3
4,302.6
6,165.4
5,083.7
6,728.3
# Instances (max)
69,843
23,263
75,012
63,658
134,016
C.2 BASELINES
Vision-Only Baselines We compare the baselines originally proposed for WSI classification, i.e.,
ABMIL (Ilse al., 2018), TransMIL (Shao al., 2021b), ILRA (Xiang Zhang, 2023), and R²T- MIL (Tang et al., 2024), and those for WSI survival analysis, i.e., DeepAttMMISL (Yao et al., 2020) and Patch-Grt GCN (Chen et al., 2021a). DeepAutMISL and Patch Cech--GC are the most representative method based on cluster and graph, respectively.
'https://e :/ hwwb.ccom/aaamme.coo ab/ CLAAM
20

---

Published as a conference paper at ICLR 2025
Vision-Language Baselines Since the vast majority of existing works focus on classification or segmentation tasks, we adapt the related methods to WSI survival analysis. These methods include MI-Zero (Lu et al., 2023) for zero-shot WSI classification, ABMIL CoOp (Zhou et al., 2022) for context prompt optimization, and OrdinalCI.IP (Li et al., 2022) for VI.-based ordinal regression.
Implementation Details Baselines implemented follows:
ABMIL,, TransMIL. ILRA, and T-M-L we adapt them to SA by replacing their predic- tion target with incidence function and using the MIE same as our VLSA. Moreover, their time discretization settings (Section 3.2) ares the same as VLSA.
DeepAttnMISL and Patch-GCN: we follow their original implementations for SA tasks. Concretely, DtepAttMMSL adopts survival modeling approach same as CoxPH (Cox, 1975). It predicts a proportional hazard. We transform it into discrete survival distribution, based on standard KM method (Kaplan & Meier, 1958; Qi al., 2023b). Patch-GCN predicts discrete hazard function for individuals and is optimized by LMLL, where survival time is discretized by quantiles.
MI-Zero: In vision-end, we try different score aggregation methods proposed by it obtain WSI representations and report the best one, i.e., mean-based aggregation. In language- end, we try different survival prompts for MI-Zero and finally adopt the best: context "the cancer prognosis reflected in the pathology image is", classes = ["very poor", "poor", " "good", "very good"). The final four survival prompts are encoded into four textual vectors by the CONCH's text encoder. They are then used to derive the prediction of IF via Eq. (6), same as our VLSA.
ABMIL Prompt: In vision-end, ABMIL is used to learn WSI-level representations. In language-end, the naive prompts (prefixed and not learnable), same as those used in MI- Zero and described above, are used for survival prediction. baseline than MI-Zero for comparisons.
adopted as a stronger
.
CoOp and OrdinalCLIP: the implementation of CoOp and OrdinalCLIP are stated in the main paper. Unless specified, they use the same setting as our VL.S..
C.3 VLSA
This section provides the implementation details of our VLSA, including the description of prog- nostic priors, survival prompt settings, hyper-parameter settings, and training settings. More details could be found in our source code.
We derive it by asking GPT-4o (a version before 2024-07-01) the following questions one by one: i) "What visual features of H&E stained pathological images are related to the prognosis of JANCER_TYPE) exclude the features that cannot be directly observed from pathology images.", " and iii) "Please briefly describe each relevant visual feature.". Then, we further process the answers from GPT-4o by manually extracting the most relevant part of visual feature descriptions. The final results for five datasets are exhibited in Table 6. From this table, we can see that common prognostic features in WSIs are "celluar atypia", "tumor infiltration". and "tumor growth pattern".
Description of Prognostic Priors
Please list them point by point.". iii "Please
Table Descriptions prognostic priors.
Summary
Description
TOGA-BLCA 12)
Cellular stypia
Abnormiliies in the size, shaps, and onganication of tumor vells. Hlgh degrees of stypia associated with aggrossive tumoes.
Naclei atypia
Variation in and shape the nuclei within tumor cells. Procounced pleumurphism typically indicates higher grade und more aggresive cancer.
Mitotic activity
The presence and frequency cell division figures (mitoses) within High milore activity suggests rapid tumer growth and pooret prognsis.
Tumer panera
arrangement
The structural araageneent of such solid mixed pattean.
Tumer infilration
Tunor calls imaaee deep the bladder wall layers, espocially detrisor musele musculuris propria, perivesicnl beyond.
(Cottinee on next page)
21

---

Published as a conference paper at ICLR 2025
(Contimued from previous page)
Item
Brief Summary
Description
Tumor invasion
Tanoo cells within blood vessels tymphatic channels.
Necrotic tumor cells
Aress of dead (ncerotic) tuner eells within the tivene. Necrosis often appears as regions lacking viable cells and con indicate minid tamsrr growth outpocing its bload supply.
Perineural Invasion
Tumor cells survonding or imading nerves,
Tumor growth pollern
Dense fibreus er connective tissue growth around the lumor indicates an agpressivg response the tumor, often seen in high grade cancers.
10
Tumer infiltration
Tumor wells penetrate the survunnding stroma, mudll, or other tissues, lack of a clear demoreation between the tumer and the noemall tissuc, known poaty defined tumar marpins.
Formation sew blood Vessels
Formation of new blood vessels within the tumor, necessary for tumor growth and spread. High levels of angingenesis indicate more aggressive tumor.
12
Carcinoma in situ
Presence of areus of flrt, gnade non invasive cuncer, known as carcinomu simu, idemiiieed by high-grade vellubar stypis discegaizedd opilbelial structure, incressed and orynical mitotic activity, loss of normal spitiiciaal orchitecture. absence umbrella cella.
TOCA-BRCA (M
Tumor size
The dimensions of the tumor observed on the slide.
Tumor boundary
The boundary between the tumor and surrounding tissue
Tumer dificremiiaion
The diflerentiation cumor cells.
"Tunorr invasioo
Presence of tumor cells within lympharic and vessels.
1ymmph node metastisis
Visual confirmation metrstasis lympph mades, clusters cells.
Cellutas marphology
The appearance of tumor cells, including their size, shaps, and muclear features.
Necrotic tumor cells
Areas within the tumor where cells have died, typically due to insufficient blood supply
Tumor infiltration
The characterictics the tissue surrounding the tumor, such fibrosis (desmoplasia) and immune cell intileasion.
Perineural Invasion
Teave cells surounding og or invading nerves.
Puttern of growth
Differeur potterns of growth cell anargement, which vary among subtypes such dactal, lobular, and mucinous carcinoms.
and irrangement
TCGA-GBMLGG
Cellolar densiry
cellular density indicates large number slosely packed cells, suggesting rapid timor growch and aggressive behuviir.
Nuclai atypia
The abnomrol appeammne cell nucke, charectoried by variations size, stape, and staining intensity. Higher maclear alypia indicates more aggresaive tumor sells with sggifiiant genetic abmormalities
Mitutic autivity
Cells undergsing division (mitssis) be idensiied figures points to increased tumer cell proliferation and aggresäveaess.
presence mitotic figures. higher number mituicc
Necrotic tumor
cells, surouunded by viable tumor cells. Necrosis is hallmark high grade and associated with poor axygen supply and rapid tamor growth. Proudopaliaddng necresis fearures rows of tamor calls Lining the necrotic areas.
Areas
Formation of new blood vessels
The ahcormal and exeessive growth of small blood vessels within the tumor. This indicates the namor's 's ability to stamolate new blood vessel fommation (angangenesics, which supports rapid growth and spread.
Tumor infilration
Tunor cells di diffusely infilrate into normal brain tissee, idencified by seatiered nomor colls within nomnal bmin tissue, unclesr houndaries between runoc and nommal tissne, and a disnupted nornal tissue architecnie.
Tumor pitiern
arraggement
Tumor cells aranged in pulisading pullem arcund mecrotic areus. This Jeature is characteristic of Glisblasioma and indicatos aggressive tomor behavior and rapid cell turnover.
TOGA-LUAD (M 8)
Tumor differentiation
The degree of vellukar differentiation within the tumur.
Tumer size
The dimetsions of the tumor mass within the tissue section, aften measured terms of its diameter or extent.
Tumor boundayy
The uppearance tumer borders within tissue.
Nuclai features
The characteristics tumor cell muclei,
shape, and staining partern.
Mitotic autivity
The presence of actively dividing wells, idensiffed by the presence milotic figures within the tumor tissue.
Necrotic tumor cells
Areas of tissue death within the tumar mass.
Pattern of tumor growth and artangentent
The overall umangeemnnt and growth puttem of turor cells within the tissue.
Tumor invasioo
The presence of tumor cells within lymphatic or blood vessels.
TOGA-UCEC 10)
Naclei atypia
The variation size, shape, and chromotin pottem of nuclei. High-grade nuckei appear larger, inregular, darker due to increased chromatin deypeccrocassial. Prominest suckooi may also be visible.
Mitotic autivity
The number misotic figures (cells undorgoing division) per high-power field High mitoric acrivity indicatos ripid cell proliferation, which a sign of tumor aggresiivenes.
Tumor
armngement
The structural puttern the tumor, including gland formution and architectural complexity Poorly difeerentiated tumors may show weidd, criteifomn, or papilltry pattems.
pattern
Tumor invasion
The extent the tumoe's s penetration into surrounding tissues, particularly the depth of myometrial invasion. It also
includes the presence of tumor cells in lymphovasclir spaces (LVSI), indicating potensil for metastasis.
Contimued on nest page)
22

---

Published as a conference paper at ICLR 2025
(Contimued from previous page)
Item
Brief Summary
Description
Tumor infilration
The appearaace of the tumer edges, which can be infitrative (inegules, spreading into surounddnng tissue) or pushing (smooth, well defined borders). Irregular, jagged margins suggest aggressive growth.
Nearotic runoer cells
Aress of dead (scerotic) taner cells within the tisoee Necrosis often appears as regions lacking viable cells and can indicate rapid tunce growth ourpacing blead supply.
Inflammatiory cell
The presence and demsity of ioflommatory odll lymphocytes macrophages) within and around the tomor.
Tumor infilration
Changes the suppotiive tisuue (stroma) surrounding the tumor. incloding desmoplasia (dense fibous tissue respunse).
Tumer dificremiaaion
Areas within the tumor where cells show squamous (llat, scale like) characterisiies Squamous differentiation can impact the behavior and classification of the tanse.
10
Tumer beteropeneity
The variability in vellular and structural charactetsties within diffecent areas of the tumar. High heterogeneity indicates diverie oell populations, often nssociated with more aggressive behavior and resistance to treamment.
Survival Prompt Settings In time discretization, the number of discrete time bins is set by C' = vNz- Specific numbers for five datasets can be found in Table For context prompt, we initialize Vas using the text "a histopathology image suggesting". We use 4 base class prompts, i.e., B = 4. Their learnable parameters, {V>) Vid,, dl V. }, are initialized using "a very poor prognosis", "a poor prognosis", "a good prognovis", and "a very good prognoxis", respectively. Following the interpolation setting in Li et al. (2022), we set the ordering distance matrix to D ((c
1)- - 1) - 1)] and adopt the linear interpolation, i.e., WDD.,)) = C-I-
D.a
Pareer-Prrameter Settings Since we adopt CONCH in our experiments, most settings for feature dimension is the same as CONCH: D = 512 and Demb 768. We simply set the coefficient of LEDD to 3 = 1 without fine-tuning, and a 100 by default.
Training Settings In model training, we set the number of epochs to 10, learning rate to 0.0002, batch size to (one bag). the step of gradient accumulation 32, and the optimizer to Adam with a weight decay rate of 0.00001. These setting are shared across five datasets, without dataset-specific fine-tuning. Please refer to our source code for more training details. All experiments are run on a machine with two NVIDIA GeForce RTX 3090 GPUs (24G).
C.4 EVALUATION METRICS
Our main evaluation metrics contain concordance index (CI), mean absolute error (MAF), and dis- tribution calibration (D-cal). They are adopted to evaluate the performance of SA models in risk discrimination and distribution calibration. We utilize the toolkit from SurvivaEEVAL (Qi et al., 2023b) for performance evaluation. The section adapted from Qi et al. (2023a). Readers could refer to Qi et al. (2023a) for more details. We show the details of these metries below.
Concordance Index (CI) It is the most popular metric in SA (Harrell Jr et al., 1996). It is often used to assess the SA model's ability in predicting high risk for the patient who experiences the event earlier. It is ranged in [0, 1), calculated by
Eis De,e, 1 R 6 CI
(20)
Ens luke, .6
where L(-) is indication function and R, the risk prediction of the i-th individual. A larger CI suggests the model is more powerful in risk discrimination.
Mean Absolute Error (MAE) It measures the mean of the absolute error between ground truth time-to-event and the predicted event time f.. It can be calculated via
MAE(f, 6,) = 6, |t, - &l (1 - (1)) t
(21)
In this formulation, hinge loss is adopted to evaluate the MAE of censored patients. Namely, as only know the actual time-to-event of censored patient is larger than t, the MAE is 0 if the prediction & is larger than t;; otherwise, it should be t. f.. The smaller the value of MAE, the better the model performance.
'https:/ ://i thub. wtt.:ttw.b.....ooaot 'https:// :/tpss:///iii thub. com/ shi wwwssssiiaaaaaaaeeaaaaeeeaaa Survi va EVAL,
10100011000 /VLSAA
23

---

Published as a conference paper at ICLR 2025
Distribution Calibration (D-cal) (Haider al., 2020) It is statistical test to check if individual survival distribution is well calibrated with the ground truth survival distribution. Its calculation follows these steps. (1) For any probability interval (M Mb] [0, 1], first obtain the uncensored individuals whose predicted survival probabilities are in [Ma 146] at their time-to-event, i.e,
D(Ha (b) = (x, 4, 6, = 1) e D S(t) E [Pas Hol
(22)
(2) The model is well calibrated in survival distribution prediction if there is
[D(Ha; [[HaaP6)]]
jAb Ma-
(23)
Typically, equal-sized, mutually exclusive probability intervals, like the time bins calculated in time discretization (see Section 3.2), are used in these two steps (Haider et al., 2020). Then, Pearson's s x² test is applied to examine if [P(u is uniformly distributed. p 0.05 means there is significant difference between predictive survival distribution and the ground truth, implying a SA model poor in distribution calibration. A well-calibrated model tends to make safer probability prediction, instead of providing over-confident estimations.
C.5 TIME DISCRETIZATION SETTINGS
This section provides the details of time discretization settings for all compared SA models. For discrete SA models, survival time often required be transformed into discrete variable first.
Conventional Settings As stated in Appendix A. discrete SA models are usually categorized into two groups: hazard-based and incidence-based. Their conventional settings in time discretization are as follows:
Hazard-basee based SA models: Most works (Chen et al., 2021b; Xu & Chen, 2023; Zhou & Chen, 2023; Zhang et al., 2024b) follow the settings used in Zadeh & Schmid (2020) and Chen et al. (2021a), ie., using 4 time bins with quantiles as cutoff points.
Incidence-based SA models: As suggested in Haider et al. (2020), the number of time bins determined by VNe. which could often lead to better probability calibration in SA models. It is adopted in many works (Yu et al., 2011; Lee et al., 2018; 8; Qi et al., 2023a;b). Particularly, DeepHit (Lee et al., 2018), as the first deep learning- based SA model with incidence function as prediction target, sets the number of time bins to / /N, and the length each time bin to equal (i.e., time cutoff points are uniformly distributed).
Settings for Baselines and VLSA We follow conventional settings in time discretization:
For the hazard-based baseline model, Patch-GCN (Chen et al., 2021a), follow its orig- inal setting to use 4 time bins with quantiles as cutoff points. We also examine its perfor- mance under the same number of time bins VL.SA A for a fair comparison (Table 11).
For the baseline models not originally proposed for SA, we adopt the same discretization settings as our VLSA, i.e., /Ne time bins with equal length.
For our incidence-based VLSA, we use VN. time bins with equal length, stated in the main paper. Besides, we also evaluate the performance of VI.SA using different time discretization settings. Please refer to Table 10) for the results.
D ADDITIONAL EXPERIMENTAL RESULTS
D.I RISK GROUPING AND KAPLAN-MEIEE ANALYSIS
This experiment is conducted to examine whether our VLSA could discriminate the patients with high-risk and low-risk. Its results are shown in Figure 5. For each dataset, the median risk of the entire cohort is used as the cutoff for risk grouping. From the results in Figure 5, we can observe that VLSA can discriminate between high- and low-risk patients with a significant difference (P- Value 0.05 by log-rank test).
24

---

Published as a conference paper at ICLR 2025
p-Value: 4.67e-04 (?)
4.50e-06
6.66c-16 ()
3.09e-06 ()
P-Value: 8.08e-08 (?)
BRCA
GBMLGG
LUAD
UCEC
Survival rate
(Yaas))
Time (fears)
Time (Mear))
Time (Hean))
Time (Years)
Figure 5: Risk Grouping and Kaplan-Meice Analysis. All patients within each dataset are grouped into two risk groups: low-risk (blue) and high-risk (orange). Patients' risk predictions are derived from VLSA. The median risk of the entire cohort is adopted the cutoff for risk grouping.
D.2 ADDITIONAL VISUALIZATION RESULTS
Prediction Interpretation via Laaguage-Ennooee Prognostic Priors Additional results are shown in Figure 6. In most cases, descriptive texts identify the key patches that help assess cancer prognosis. Detailed descriptive texts are provided in Table
P
Pg
Pg
Pg
Pg
Texts
Texts
Teevermee taniirr
Tome of deat
Youuk
oemead dear
Representive patches
Representive patches
2.6
Contributon nok
Contributin dek
2.1
1.6
-0.4
-0.9
-0.4
Language encoded prognostic
Language-enoxdldd prognastic
Figure 6: Additional results of prediction interpretation via language-encoded prognostic pri- ors. The two examples are from the last two datasets, TCGA-LUAD (left) and TCGA-UCEC (right).
Visualization of Survival Prompts' Ordinality Additional results are shown in Figure 7. These results verify that there is often a better ordinality in the survival prompts optimized by VLSA.
w/o Ordinal Prompt
VLSA
Ordinal Prompt
VLSA
Acc 65.15%
Acc 100%
Acc 75%
Acc 89.29%
Figure 7: Additional results of visualizing survival prompts' ordinality. The visualization results are from the model trained with the last two datasets, TCGA-LUAD (a) and TCGA-UCEC (b).
D.3 ANALYSIS OF COMPUTATION EFFICIENCY
25

---

Published as a conference paper at ICLR 2025
In this experiment, evaluate the computation ef- ficiency of all models. The number of learnable pa-
Table Computation efficiency evalua- tion. A random WSI (from BLCA) with 20,265 instances is used to measure the MACs in model inference.
rameters (# Params) and Accuuply-Accumulatt oper-
ations (# MACs) are taken as main evaluation met- rics. From the results shown in Table 7, we can that VLSA not only has fewer learnable parameters (284.16K) than all baseline models except for AB- MIL but also is the most efficient one (0.25G) in in- ference cost. Our further analysis is follows.
Model
# Params
MACs
ABMIL
137.74K
2.68G
TransMII.
683,55K
14.23G
IIRA
3.16M
34.79G
(1) Our vision-end model only has two groups of learnable parameters: Try for fine-tuning textual features and ii) one linear layer for projecting image features. Such network is very simple relatively yet demonstrates strong capabilities (Table 1) in survival prediction. One critical factor behind this is that the well-aligned lision-language embeddings from foun- dational VLMs are effectively utilized to improve vi- sual representation learning.
R'T-MIL
2.44M
29.62G
DeepAttMMSL
329.22K
2.68G
Patch-GCN
363.91K
3.41G
ABMILPromp
394.24K
2.67G
CoOp
434.18K
2.68G
OrdinalCLIP
409.60K
2.68G
VLSA
284.16K
0.25G
(2) Our survival prompts can be computed advance and they are subsequently reused in model inference for different individuals. Therefore, Fresa is not involved explicitly in the process of model forward inference. This enhances the computation efficiency of individual predictions.
D.4 COMPARISON WITH HAZARD-BASED SURVIVAL MODELING
Since huzard function is also utilized in some SA models, we further try to take it as the prediction target of VLSA. Following Zadeh Schmid (2020), the settings of hazard-based survival modeling for VLSA are follows: i) survival time is discretized into 4 bins based on quantile time points; ii) we derive the prediction of individual hazard function by applying Sigmoid(-) to the scaled simi- larity scores, i.e., (T- cos( (fimage, fícx) fex))== [= and iii) the MLE loss corresponding hazard function prediction is adopted to optimize the model.
The results are shown in Table 8. From these results, observe that there decrease of 1.4% in overall CI performance. Additionally, when modeling individual hazard function, the predicted time-to-event is often largely biased. One possible reason is that hazard- based survival modeling may not be suitable for VL paradigm since it relies on Sigmoid- based probability transformation and this makes failed establish connection with conventional multi-class classification. By contrast, incidence-based survival modeling makes SA tasks similar to the classification with partial labels, as reflected by Eq. (9), so could often work better with current VI.-based prediction manner.
Table 8: Comparison with hazard- survival modeling.
TCGA
Average CI
D-cal Count
Prediction target
BLCA
BRCA
GBMLGG
LUAD
UCEC
MAE
0.5934
0.6435
0.7908
0.6089
0.7721
Hazard function
0.6817
72.46
(LL 0.018)
(_ 0.072)
(L 0.014)
(L 0.049)
(.L 0.019)
0.6176
0.6652
0.8002
0.6370
0.7571
Incidence function
0.6954
25.15
(1: 0.025)
(= 0.057)
( 0.010)
(#: 0.027)
(d: 0.045)
D.5 COMPARISON WITH MULTI-MODAL. SURVIVAL. ANALYSIS METHODS
In this experiment, compare our VLSA with multi-modal SA models. The textual description of prognostic priors are adopted in VLSA to provide extra auxiliary signals for weakly-supervised learning. An alternative approach to leveraging these textual priors is multi-modal representation learning, i.e., integrating and fusing visual and textual features. Accordingly, here we compare our proposed method with multi-modal SA models.
Four representative multi-modal SA models are adopted as the baselines for comparisons: MCAT (Chen et al., 2021b), MMPTa (Song al., 2024b), MMPor (Song et al., 2024b), and
26

---

Published as a conference paper at ICLR 2025
MОTСat (Xu & Chen, 2023). We replace their gene input with the Language-encodee prognostic priors, i.e. Euxx(Tpog) used VLSA. In addition to this settings, the other settings and model implementations keep the same as those in their official codes.
The results of this experiment are shown in Table 9. These results show that our VLSA could often perform better than the compared multi-modal SA models, with an improvement of 2.85% over the runner-up in average CI. This improvement could indicate the superiority of our scheme.
Table 9: Comparison with Multi-Modal Survival Analysis Models. These methods are origi- nally proposed for learning multi-modal representations from WSIs and gene data. We replace their gene input with the language-encoded prognostic priors Ejex prgy) for comparisons.
TCGA
Average CI
D-cal Count
Method
BLCA
BRCA
GBMLGG
LUAD
UCEC
MAE
MCAT
0.5899
0.6322
0.7980
0.5768
0.6827
0.6559
31.44
3
(# 0.020)
0.035)
(# 0.020)
(# 0.043)
0.022)
0,5857
0.5944
0.7917
0.5654
0.6928
0.6460
31.08
(+ 0.040)
(+ 0.016)
(+ 0.012)
(+ 0.055)
(# 0.049)
MMP
0.5961
0.6315
0..9448
0.5870
0.6915
0.6602
31.35
(# 0.031)
( 0.041)
(#: 0.019)
(# 0.071)
(= 0.052)
MOTCat
0.6227
0.6301
0.7967
0.5617
0.7234
0.6669
29.73
3
(# 0.029)
0.047)
(#: 0.026)
(+ 0.070)
0.033)
0.6176
0.6652
0,8002
0.6370
0.7571
VLSA
0.6954
25.15
(#: 0.025)
(+ 0.057)
(#: 0.010)
(# 0.027)
(= 0.045)
D.6 DIFFERENT TIME DISCRETIZATION SETTINGS
Different Time Discretization Settings for VLSA Here we examine the impact of time discretiza- tion on model performance. Specifically, we try to cut survival time into 4 bins using quantile time points. The results are shown in Table 10. We can see that quantile time points or 4 time bins could often lead to poor calibration in predictive survival distribution. poorly-calibrated distribu- tion often indicates over-confident survival predictions. Thus, we follow the setting of Haider et al. (2020) in time discretization, i.e., uniformly discretize time into /Ne bins.
Table 10: Different time discretization settings for VLSA. N. is the number of patients with event. Time cutoff
TCGA GBMLGG
Average CI
D-cal Count
#Cuts
BLCA
BRCA
LUAD
UCEC
MAE
Quantile
0.6238
0.6483
0.7962
0.6240
0.7295
0.6844
24.40
(#: 0.036)
(= 0.060)
(#: 0.014)
(# 0.028)
(#: 0.057)
0.6255
0.6567
0.8006
0.6150
0.7300
Quantile
VN.
0.6856
21.92
2
0.015)
0.064)
(+
0.031)
(+ 0.038)
0.6124
0.6458
0.7907
0.6312
0.7674
Uniform
0.6895
23.25
1
(1: 0.020)
(# 0.056)
(# 0.010)
(1: 0.038)
(d: 0.041)
0.6176
0.6652
0,8002
0.6370
0.7571
Uniform
V
0.6954
25.15
5
(1: 0.025)
( 0.057)
(LL 0.010)
(1 0.027)
(L 0.045)
Comparison with Patch-GCN under the Same Number of Time Bins We the number of time bins for Patch-GCN and VLSA to further compare their performance. The results are shown in Table 11. From these results, we can see that our VLSA could often outperform Patch-GCN in terms of average CI and MAE even when the number of time bins is set to the same value.
D.7 ADOPTING PLIP AS VLM FOR SURVIVAL ANALYSIS
As another foundational VLM in computational pathology, PLIP (Huang et al., 2023) is pretrained on the pathology image-text pairs crawled from Twitter. It is the first work exploring VLMs for pathology, developed before CONCH (Lu et al., 2024). We further adopt for VL-based methods in this experiment.
27

---

Published as a conference paper at ICLR 2025
Table 11: Comparison with Patch-GCN under the same number of time bins.
TCGA
Average
D-cal
#Cuts
Method
BLCAA
BRCA
GBML.GG
LUAD
UCFC
CI
MAE
Count
0.6124
0.6375
0.7999
0.5922
0.7212
Patch-GCN
0.6726
26.70
2
(+ 0.03
(+ 0.033)
(= 0.021)
(+ 0.053)
(+ 0.025)
0.6124
0.6458
0,7907
0.6312
0.7674
VLSA
0.6895
23.25
1
(+ 0.020)
(= 0.056)
(= 0.010)
(# 0.038)
(#: 0.041)
0.6054
0.6300
0,7890
0,5912
0.6967
Patch-GCCN
0.6625
28.65
5
(1 0.049)
(1 0.068)
(= 0.020)
(1: 0.029)
(1 0.030)
0.6176
0.6652
0,8002
0.6370
0.7571
VLSA
0.6954
25.15
5
(+ 0.025)
(+ 0.057)
0.010)
0.027)
0.045)
The results of this experiment are shown in Table 12. From these results, we can find that, compared with CONCH, VL-based methods with PLIP often suffer from large decreases in overall perfor- mance. In particular, the zero-shot method MI-Zero makes survival predictions like random guessing (CI 0.519). The other three models with supervised training perform almost identically. One main reason for this could be that CONCH provides better VI.-aligned latent space than PLII since it is pretrained on high-quality data at a larger scale (Lu et al., 2024).
We note that VL-base methods stand on the shoulder foundational VLMs and the capability of foundational VLMs could largely determine their performance bounds. Our VLSA is no exception. When vision and language embeddings are well-aligned, language-encode prognostic priors can effectively guide instance aggregation and improve prognostic visual representation learning.
Table 12: Adopting PLIP as VLM for survival analysis.
Method
VLM
TCGA GBMLGG
Average
D-cal Count
BLCA
BRCA
LUAD
UCEC
CI
MAE
0.4990
0.5048
0.5374
0.4681
0,5855
PLII
(= 0.035)
(# 0.056)
(+: 0.048)
0.081)
(4 0.059)
0.5190
27.22
4
MI-Zerosurv'
0,5541
0.5788
0.3842
0.5209
0,6623
CONCH
0.5400
25.63
0
(- 0.034)
(+ 0.028)
(+ 0.063)
(+ 0.049)
(+ 0.059)
0.5753
0.5458
0.7929
0,5838
0.6841
PLIP
0.6364
29.69
5
CoOp
(- 0.022)
(+ 0.026)
(+ 0.031)
(+ 0.014)
(+ 0.064)
0,5971
0.5994
0.7853
0,5750
0.6840
CONCH
0.6482
28.70
5
(= 0.033)
(# 0,086)
(#: 0.015)
(#: 0.064)
(# 0.070)
0.5707
0.5467
0.7913
0,5648
0.6850
PLIP
0.6317
29.62
5
OrdinalCLIP
(= 0.037)
(# 0.038)
(1: 0.035)
(#: 0.036)
(1: 0.053)
0.6037
0.6202
0.7893
0.6053
0.6836
CONCH
0.6604
28.01
5
(= 0.043)
(# 0.046)
(#: 0.018)
(#: 0.065)
(#: 0.036)
PLIP
0.5780
0.5469
0.7626
0.5657
0.7038
0.6314
27.41
5
(= 0.048)
(#: 0.034)
(:: 0.023)
(#: 0.013)
(d: 0.029)
VLSA
0.6176
0.6652
0,8002
0.6370
0.7571
CONCH
0.6954
25.15
5
(= 0.025)
(# 0.057)
(1: 0.010)
0.027)
(1: 0.045)
E LIMITATION AND FUTURE WORK
Although promising results are obtained in this study, we note that there are still some limitations in this study. (1) The number of datasets and the diversity of cancer types are limited. Due to the extremely-high resolution of histopathology WSIs, it is usually challenging to collect tens of thousands of images and preprocess them into feasible training data.
(2) The textual descriptions about cancer-specific prognostic prior are derived from GPT-4o. They may be lacking in knowledge completeness and accuracy. More holistic and accurate prognostic prior knowledge are likely to provide stronger guidance for weakkl--uuerrvied MII. and perform better in survival prediction. (3) The foundational VLM that our empirical results rely on CONCH (Lu et al., 2024), a recent state- of-the-art VLM for CPATH. More pathology VLMs may be needed to further validate our VLSA. (4) Many -fate-of-the-an approaches for WSI classification (Zhang et al., 2022: Xiong et al., 2023; Tang et al., 2023; Qu et al., 2023; Shi et al., 2024; Fourkioti al., 2024; Tang et al., 2024) have shown strong capabilities in representation learning. VLSA could take inspiration from or combine with them further improve its performance in SA tasks.
28

---

Published as a conference paper at ICLR 2025
We note that VL-base approaches stand on the shoulder of foundational VLMs and their perfor- mances are closely correlated with the capability of VLMs. Our VLSA is no exception. We strongly believe that VLSA could further refresh the state-of-the-arr performances in SA tasks with the tinual advancements of VLMss. In the future, we will continue to follow the frontier of VLMs in CPATH and leverage advanced tools to study the potential of VLSA in cancer prognosis.
29


Reviews:
[{"review_id": "OzBwg4fP7z", "reviewer": "Reviewer_8wxk", "summary": "This paper introduces a Vision-Language-based Survival Analysis (VLSA) paradigm to improve cancer prognosis prediction from histopathology whole-slide images by leveraging pathology foundation models for enhanced data efficiency and weakly-supervised learning. VLSA incorporates prognostic language priors and ordinal survival prompts to guide visual feature aggregation. The authors have conducted multiple experiments and ablation studies to evaluate the method proposed.", "questions": "mentioned in weaknesses.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Overall, the paper is well-written and tries to address an important challenge in digital pathology. The proposed solution seems to also outperform the reported benchmark in the paper.", "weaknesses": "Major Comments:\n1. The introduction to the paper highlights data scarcity for survival analysis as a major problem that this paper claims to solve. However, it seems like this paper addresses data efficiency in terms of model size and MACs, not the limited amount of training data available for survival analysis. \n2. The number of bins being set at sqrt(N events) is given little justification. It also calls into question the comparison with other models (eg. PatchGCN used 4 bins for all datasets in the original paper). Additionally in the Appendix “Comparison with Hazard-Based Survival Modeling” 4 bins were used as opposed to the number used in the original experiments. The additional experiments in Appendix E don’t provide sufficient justification for using uniform sqrt(N events).\n3. PatchGCN was trained on 4 bins in the original paper. The VLSA model should be compared with the baselines with the different bin sizes being considered for a fair comparison.\n4. Given the variability of survival evaluation metrics (eg c-index) and the fairly marginal improvement over the baselines the authors should do a deeper analysis to showcase the improvements of their model. This could be done by plotting the survival curves of “high-risk” vs “low-risk” cases.\n5. In Appendix E, there appear to be patches comprising of mostly white-space. Was any preprocessing done to filter out patches like this, or patches with significant blur or artifacts. \n\nMinor Comments:\nSpelling error at line 44 “While existing approaches have made exciting progress…”\nIn Appendix Table 6, there seems to be little point in labeling the maximum values.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "s71WKh4k0E", "reviewer": "Reviewer_f7nn", "summary": "This paper aims to overcome two challenges in survival analysis including overfitting due to data scarcity in CPATH, and insufficient supervision for survival analysis. To address the challenge of scarce training data, the authors proposed to apply visual-language models to survival analysis by utilizing the power of the recent VL pathology foundation model. To compensate for supervision signals, the authors introduced some extra prior descriptions of prognosis by querying GPT-4o. Some ordinal prompts are designed to adapt VL models to survival analysis, although their performance improvements are marginal.", "questions": "1. I‘m not sure how the authors applied MI-Zero to survival analysis, which was supposed to be designed for classification.\n2. What’s the difference in prompts between OrdinalCLIP and Ordinal Survival Prompt?\n3. In equations 2 and 3, how can the model ensure learning multi-level representations? It seems there is no constraint to achieving this goal.\n4. CONCH is a large foundation model pretrained on a great amount of data including public data. In this case, to avoid data contamination, how can the author ensure test samples never occur in the pretraining materials of CONCH in the few-shot setting?\n5. I’m confused about what the specific events are referring in these experiments. If overall survival, there should be only one event. However, the authors mentioned that the events can occur multiple times. Is it longitudinal data?\n6. The motivation for using EMD should be elaborated. For example, why is EMD aware of ordinality?\n7. It is difficult to understand Figure 3(a) is trying to explain. What do the horizontal and vertical axes represent, respectively?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. This is the first work to adapt the VL foundation model to survival tasks by designing ordinal prompts tailored to survival analysis tasks.\n2. The introduction of prognostic prior description was empirically validated to significantly improve the performance of survival analysis.\n3. The motivation is basically clear although the design of the ordinal incidence function is a bit confusing.\n4. Interpretation brought by language priors for survival tasks is interesting and makes the prediction more transparent.", "weaknesses": "1. The motivation of ordinal IF is confusing. The authors assume there is a consecutive decline in probability for those classes away from the time when the event occurs, which raises doubts about its reasonableness when events occur multiple times. This assumption is only concerned with the event that first occurs. \n2. Ordinal Survival Prompt seems to ignore the status of event indicators (i.e. censorship) when designing the prompts. For different censorships, there should be different outcomes reflected in the survival prompts even when their follow-up time are the same. When the case is censored, the only thing we can know is the event didn’t occur before the follow-up time. For example, we cannot simply assign a case with a short follow-up time into a bin of “very poor prognosis”.\n3. Experiment comparison is **NOT** adequate in the following aspects:\n\n    a. To validate the effectiveness of prompts tailored to survival tasks, a stronger baseline equipped with naive prompts that simply formulate survival tasks as classification tasks, should be compared.\n\n    b. **To support the authors’ claim that \"highly-expressive modern networks may cause overfitting when facing present small WSI data”**, the current “highly-expressive” models should be compared with the proposed method. Furthermore, the extra text description generated by GPT-4o should be an additional modality. Therefore, some representative multimodal survival models should be compared, **which can validate the efficacy of utilizing the pathology foundation model**. I understand there have been a few models that integrated WSI and text data in the past. However, by replacing the gene data with text data, some multimodal survival models using WSI and gene data can be adapted for comparison, e.g. MCAT [1], MOTCat [2], MMP [3] etc.\n\n    c. Unimodal MIL models are a bit out-of-date. More “highly-expressive” SOTA methods should be compared, such as CAMIL[4], R2T [5], etc.\n\n4. From the results of Tables 3 and 8, performance gains from two ordinal prompts are marginal or even getting worse. The reason should be justified.\n5. Comparisons in Few-shot learning may not be fair. The capability of few-shot prediction greatly relies on how the visual and text spaces are well-aligned. CONCH is a pathology VL model that aligns visual and text spaces well. However, CoOp and OrdinalCLIP are equipped with ABMIL, which is not aligned with the text space, for few-shot prediction by using such few samples. For a fair comparison, the author should replace prompts with the ones in CoOp and OrdinalCLIP only based on CONCH to validate the efficacy of survival prompts.\n6. The code is unavailable which may affect the reproduction.\n\nIf my concerns could be addressed well, I would adjust my scores.\n\n[1] Multimodal co-attention transformer for survival prediction in gigapixel whole slide images, ICCV, 2021\n\n[2] Multimodal optimal transport-based co-attention transformer with global structure consistency for survival prediction, ICCV, 2023\n\n[3] Multimodal Prototyping for cancer survival prediction, ICML, 2024\n\n[4] CAMIL: Context-Aware Multiple Instance Learning for Cancer Detection and Subtyping in Whole Slide Images, ICLR, 2024\n\n[5] Feature re-embedding: Towards foundation model-level performance in computational pathology, CVPR, 2024", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yViMEtVzwG", "reviewer": "Reviewer_q9C6", "summary": "The paper proposes a Vision-Language-Based survival analysis (VLSA) paradigm, addressing two key challenges: data scarcity and weakly supervision of gigapixel whole slide image (WSI) analysis. The framework leverages foundational vision-language models with three key innovations: (1) language-encoded prognostic priors to guide instance aggregation, (2) ordinal survival prompt learning for encoding continuous time-to-event labels, and (3) ordinal incidence function prediction. The approach demonstrates strong performance improvements while maintaining interpretability through Shapley values.", "questions": "1. The paper focuses on vision-language models while foundational vision models like UNI [1] have shown strong performance in computational pathology. Could the authors discuss their rationale for prioritizing vision-language models and perhaps provide comparative experiments with vision-only foundation models to better justify this architectural choice?\n2. Given the recent success of multimodal approaches in survival analysis (e.g., MCAT [2], MOTCAT [3], CMTA [4], MoME [5]), could the authors address why these methods were not included in the baseline comparisons? A comparison with these state-of-the-art multimodal methods would help better contextualize VLSA's contributions to the field.\n3. To strengthen the clinical relevance of the results, could the authors provide Kaplan-Meier survival curves for risk stratification groups, along with appropriate statistical significance tests (e.g., log-rank test)? This would help validate the model's practical utility in clinical settings.\n4. Regarding the experimental comparisons within the VLM group, the baseline models (CoOp and OrdinalCLIP) were originally trained on natural images rather than pathology data. Could the authors discuss how this might affect the fairness of the comparisons and perhaps provide additional experiments with VLMs specifically trained or fine-tuned on pathological images?\n\n[1] Towards a General-Purpose Foundation Model for Computational Pathology\n\n[2] Multimodal Co-Attention Transformer for Survival Prediction in Gigapixel Whole Slide Images\n\n[3] Multimodal Optimal Transport-based Co-Attention Transformer with Global Structure Consistency for Survival Prediction\n\n[4] Cross-Modal Translation and Alignment for Survival Analysis\n\n[5] Mixture of Multimodal Experts for Cancer Survival Prediction", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1.\tThe authors have proposed a complete framework for VLM-compatible survival analysis, with novel components for handling continuous time labels and maintaining ordinal relationships\n2.\tThe comprehensive experiments demonstrate the efficacy of the method and each proposed module through thorough ablation studies\n3.\tThe method incorporates interpretability using Shapley values, which is crucial for clinical applications, and provides explanations through natural language descriptions", "weaknesses": "1.\tThe fundamental necessity of language priors for addressing data scarcity requires stronger justification. While the authors argue that language priors can mitigate data scarcity, they do not sufficiently demonstrate why this approach is superior to using large-scale vision-only foundation models (e.g., UNI). A direct comparison with vision-only approaches would strengthen this argument\n2.\tThe experimental validation has some limitations in terms of baseline comparisons. The pre-trained VLMs used for comparison (CoOp, OrdinalCLIP) are trained on natural images rather than pathology data, leading to a suboptimal performance\n3.\tThe survival analysis evaluation omits standard clinical protocols, notably Kaplan-Meier analysis and statistical significance testing between risk stratification groups. It would be better to see these experimental results and analysis\n4.\tThe comparative analysis excludes several state-of-the-art methods in WSI classification, including DTFD-MIL [1], HAG-MIL [2], MHIM-MIL [3], and recent VLM approaches such as TOP [4] and ViLa-MIL [5]\n\nMiscellaneous:\n1.\tThe terminology regarding \"cross-attention\" in line 163 requires revision, as similarity matching differs from the so-called cross-attention mechanisms\n2.\tFigure 1 contains rendering issues in “context prompt” that should be addressed\n\n\n[1] DTFD-MIL: Double-Tier Feature Distillation Multiple Instance Learning for Histopathology Whole Slide Image Classification\n\n[2] Diagnose Like a Pathologist: Transformer-Enabled Hierarchical Attention-Guided Multiple Instance Learning for Whole Slide Image Classification\n\n[3] Multiple Instance Learning Framework with Masked Hard Instance Mining for Whole Slide Image Classification\n\n[4] The Rise of AI Language Pathologists: Exploring Two-level Prompt Learning for Few-shot Weakly-supervised Whole Slide Image Classification\n\n[5] Dual-scale Vision-Language Multiple Instance Learning for Whole Slide Image Classification", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)