Gradient-based Explanations for Deep Learning Survival Models
Authors: Sophie Hanna Langbein
Niklas Koenen
Marvin N. Wright
Venue: ICML 2025 poster (2025)
Keywords: Deep Learning
Survival Analysis
Explainable Artificial Intelligence
Interpretable Machine Learning
XAI
IML
Feature Attribution
Primary area: applications->health_medicine
TL;DR: We introduce gradient-based explanation methods for survival neural networks, offering improved interpretability, time-dependent insights, and faster, more accurate alternatives for analyzing medical and multi-modal data.

Abstract: Deep learning survival models often outperform classical methods in time-to-event predictions, particularly in personalized medicine, but their "black box" nature hinders broader adoption. We propose a framework for gradient-based explanation methods tailored to survival neural networks, extending their use beyond regression and classification. We analyze the implications of their theoretical assumptions for time-dependent explanations in the survival setting and propose effective visualizations incorporating the temporal dimension. Experiments on synthetic data show that gradient-based methods capture the magnitude and direction of local and global feature effects, including time dependencies. We introduce GradSHAP(t), a gradient-based counterpart to SurvSHAP(t), which outperforms SurvSHAP(t) and SurvLIME in a computational speed vs. accuracy trade-off. Finally, we apply these methods to medical data with multi-modal inputs, revealing relevant tabular features and visual patterns, as well as their temporal dynamics.

Gradient-based Explanations for Deep Learning Survival Models
Sophie Hanna Langbein 12
Niklas Koenen 12 Marvin N. Wright 123
Abstract
Deep learning survival models often outperform classical methods in time-to-event predictions, particularly in personalized medicine, but their "black box" nature hinders broader adoption. We propose framework for gradient-based explana- tion methods tailored to survival neural networks, extending their use beyond regression and clas- sification. We analyze the implications of their theoretical assumptions for time-dependent ex- planations in the survival setting and propose ef- fective visualizations incorporating the temporal dimension. Experiments on synthetic data show that gradient-based methods capture the magni- tude and direction of local and global feature effects, including time dependencies. We intro- duce GradSHAP()), gradient- -aaed counterpart to SurvSHAP(I), which outperforms SurvSHAP(1) and SurVLIME in computational speed vs. accu- racy trade-off. Finally, we apply these methods to medical data with multi-modal inputs, revealing relevant tabular features and visual patterns, as well their temporal dynamics.
OZORRI
prodices land XMI oedich wi dvey lurning
medirmalllaa whw.maallaiaa... indlaing
SCWI
Elkiie teaakatteenceaai
GraRSHAPO
coeercerpeeske stile
Fouteun Fiftors
Cherpenite
Surved
Tabutar
imago
Bizne
Peciikiity
Data
Modd Transs
Dayroowugg
Figure I. Overview of our workflow for generating time- dependent post-hoc explanations using gradient-based methods by the example of overall brain cancer survival prediction. The approach utilizes survival deep learning model with multi-modal input data, providing insights into the temporal dynamics of feature effects through tailored visualizations for different feature types. (images: Flaticon.com)
cal research and evidence- basedd medicine. Deep learning methods hold significant potential for advancing survival analysis by framing pathogenic identification as data- driven problem, uncovering correlations between patient profiles and disease phenotypes, and seamlessly learning from unstructured or high- nigh-dimensional data such as images, text, or omics, thereby revealing hidden and complex pat- terns undetectable by classical approaches (Zhang et al., 2019). While machine learning models show great promise for survival analysis, their inherent opacity raises legitimate concerns, in fields like life sciences, interpretability is crucial to support sensitive decision-making mitigate biases, promote equity, and ensure compliance with regulatory stan- dards (Rahman & Purushotham, 2022: Vellido, 2020). In recent years, several post-hoe eXplainable AI (XAI) meth- ods for population-
1. Introduction
As medical databases expand to include patients' detailed medical history and genetic information, healthcare is shift- ing from population-based models and traditional statistical approaches targeting the "average" patient to more complex personalized medicine, which tailors diagnoses to individ- ual patient characteristics. Survival analysis is fundamental to medical data analysis, modeling time-to-event outcomes while accounting for censoring, enabling personalized risk predictions, and assessing treatment effects to advance clini-
(global) and individual (local) in- sights into the ppeissionmmkinn process of machine learning survival models have been proposed (Langbein et al., 2024). For personalized medicine, the local model-agnostic ap- proaches SurvLIME (Kovalev et al., 2020), extension of LIME (Ribeiro et al., 2016), and SurvSHAP(O) (Krzyzinski et al., 2023), a generalization of SHapley Additive exPla- nations (SHAP) (Lundberg & Lee, 2017), have prevailed. However, no methods specifically targeted at or practical to
'quual contribation 'Leibniz Institute for Prevention Research and Epidemiology - BIPS, Bremen, Germany 'Faculty of Math- ematics and Computer Science, University of Bremen, Germany 'Department of Public Health, University of Copenhagen. Den- mark. Correspondence to: Marvin N. Wright <wright@leibniiz bips.de>.
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Gradient-based Explanations for Deep Learning Survival Models
survival deep learning models have been introduced.
censored setting, in which the data consist of n triplets, i.e..
D (xx1) y(0) 860) -- The first component, xx(1) (a (0) 22 ................................ x0)) X, represents the p-dimensional vec- tor of predictive features for individual i, which may in- clude data types commonly used in classical supervised learning, such as images or tabular data. The observed right-censored time y is defined as the minimum of the event time z( I and the censoring time c(t) E R i.c., y( min cl The binary event indicator 869) E {0, 1} takes the value if the observation is cen- sored (t( > c( and 1 if the event occurs (l(9) < c())), For clarity, we will omit the superscript for an instance in the remaining text when it is not necessary.
In this paper, introduce formal framework general- izing gradient-based explanation methods for individual predictions (i.c., local) to survival neural networks (NNs), extending their applicability to functional form outcomes beyond traditional regression and classification tasks, and showcasing their applicability in multi-modal patient-level survival prediction.
Contributions. We extend a set of six representative gradient-based explanation methods (c.g., Saliency, Integrat- edGradient, GradSHAP, and G Input) to time-to- event survival analysis, addressing a crucial gap in survival XAI research. Our main contributions are:
2.2. Survival Distribution Representations
(1) We adapt and systematically assess gradient-based XAI methods in the context of time-to-event func- tional outcomes, analyzing both their theoretical as- sumptions and practical challenges, e.g., for the gradi- ent calculation in deep survival NN models.
Key quantities to be modeled and predicted in survival anal- ysis are distributional representations of the random variable T on the time domain T, typically a subset of RØ. We gen- eralize them as f x T > I, a set of functions that map value combinations from the feature space X and the time space T one-dimensional outcome. The most popular representations are survival S, hazard h, and the cumulative hazard function H.
(2) We develop visualization and interpretation tech- niques for functional outputs tailored to the different gradient- gradient-based explanation methods. In doing so, we contribute to the ongoing debate and disagreement re- garding gradient-based methods (Sturmfels et al., 2020; Krishna al., 2024: Koenen Wright, 2024b) by clarifying how implicit or explicit baselines in these methods influence survival explanations.
Definition 2.1 (Survival Function). The survival function S X' x T - [0, 1] describes the probability of the time-to- event (survival time) being greater than or equal to specific time point t 2 conditional on the observed features x E X
(3) We introduce GradSHAP()), a gradient-buat based. model- specific counterpart SurvSHAP() for SHAP-like explanations. A quantitative comparison confirms that the gradient-based approach outperforms the sample- based version and SurvLIME, offering a better balance between computational speed and local accuracy.
S(t|#) = PT 2 t/x) = 1 P(T t|z))
(I)
Definition 2.2 (Hazard Function). The hazard or risk func- tion h X x T IR describes the instantancous risk of occurrence of the event of interest in an infinitesimally small time interval it, t + Af] for continuous t E R given that it has not yet occurred before time and conditional on the observed features x X
(4) Using multi-modal real-world brain cancer sur- vival example with tabular and histopathological im- age inputs, we demonstrate that the methods can feasi- bly identify prediction-relevant features including their temporal dynamies and visual patterns (see Fig. 1). (5) All methods and visualization tools are imple- mented in our open-source R package survi nng', which supports corshe survival models from survival wetivviammddee 1mode 1s (Sonabend, 2024) and Py Torch models trained in pycox (Kvamme et al., 2019).
P(t T St+ A&T 2 f.
h(t[x)) = lim A/-50
(2)
A
d log S(t
dt
Definition 2.3 (Cumulative Hazard Function). The cumu- lative hazard function H X x T TR.t describes the accumulated risk of experiencing the event of interest up to specific time t E R conditional on the observed features x E X
2. Background
H(1))) = f h(ulx)dd = log(S(t)
2.1. Survival Data
(4)
We are considering survival analysis as supervised pre- diction task for the time-to-event distribution of a given dataset D. We limit ourselves to the standard right-
2.3. Related Work
Deep learning survival models often extend the Cox regres- sion framework by using NNs to parameterize the log-risk
2

---

Gradient-based Explanations for Deep Learning Survival Models
function, optimizing a Cox-based loss, i.e., the negative log- partial likelihood of the Cox model (Cox, 1972). Examples include DeepSurv (Katzman et al., 2018), which employs feedforward NNs to capture non-linear feature-hazan rela- tionships while adhering to the proportional hazards (PH) assumption. It states that the hazard ratio between any two individuals remains constant over time, meaning the effect of features on the hazard function is multiplicative and does not vary with time. CoxTime (Kvamme al., 2019) incorpo- rates time-dependent feature effects by including time as additional feature. Another category of survival NNs adapts discrete- time methods, treating time as discrete to leverage classification techniques. The most prominent model in this category is DeepHit (Lee et al., 2018), which directly mod- els the joint distribution of survival times and event prob- abilities without assumptions about the stochastic process, allowing dynamic feature-risk relationships. Additional ap- proaches include methods based on srccewise-expoeentii models, ordinary differential equations, and ranking tech- niques. For a comprehensive review, see Wiegrebe et al. (2024). To date, the interpretability of survival NNs has primarily been addressed through model-agnostic, post-hoe methods. Prominent local XAI techniques in this domain include SurVLIME and SurvSHAP(t). For comprehensive review of interpretabiliiyy methods in survival analysis, we refer to Langbein et al. (2024). So far, application-focute focused studies have employed existing model-specific XAI meth- ods, such as simple gradients, to analyze deep learning survival models- - primarily to identify important nodes or generate saliency maps for singular images (Mobadersany al., 2018; Hao al., 2019; Cho et al., 2023). However, to the best of our knowledge, no study has explicitly extended these methods to general time time-dependent explainability ap- proaches for survival NNs.
depend its architecture; while many survival DL models use feedforward NNs, other architectures such convolu- tional NNs (CNNs), recurrent NNs, generative adversarial networks, and autoencoders are also employed (Wiegrebe et al., 2024). Below, we discuss the relevant feature modali- ties and their coverage in this work.
(1) Time dependence: Time dependence can be incorpo- rated in two ways: 1) time-varying effects of time-constant features (TVE) or 2) time- mareivayingg features (TVF). The PH assumption simplifies the feature effect on the hazard scale to a one- one-dimensional setting f X'' - y similar to standard regression or classification. However, attribution values are generally not time-constant, even in PH models, when eval- uated on survival or cumulative hazard scales, necessitating time-dependent feature attribution computation and visual- ization. TVF in time-to-event prediction are analogous to longitudinal input data and require specialized architectures, such as recurrent NNs, along with tailored interpretability methods (Ferreira et al., 2021).
(2) Data modalities: Mixed tabular data are one of the most common input types for survival analysis and many sur- vival deep learning methods have been (first) developed for it (Katzman et al., 2018; Kvamme et al., 2019; Lee et al., 2018). High-dimensional (multi-)omics data are another popular input type for survival NNs, with many specialized NNs developed for this purpose (Ching et al., 2018; 8; Hao et al., 2018; 2019). Conceptually, feature attribution tech- niques apply similarly to high- and oiw-dimesiional inputs, as feature contributions are considered individually. Some omics-specific NNs assign biological meaning to network nodes, enabling feature attributions to quantify the impact of biologically sigmificant quantities. Another key data modal- ity for survival NNs is (often medical) image data, typically processed using convolutional NN architectures (Zhu et al., 2016; Mobadersany et al., 2018: 11; Tang et al., 2019). Since many gradient-based feature attribution methods were origi- nally designed for image data, their adaptation to survival models is straightforward for single outcome time point, where traditional saliency maps (Simonyan et al., 2014) can be generated. Saliency across multiple time points can be represented using different maps, colors or visual markers in a single map. Other input formats, such as text data, are less frequently used for time-to-event predictions and are not covered in this work. In our experiments, we consider DeepSurv, Cox and DeepHit as a representational set of deep survival models.
3. Taxonomy of Deep Survival Models
For the application of gradient-based feature attribution tech- niques, we categorize survival NNs based on two criteria: (1) supported input feature modalities and (2) their pre- diction outcomes.
Prediction outcome. In survival NNs, the prediction out- come refers to the network's output, f : X' x T + y. These outcomes correspond to the distributional representations discussed in Sec. 2.2 as, e.g., the hazard function or survival function. Differentiating based on prediction outcome is critical, as it constitutes the quantities decomposed during attribution and capturing the effects of changes in input features.
4. Gradient-based Survival Explanations
Gradient- feature attribution methods are a set of local model-specific XAI techniques that assign relevance scores to input features based on their contribution to the prediction of a NN. These methods efficiently leverage the automatic
Input feature modalities. Differentiation by feature modality is essential, as it affects how attribution values are visualized. The input modalities a network can process
3

---

Gradient-based Explanations for Deep Learning Survival Models
Output sensitivity
Attribution by Decomposition
2r( dar,
Goal: f(t a)
Goal: f(t a) - f(t ()
Goal: f(t a))
Ez [f(s)
[)]
Grad(U::
GXI(I
0f(1)))) -2
IntGrad(t)
GradSHAP(D
SGG): Fx Oz,
SG
dx,
1f + De, 4.
(Ft Rd)
0fte afx #1)
Bfdd + ofx 83)
8r.
Figure 2. Mathematical representations of gradient-based feature attribution methods adapted to survival NNs. Each block corresponds a different underlying objective. For example, in the case of feature- wise relevances Ri obtained from GxI(t,, the goal is to achieve a sum that equals (f)æ), i.c.. Σ/ R3 f(1)z)).
differentiation capabilities inherent modern deep learning libraries (Chollet et al., 2015; Paszke al., 2019) to quan- tify how each input feature influences the model's output (Ancona et al., 2019; Koenen & Wright, 2024a). In their original formulation, these methods are defined for scalar outputs resulting in an attribution value Rg for each feature j. However, in the survival context, the outcome is repre- sented as a prediction vector for different time points, which complicates the computation and adds an additional dimen- sion to the explanations. Instead of being applied a single f(x) R. survival XAI method is applied at each dis- cretized time point (folaz)).... [(rr|ar (ær), resulting in an at- tribution value R²² for each feature j and time point fgg thus, an ensemble of explanations including their temporal inter- play. Computationally, a straightforward implementation not always feasible. For example, replicating a single in- stance across multiple time points, as required by Cox violates the assumption of independence between samples and leads to unintended gradient accumulation. Therefore, in the following, we extend the most common gradient- based feature attribution methods to survival networks and thereby propose their time-dependent counterparts Grad(t), SG(1), Gx)(), IntGrad(() and GradSHAP(1 AP)); see Fig. 2 for an overview of the proposed methods.
point i, as shown in Fig. 2. This captures how sensitive the prediction is to changes in each feature and, over all time points, how the relevance evolves over time.
SG(t). Smilkov et al. (2017) propose smoothed gradients (SmoothGrad) as an extension to reduce noise in the raw gradients. Equivalently, in SG((), the expected values of the gradients are computed over random Gaussian perturbations of the input, & (N0 o0) (see Fig. 2). In practice, the expected value of the gradients is estimated as an average over K samples, so that the accuracy of the estimation can be improved by larger values of K. The Gaussian standard deviation a controls the sharpness of the explanation and is mostly indirectly specified through a noise level o = Zmaz -min determining the proportion of the total range of the domain that covered by the standard deviation o. domain
Note, that in the survival setting, we can also perturb inputs over different time points (e.g.. for Cox Time) to capture the temporal dynamics of feature effects.
4.2. Attribution-base Methods
Feature attribution methods typically aim to approximate a decomposition of some prediction- based quantity of a model into feature-wise additive contributions (Ancona et al., 2019; Shrikumar et al., 2017; Sundararajan et al., 2017). The quan- tity depends on the method and is often the model's output or a baseline-adjusted version. This property is commonly referred to local accuracy.
4.1. Output-Sensitivity Methods
Although they do not represent attributions in the classical sense", output-sensitiv methods are often categorized as feature attribution. As pointed out in Koenen & Wright (2024b), methods in this category do not produce actual attributions but rather local importance tendencies.
GxI(t). A simple and computationally efficient approach for decomposing a prediction is to multiply the gradient by the corresponding feature value, as proposed by Shriku- mar et al. (2017) known as Gradient Input. Our survival adaption is denoted as GxI(t) in Fig. 2. Mathematically, this method is based on a first-order Taylor expansion at the implicitly set reference value zero, effectively providing linear approximation of the model's output (Ancona et al., 2019; Montavon et al., 2017). However, since survival func- tions are inherently highly non-linear, this approach often leads limitations in the accuracy of the decomposition.
Gtad(t). The Gradient method, also known as Vanilla Gradient or Saliency Maps in the image domain, devel- oped by Simonyan et al. (2014), is one of the earliest and most intuitive attribution methods in deep learning. For Grad(t) we compute relevance scores as the (absolute) par- tial derivatives of the target outcome f(((z) with respect to the corresponding input feature x, at a particular time
TThis refers to (approximate) decomposition of the output, such as local accuracy (Lundberg & Lee, 2017) completeness (Sundararajan et al., 2017).

---

Gradient-based Explanations for Deep Learning Survival Models
IntGrad((). The Integrated Gradients (IntGrad) method by Sundararajan et al. (2017) attributes the contribution of each feature by comparing a model's prediction for given input æ to that of a baseline instance ă. This method sat- isfies several desirable properties, including completeness, sensitivity, linearity, and implementation invariance, making a widely accepted approach for feature attribution. Int- Grad computes contributions by integrating gradients along path from the baseline to the input, typically following straight line. In practice, this integral is approximated by averaging gradients at discrete intervals. The generalized IntGrad(t) results in a decomposition of the targeted curve of x minus the baseline curve of ह, e.g., for the survival curve, S(1]2) - ((())) for each time point l. Typical base- line values are zeros or the feature mean representing the "average patient". However, especially with non-linear or multi-modal distributions, careful selection of the reference value is required. It can be shown that for non-negatively homogeneous model and ã = O, IntGrad is equivalent to Grad x Input (Hesse et al., 2021).
toniadi et al., 2021). To ensure a comprehensive evaluation strategy, we conduct experiments on simulated and real data to answer the following research questions:
1. Can gradient- based methods correctly identify time- (in)dependent effects in different survival NNs?
2. How does GradSHAP(t) compare to SurvSHAP()) and SurvIIME in terms of local accuracy, computational speed and global feature rankings?
3. How can gradient-basee methods be feasibly leveraged to explain survival predictions based on multi-modal input data?
All experiments, including the figures, can be reproduced using our code on GitHub".
5.1. Experiments on Simulated Data
5.1.1. TIME-INDEPENDENT EFFECTS
Setup. In the first experiment, we generate synthetic data to demonstrate that gradient-based explanations can accu- rately capture time- ceme--ndependent local feature effects, pro- vided the models correctly identify them. The data consist of N = 10, 000 observations simulated from standard Cox PH model. The baseline hazard function is monotonically increasing and modeled using a Weibull distribution with a shape parameter 7 of 2.5. The features include one "harm- ful" feature xı with a log hazard ratio of 1.7, one "protective" feature 22 with log hazard ratio of -2.4, and one feature with no effect the hazard 23. The maximum follow-up time is set to t = 7. More details on the simulation setting, training process, fitted models and selected observations can be found in Appendix A. For all experiments, we use our R package survinng".
GradSHAP((). In many cases, it is most meaningful for the baseline value è to conceptually reflect the complete absence of features. However, for tabular data, choosing an adequate baseline can be challenging since a zero or mean value does not necessarily coincide with feature absence and may be aut-of-distribution In survival analysis, zero as a baseline can misrepresent missingness because it of- ten carries specific clinical meaning (c.g., a zero lab value might imply a specific medical condition), leading to biased explanations if not carefully chosen. Our time-dependent extension, GradSHAP()), of the GradSHAP method (Lund- berg& & Lee, 2017; Erion et al., 2021) addresses this by taking the expectation of IntGrad(t) explanations evaluated at randomly drawn reference values from 'D instead of using single potentially off-mmanfoldd baseline value (see Fig. 2). In survival analysis, using the expectation over a reference distribution can provide a generalized baseline, reflecting the "mean patient's 's survival time" and offering more stable and less biased comparisons. In practice, the expectation is estimated by Monte Carlo integration using the sample average of randomly drawn baseline values from D and parameterized points a ~ (100 1) for the integration path. This results in a decomposition of f((]ar)- - E.g ({( and describes a gradient-based approximation of SHAP values at selected time points.
Results. These first experiments aim to show how differ- ent gradient-based explanation methods identify the effects of lime-independent features. For this purpose, we split the data into training (9, 500 observations) and test set (500 observations) and fit a DeepSurv (Katzman et al., 2018), a CoxTime (K (vvamee et al., 2019), and a DeepHit (Lee et al., 2018) model to the training set.
Grad(t) (Fig. A.2) and SG(() (Fig. A.4) are output-sensitive methods; as such they indicate the models' sensitivity to fea- ture changes rather than appropriately capturing local effects on the survival prediction. Therefore, the global ground- truth effects are accurately reconstructed, with 21 having a negative, 22 stronger positive, and 23 no substantial effect survival over time across all models for both of the ran- domly chosen instances. In contrast, the attribution curves in Figures A.3, A.5 - A.8 capture feature-wise local effects.
5. Experiments
Unlike for the evaluation of MI. models, there is no well- established framework for XAI evaluation due to the ab- sence of a definitive ground truth for explanations and the reliance on an imprecise black-box model trained with im- perfect data (Liu al., 2021; Vilone & Longo, 2021; An-
5

---

Gradient-based Explanations for Deep Learning Survival Models
GxI(t) uses simple scaling of the sensitivity account for the magnitude of the feature's s contribution to the predic- tion. Thus, despite a negative global ground-truth effect of I2, since 22 < 0 for the 13th observation, its attribution curve only takes positive values as highlighted in Fig. 3. By multiplying the gradient by the input value, the method implicitly assumes that the relationship between the input and the model's 's output is locally linear near zero, which can produce misleading interpretations of feature contribu- tions due to the inherent non-linearity of survival prediction curves. The cubic shape of the survival prediction curves leads to approximately parabolic relevance curves when considering time-indee -ndeeeendddent feature effects, primarily be- cause the survival probability tends to exhibit fewer changes at the extreme ends of time. This behavior is particularly pronounced in Cox-based NNs due to the shape assump- tions inherent in their design. Since the survival curves in the Duer model are constrained to be proportional, the resulting relevance curves also exhibit approximate pro- portionality, Consequently, in time- -ineeeddeenndet scenarios, Time and DeepSurv yield similar results, indicating that CoxTime identifies the lime-independence of the features.
include one time-dependent feature #1 with a "harmful" ef- fect for < 2 and a "protective" effect for t 2, well as two time-independent features: one with a "harmful" effect (22) and one with a stronger "protective" effect (æz). Ad- ditionally, there is one feature with no effect on the hazard (za). The maximum follow-up period is set to t = 7. Fur- ther details on the simulation settings, training process and fitted models are provided in Appendix A.2.
Results. These experiments are designed to demonstrate how the different gradient-based explanation methods cap- ture the effects of time-dependent features. The relevance curves derived from output-sensitive methods (Figures A.12, 14) effectively reveal the dineddependent effect of x1 on the survival predictions, by indicating positive effect at ear- lier times and a negative effect later on. This time-dependent effect is accurately captured by CoxTime and DeepHit (as illustrated in Fig. 4), but not by DeepSurv, which is inher- ently constrained by the PH assumption and thus unable to model time-dependence. These results underscore the abil- ity of gradient- methods to uncover such differences between models, focusing on explaining model behavior rather than data, and are thus valuable for assessing whether time-dependent variables are correctly modeled.
DeepSurv
Instaco
Intance 327
Abriudtoo Suge Greatt
Irstance 10: 79
Inslance ID: 428
SttiibaiioSSttt Grad00
-23
0.0
-033
Feature
-0.4
DeepSurv
Indoooothh
Ab S/tt: G.00
0.0
Time
-25
Feature
Figure 4. Grad(t) relevance curves for the selected observations and the DeepHit model trained on the time-dependent simulation dataset.
Time
Feature
In addition to time-dependence in feature effects, difference- to-reference methods (i.e., IntGrad(t) and GradSHAP(O) provide insights into the relative scale, direction, and magni- tude of feature effects by comparing predictions to mean- ingful reference. as displayed in Figures A.20, A.23 and A.26. Contribution plots (Figures A.21, A.24 and A.27) effectively visualize the normalized absolute contribution of each feature to the difference between reference and (sur- vival) prediction over time, as shown for the GradSHAP(I) method and the CoxTime model for the two selected ob- servations in Fig. 5. Complementarily, force plats (Fig- ures A.2 A.25 and A.28) emphasize the relative contribu- tion and direction of each feature at a set of representative survival times, likewise exemplarily highlighted in Fig. 5. For example, the opposite effects of low vs. high values
Figure 3. Grad(t) (top) and GxT(f) (bottom) relevance curves for selected observations using the DecpSurv model trained on the time indepeddennn simulation dataset. The relevance values for each feature are represented by different colors (y-axis) and are plotted across time (x-axis), highlighting the temporal dynamics of feature contributions.
5.1.2. TIME- AME-DEPENDENT EFFECTS
Setup. In this experiment, we generate synthetic data to demonstrate that gradient-based explanations can accurately capture variables with local time-dependent effects, pro- vided the models correctly identify them. The dataset con- sists of N = 10, 000 observations simulated from a Weibull model analogous to Sec. 5.1.1 with 7 1.5. The features
6

---

Gradient-based Explanations for Deep Learning Survival Models
of 21 are effectively captured in the plots. In observation 79, low 2i positively influences survival at later time points (t > 2) compared to the overall average survival in the dataset, resulting in its largest contributions occur- ring at these times. Conversely, in observation 428, a high Ti induces substantial contributions at earlier time points (l < 2), but negatively impacts survival at later times, re- flecting its early event as a consequence of the high æ1 and the strong negative effect of 2t-- The average normalized absolute contribution, displayed on the right side of the contribution plots, offers time-independent measure of feature importance, confirming the dominance of 23 for the survival prediction of instance 428. Additionally, the visualizations suggest that CoxTime partially attributes the time-varying effect of 21 to the other features, as the model,
terpretations by measuring the "gain" of each feature to prediction (Chen et al., 2023). In the survival context, the only existing estimation approach is the model-agnostic SurvSHAP() method (Krzyzińsski et al., 2023). This sample- based strategy becomes computationally inefficient for high- dimensional feature spaces or deep NNs. Our proposed extension, GradSHAP((), provides a model-specific coun- terpart, leveraging gradients for more efficient and scalable attribution. In the following, we compare both methods in simulated settings in terms of accuracy, runtime, and their ability to correctly estimate global feature rankings, partic- ularly in comparison to SurvLIME. Further details of the simulations and comparable results for the other not-shown model classes can be found in Appendix A.3.
being non -on-aaaaettiic and lacking explicit knowledge of the dependent functional form, struggles to precisely disentangle and localize this effect.
Local accuracy. To evaluate the accuracy of the method, we use the time-dependent adaption of the local accuracy measure proposed in Krzyzińsski et al. (2023), which is function of t:
Abriuton Gorbtnnn
Ex (ftt)z) - [a(((() - -i R=(t)a))
0.
(5)
Em [f([[=]]
This metric measures the decomposition error global level and normalizes it against the mean prediction at each time point. The simulation setup follows the same structure as described in Sec. 5.1.1, with p = 30 features and 1, 000 training instances. The features have a uniformly increasing effect strength from 0 on the log-hazard, with alternating signs. The results for a DeepSurv model are shown in Fig. 6. In simulations, all 100 test samples are explained using the entire test set as the baseline dataset, while the number of interval samples (indicated in parentheses) for estimating the integral in GradSHAP(t) is varied. Our GradSHAP() explanations provide highly accurate approximations, even with a limited number of integration samples. Upon user demand, accuracy can be further improved by increasing the number of integration samples, albeit at the cost of a longer computation time.
Fosture
34
Comparison pet io (R pottre
Corfirea
Ttakios 12:13 73
takari 425
Corettiddt Gradivanni
1000
16
Tire
Festure
CocTiiee
Traturce 13
Troance D:422
Contittaton DISHHAR0
-2.
-44
Runtime. A key advantage of GradSHAP()) is its supe- rior scalability in higher dimensions, making it especially valuable for deep NNs. Fig. 6 illustrates the runtime of SurySHAP() and GradSHAP()) (with 5, 25, and 30 inte- gration samples) as a function of the input dimension p for DeepHit and DeepSurv. The plot demonstrates that the gradient- based method is significantly faster and main- tains good scalability even for larger p. However, it also highlights the trade-off between accuracy and runtime: in- creasing the number of integration points results in longer yet still competitive computation times. This is mainly due to the computation of the gradients of temporary instance
Figure 5. GradSHAP(t) relevance curves, with corresponding sur- vival prediction curves, reference curve and their difference (top), contribution plots (middle) and force plots (bottom) for the se- lected observations and the CoxTime model trained on the time- dependent simulation dataset.
5.2. GradSHAP(() vs SurvSHAP(I)
One of the most established XAI methods are SHAP val- ues, which - rooted in game theory - offer intuitive in-
7

---

Gradient- Explanations for Deep Learning Survival Models
Ueded Sansupp iasie/d1 Gradlind
accuracy and efficiency with dramatically reduced runtime for deep survival models, which is essential advantage for scaling interpretability in high-dimensional survival settings.
Duupdure
30-00
Loca ecuuacy
1+-40
do
Isttrce-wiee Local Accuracy
Rustine Comparison
1n-00
12
01
Suntor 11feaa 181
Time
Tur regle
Figure 6. Runtime (left) and local accuracy (right) comparison of SurvSHAP(I) and GradSHAP() with varying numbers of integra- tion samples (5, 25, 50) showing the trade-off between accuracy and efficiency.
dundanng
Loc LOGANAy Jmorgg art Iespers
Tene Biei
Figure 7. Instance- wise local accuracy (left) and runtime (right) for GradSHAP() and SurvSHAP()) on a worldworld survival model with high-dimensional image inputs. Both methods achieve similar accuracy, while GradSHAP()) is considerably faster in all settings.
for each sample and each integration point, leading to computational complexity of Onn Rsamples Ramt).
Practical feasibility. To complement our simulation- based runtime analysis, we demonstrate that GradSHAP(t) enables SHAP-like explanations for high-dimensional in- puts much faster than its model-agnostic counterpart SurySHAP(() in a real-world example. Using the dataset and DeepHit model from Sec. 5.3, we replace the original ResNet34 (He et al., 2016) with smaller variant (ResNet18) and reduce the input image size from 226x226 to 32x32, which still constitutes a high-dimensional input space. In this experiment, we explain a single instance and compare GradSHAP(t) and SurvSHAP() across several parameter settings in terms of runtime and time-averaged instance- wise local accuracy, i.c., an aggregated measure for the prediction-normaliae decomposition goal of a single expla- nation (local explanation level)
Global importance ranking. Beyond patient- -wise local effects, it important to assess whether features consistently rank as influential on a global level. Stable importance rank- ings indicate that the model captures robust patterns rather than instance-specifie artifacts. For this simulation, use p 5 features, each having an evenly increasing effect on the log-hazard function with alternating signs. Additionally, we include the importance ranking of SurvLIME weights as a competing global XAI method. SurvLIME is an exten- sion of the LIME framework adapted for survival models, fitting local surrogates. As shown in Fig. 8, the feature im- portance rankings of SurvSHAP(t) and GradSHAP(t) are nearly identical and agree with the Sata-generating process. The discrepancy to SurvLIME is consistent with observa- tions reported in previous studies (Krzyziński et al., 2023).
((11)) - Ea [[([a)] Ei=n (1(())))
E.
(6)
Teatures Uncressing Inportencel
Ex [f(((=)]
SaNdHOP3)
SUNUME
Figure 7 summarizes the results. The left panel shows the time-averaged instance-wise local accuracy values (lower is better) across all tested configurations. Overall, Grad- SHAP(t) achieves similar or even slightly better accuracy levels to SurvSHAP()) The right panel compares the run- time on a logarithmic scale. While both methods scale with the number of samples, GradSHAP(() is substantially faster, particularly in settings with a higher number of samples (0), requiring only fraction of the computation time compared to SurvSHAP(). For example, even in its most innpute-intensiee configuration, GradSHAP(() completes in a couple of seconds, whereas SurvSHAP() requires sev- eral minutes for similar accuracy. This demonstrates that GradSHAP() offers feasible trade-off between estimation
113
101
Figure 8. Global importance rankings for GradSHAP(). SurvSHAP(), and SurvLIME across 300 test samples. Grad- SHAP(t) and SurvSHAP(() show consistent rankings aligned with the data gerrratennrtingggg process.
5.3. Example on Real Multi-modal Medical Data
Despite the critical role of time-to-event prediction in medi- cal decision decision-making making, deep learning remains underutilized in this domain, even with its success in other medical ap-
8

---

Gradient-based Explanations for Deep Learning Survival Models
plications. Our primary motivation for developing gradient- based explanation methods for survival deep learning is to help researchers harness the unique abilities of deep learn- ing to extract and integrate complex features from high- dimensional, unstructured data while ensuring interpretabil- ity, transparency in individual-Ievel decision-making. and facilitating new domain knowledge discovery.
Imarce de B:T004-E-1155
Forca Plae GosSMMP
In order to show a use case, we apply the methods to CNN-based extension of DeepHit model trained real-world multi-modal medical dataset predicting overall survival in diffuse gliomas (Mobadersany et al., 2018). We use four tabular features selected based on previous results (age, sex, absence or presence of IDH mutation and 1p/19q codeletion) and histologic images of the regions of interest of whole-slide image tissue sections from formalin-fixed. paraffin-em -rraffineemeedded specimens from The Cancer Genome Atlas (TCGA) Lower-Grade Glioma (LGG) and Glioblas- toma (GBM) projects. The molecular features are known to be helpful for predicting survival in gliomas and other brain tumors. Based on the WHO's ss histologic classification of gliomas (Park et al., 2023), the isocitrate dehydrogenase mutation (IDH) involves alterations in the IDHI or IDH2 genes and is associated with more favorable prognosis. The 1p/19 codeletion refers to the simultaneous loss of parts of chromosomes 1 and 19. The absence of a 1p 9q codeletion is associated with more invasive and treatment- resistant gliomas, leading to a worse survival prognosis. For the image data of resized shape 226x226, we use a standard ResNet34 architecture (He et al., 2016). The high-level representations extracted from the ResNet are flattened to 256 features and fused with the tabular data. This combined representation is then passed into final dense network, which serves as the base model for a multi-modal DeepHit architecture. This multi-modal model is trained on a total of 1, 239 training samples and evaluated on 266 test samples, achieving C-index of 0.713 and an integrated Brier score of 0.092. We use the standard DeepHit loss function with an a-value of 0.5 to balance the rank loss and log- likelihoodd loss equally. For an individual-level temporal explanation, we use GradSHAP(() with 100 baseline samples and 20 integration points, computation took almost 12 minutes.5
Time
Figure 9. GradSHAP(I) explanation for a multi-modal Di model, showing temporal contributions of tabular genetic and clinical features (top). Below, corresponding image patches over time are visualized, with the original histology image shown at the bottom left.
eventually shifts below the x-axis, indicating a survival disadvantage over time, potentially driven by the absence of the 1p/19q codeletion and the diminishing effect of the IDH mutation at later time points. The aggregated effect of the image modality consistently indicates a negative impact on the survival probability. This is further evidenced in the image explanations (bottom), predominantly highlighting cells with a negative contribution.
6. Conclusion
In this work, we introduce set of novel model-specific local XAI methods for survival deep learning. The meth- ods are the first to provide ieeeddepndeent explanations for multi-modal data, including images, in the survival setting, but is likewise easily generalizable to any functional out- puts. Our work equips practitioners with a toolkit to derive meaningful insights from fitted survival NNs while account- ing for underlying model assumptions grounded in survival analysis theory and the interpretability offered by differ- ent gradient-based techniques. This promotes transparency, accountability, and faimess in sensitive applications such as clinical decision-making the development of targeted therapies, medical interventions, and other healthcare con- texts. However, tions do not imply causal relationships, as the models lack knowledge of the true causal structure of the data- genta--eneratig process. Future work will extend these methods beyond right-censored survival to include competing risks, multi- state models, and recurrent events. We also aim to develop targeted XAI methods for deep learning that can detect both individual feature effects and interactions.
Fig. shows the force plot of GradSHAP() explanation for a 43-year-old male patient with an IDH mutation but no 1p/19q codeletion. The black solid line represents the difference between the patient's survival prediction and the dataset-wide average. Initially, the line remains above the x-axis, suggesting a survival advantage in the early months likely due to the protective effect of the IDH mutation, which contributes positively at all time points. However, the curve
important to note that these explana-
"Attempts to apply SurvSHAP() with the same setup but just single baseline sample did not complete within week and used almost 800GB RAM, showeasing the advantage of gradient-based explanations for deep survival models.

---

Gradient-based Explanations for Deep Learning Survival Models
Acknowledgements
surv R package. Journal of Statistical Software, 97(3):
1-27, 2020. doi: 10. (0.18637/jss.v097..0
The results shown here are in part based upon data generated by the TCGA Research Network:
Chen, H., Covert, I. C., Lundberg, S. M., and Lee, S.-I. Algorithms to estimate Shapley value feature attributions. Nature Machine Intelligence, 5(6):590 -019000011-000 2023.
https//www..cnccergoveeeec This project and the authors were supported by the German Research Foundation (DFG),
Grant Numbers: 43761 1051, 459360854.
Ching, T., Zhu, X., and Garmire, L. X. Cox-nnet: An artificial neural network method for prognosis prediction of high-throughput omics data. PLoS Computational Biology, 14(4):e10060)) 1006076, 2018. doi: 10.1371/jourall. pcbi. 1006066.
Impact Statement
This paper introduces framework for interpreting indi- vidual predictions of survival analysis deep learning mod- els. which has the potential enable more transparent and actionable predictions of time-to-event outcomes in criti- cal fields such as healthcare, insurance, and personalized medicine. Therefore, this work addresses ethical concerns regarding the opacity of deep learning models, ensuring that predictions can be understood and trusted by stake- holders. This transparency is erucial for mitigating biases, fostering equitable decision-making. and ensuring compli- ance with regulatory standards. It needs to be stressed, that interpretable machine learning methods are useful to dis- cover knowledge, to debug or justify, as well as control and improve models and their predictions, but not to draw causal conclusions from the data. While there are risks of misuse and misunderstanddin, we believe the net positive impact substantially outweighs the risks. Future societal consequences include enhanced patient care through person- alized treatment plans, improved risk assessment in critical industries, and broader public trust in Al-driven systems.
Cho, H. J., Shu, M., Bekiranov, S., Zang, C., and Zhang, A. Interpretable meta-learning of multi-omics data for survival analysis and pathway enrichment. Bioinformat- ics, 39(4): 13, 2023. doi: 10.109//bionfformatis// btad113.
Chollet, F. et al. Keras. https::/kkeeat iettaas.oo.20 2015.
Cox, D. R. Regression models and life-tables. Journal of the Royal Statistical Society: Series B (Methodological), 34(2):187-220, 1972.
Erion, G., Janizek, J. D.., Sturmfels, P., Lundberg, S. M., and Lee, S.-I. Improving performance of deep learning models with axiomatic attribution priors and expected gradients. Nature Machine Intelligence, 3(7):620-631, 2021. doi: 10. 10.10880/442556-00--003333
Ferreira, A., Madeira, S. C., Gromicho, M., de Carvalho, M., Vinga, S., and Carvalho, A. M. Predictive medicine using interpretable recurrent neural networks. In Interna- tional Conference on Pattern Recognition, pp. 187-202. Springer, 2021. doi: 1000079778-3-000-6776663
References
Ancona, M., Ccolini, E., Öztireli, C., and Gross, M. Gradient-based attribution methods. In Samek, W., Mon- tavon, G., Vedaldi, A., Hansen, L. K., and Müller, K.- R. (eds.), Explainable Al: Interpreting, Explaining and Visualizing Deep Learning, pp. 169-191. Springer In- ternational Publishing, Cham, 2019. doi: 10.1007/ 978-3-030-28954-6.99
Hao, J., Kim, Y., Mallavarapu, T., Oh, J. H., and Kang, M. Cox-PASNet: Pathway-base sparse deep neural network for survival analysis. In 2018 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), pp. 381-386. IEEE, 2018 doi: 10.1109/BIBM.2011 2018. 8621345.
Antoniadi, A. M., Du, Y., Guendouz, Y., Wei, L., Mazo, C., Becker, B. A., and Mooney, C. Current challenges and future opportunities for XAI in machine learning- based clinical decision support systems: A systematic review. Applied Sciences, 11(11))5088, 2021. doi: 10. 3390/app1 1155088.
Hao, J., Kosaraju, S. C., Tsaku, N. Z., Song, D. H., and Kang, M. PAGE-Net: Interpretable and integrative deep learning for survival analysis using histopathological im- ages and genomic data. In Pacific Symposium on Biocom- puting 2020, pp. 355-366. World Scientifie, 2019. doi: 10.1142/97898 1215636_0032.
Bender, R., Augustin, T., and Blettner, M. Generating sur- vival times simulate Cox proportional hazards models. Statisties in Medicine, 24(11):1133-1723, 2005. doi: doi: doi:1001002/smm220533
He, K., Zhang, X., Ren, S., and Sun, Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattrrn Recognition (CVPR), pp. 770-778, 2016. doi: 10. 10.110//VVP..201..000
Brilleman, S. L., Wolfe, R., Moreno-Betancur Betancur, M., and Crowther, M. J. Simulating survival data using the sim-
Hesse, R., Schaub-Meyer S., and Roth, S. Fast axiomatic attribution for neural networks. In Advances in Neural
10

---

Gradient-based Explanations for Deep Learning Survival Models
Information Processing Systems, volume 34, pp. 19513- 19524. Curran Associates, 2021.
Lundberg, S. M. and Lee, S.-I. A unified approach to inter- preting model predictions. In Guyon, I., Luxburg, U. V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, R. (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
Katzman, J. L., Shaham, U., Cloninger, A., Bates, J., Jiang, T., and Kluger, Y. DeepSurv: Personalized treatment rec- ommender system using a Cox proportional hazards deep neural network. BMC Medical Research Methodology, 18(1):24, 2018. doi: 11186//1287--08-000
Mobadersany, P., Yousefi, S., Amgad, M.. Gutman, D. A., Barnholtz-Sloan, S., Velázquez Vega, E., Brat, D. J., and Cooper, L. A. Predicting cancer outcomes from his- tology and genomics using convolutional networks. Pro- ceedings of the National Academy of Sciences, 115(13); E2970- E0979, 2018. doi: 10. 10.10733/naa..171113911
Koenen, N. and Wright, M. N. Interpreting deep neural networks with the package innsight. Journal of Statistical Software, 111(8):1-52, 2024a. doi: 10. 18637//ss..v111. i08.
Koenen, N. and Wright, M. N. Toward understanding the disagreement problem in neural network feature attri- bution. In World Conference on Explainable Artificial Intelligence, pp. 247-269. Springer, 2024b.
Montavon, G., Lapuschkin, S., Binder, A., Samek, W., and Müller, K.-R. Explaining nonlinear classification deci- sions with deep Taylor decomposition. Pattern Recogni- tion, 65, 2017. doi: 10. 10. www.///...a..ogg.... 2016.11...11111 11.00.0.008...
Kovalev, M. S., Utkin, L. V., and Kasimov, E. M. SurVLIME: A method for explaining machine learning survival mod- els. Knowledge-Batte Based Systems, 203:106164, 2020. doi: (0.1016/j.koosys.200 200.0 1066.4
Park, Y. W., Vollmuth, P., Foltyn- Dumitru, M., Sahm, F., Ahn, S. S., Chang, J. H., and Kim, S. H. The 2021 WHO classification for gliomas and implications on imaging diagnosis: Part 1-key points of the fifth edition and sum- mary of imaging findings on adult-typo diffuse gliomas. Journal of Magnetic Resonance Imaging, 58(3):677-689, 2023, doi: https://doi.or// wttpg/11.100000 1000000/mmri.2844
Krishna, S., Han, T., Gu, A., Wu, S., Jabbari, S., and Lakkaraju, H. The disagreement problem in ex- plainable machine learning: A practitioner's perspec- tive. Transactions on Machine Learning Research, 2024. URL https::/openrevieee net. forum? JESY2WTZE
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Kopf, A., Yang, E., DeVito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., and Chintala, S. PyTorch: An imperative style, high- nigh-performance deep learning library. In Advances in Neural Information Processing Systems 32, pp. 8024 8035, Curran Associates, Inc., 2019.
Krzyziński, M.. Spytek, M., Baniecki, H., and Biecek, SurvSHAP (t): Time-dependent explanations of machine learning survival models. Knowledge-Bass Based Systems, 262:110234, 2023. doi: 2.11016j.knosss2222.10222.
Kvamme, H., Ørnulf Borgan, and Scheel, I. Time-to-event prediction with neural networks and Cox regression. Jour- nal of Machine Learning Research, 20(129):1-30,, 2019. URL http://jmll jmlr. tttp://lr.r.................eeee httg//aperr////ee httpp/pppp///--/eeeee wwtps/////////////// html.
Rahman, M. M. and Purushotham, S. Fair and interpretable models for survival analysis. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 1452-1462, 2022.
Langbein, S. H., Krzyzińiski, M., Spytek, M., Baniecki, H., Biecek, P., and Wright, M. N. Interpretable ma- chine learning for survival analysis. arXiv preprint, arXiv:2403. 10250, 2024.
Ribeiro, M. T., Singh, S., and Guestrin, C. "why should I trust you?" Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 1135-1144, 2016.
Lee, C., Zame, W., Yoon, J., and Schaar, M. v. d. Deep- Hit: A deep learning approach to survival analysis with competing risks. Proceedings of the AAAI Con- ference on Artificial Intelligence, 32(1), 2018. doi: v3300//aa..v32...... 11842.
Shrikumar, A., Greenside, P., Shcherbina, A., and Kun- daje, A. Not just black box: Learning important fea- tures through propagating activation differences. arXiv preprint, (arXiv 1605.01713), 2017. doi: 10.48550/arXiv. 1605.01713.
Liu, Y., Khandagale, S., Khandagale, S., White, C., and Neiswanger, W. Synthetic benchmarks for scientific re- search in explainable machine learning. In Vanschoren, J. and Yeung, S. (eds.), Proceedings of the Neural In- formation Processing Systems Track on Datasets and Benchmarks, volume 1, 2021.
Simonyan, K., Vedaldi, A., and Zisserman, A. Deep in- side convolutional networks: Visualising image classi- fication models and saliency maps. arXiv preprint, (arXiv: 13126.13.,.60344. 2014. doi: 10.48550/arxiv.131 1312. 6034.

---

Gradient-based Explanations for Deep Learning Survival Models
Smilkov, D., Thorat, N., Kim, B., Viégas, F,, and Wat-
tenberg, M. SmoothGrad: Removing noise by adding
noise. arXiv preprint, aarXiv:176.00322. 2017. doi: 10448550HaXxx.1166.03333.
Sonabend, R. survivalmodels: Models for Survival Analy- sis, 2024. URL httpt://cCcAN R-pooject .org/
package sus R package version 0.1.191.
Sturmfels, P., Lundberg, S., and Lee, S.-I. Visualizing the impact of feature attribution baselines. Distill, 5(1): 2020.
Sundararajan, M., Taly, A., and Yan, Q Axiomatic attri- bution for deep networks. In Precup. D. and Teh, Y. W. (eds.), Proceedings of the 34th International Conference Machine Learning, volume 70 of Proceedings of Ma- chine learning Research, pp. 3319-3328. PMLR, 2017. URL www.thttp//cmmm / /ttp:/eeccceeeeeseoooooo mlr www....rssss/00 sundararajanl html.
Tang, B., Li, A., Li, B., and Wang, M. CapSurv: Capsule network for survival analysis with whole slide pathologi- cal images. IEEE Access, 7:26022-26030, 2019.
Vellido, The importance interpretability and visualiza- tion in machine learning for applications in medicine and health care. Neural Computing and Applications, 32(24): 18069-18083, 2020.
Vilone, G. and Longo, L. Notions of explainability and evaluation approaches for explainable artificial intelli- gence. Information Fusion, 76:89-106, 2021. doi: htt.::..inffuu.222000...
Wiegrebe, S., Kopper, P., Sonabend, R., Bischl, B., and Bender, A. Deep learning for survival analysis: A review. Artificial Intelligence Review, 57(3):65, 2024. doi: 10.
1007/s10462-023- 1100-0222-110-1-33 10681-3.
Zhang. S., Bamakan, S. M. H., Qu, Q., and Li, S. Learning for personalized medicine: A comprehensive review from deep learning perspective. IEEE Reviews in Biomedical Engineering, 22:194-008, 2019. doi: 10. 109/RBME. 2018.2864254.
Zhu, X., Yao, J., and Huang, Deep convolutional neural network for survival analysis with pathological images. In 2016 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), pp. 544-547, 2016. doi: 10.
WWW.BBBM.011..88255555
12

---

Gradient-based Explanations for Deep Learning Survival Models
A. Experiments on Simulated Data
All simulations and real-data examples presented in this manuscript, along with the corresponding code, are available in our GitHub repository at https: /ttp::://thtbbb com/ cow/bips-oeoeeeceeeceeocoeei htt.srvvialaaaaaaaeta.aaaaaaaaae -etaal--------- ICML/ ensure transparency and reproducibility.
A.1. Time-independent Effects
For chosen observation, the hazard function from which the data are generated of the form:
h(1x)) A 0- exp(1.7x1 2.4x2)
with A == 0.1 and 2.5 and æ1, 22, 23 (0, 1). To generate the event times g(0) for instance i, the method of (Bender et al., 2005) is applied and the simsurv package (Brilleman et al., 2020) is used. Observations are artificially censored for {(1) 2 7.
The data is split into training (9, 500 observations) and test set (500 observations) and DeepSurv, CoxTime and DeepHit models with two dense layers with nodes are to the training data without tuning, using 500 epochs, early stopping, batch size of 1, 024 and a dropout probability of 0.1 applied to all layers. For any other hyperparameters, including the activations the default values set in the pyсоx (K (Kvamme al., 2019) Python package used, which are based on the default values suggested by (Katzman et al., 2018; Kvamme et al., 2019; Lee et al., 2018). More details are provided in our code supplement.
The models' performance expressed in the Brier score is shown in Fig. A.1 and Table A. shows the Concordance index (C-index) and Integrated Brier Score (IBS) as aggregated performance measures. DeepSurv slightly outperforms CoxTime and DeepHit in the Brier Score, likely because of its inherent assumption of proportional hazards, which the data simulated from a Cox model with time-independent effects are subject to. Even though performs similarly to DeepSurv in Brier Score, this does not necessarily imply conformity to the PH assumption; it has to be assessed, for instance using gradient-bast based explanations. DeepHit has higher C-index and IBS, suggesting poorly calibrated probabilistic predictions compared DeepSurv and CoxTime, which a consequence of the emphasis on C-index maximization in the loss function.
Table A. Performance metrics for different sur- vival models fitted on time-independent simula- tion data (C-index: higher is better, 0.5 indicates random prediction: IBS: lower better).
MODEL
C-INDEX (t)
IBS (4)
COXTIME 0.807
0.1
DEEPSURV 0.809
0.099
DEEPHIT 0.809
0.142
Two observations are randomly chosen to illustrate the gradient-basee based explanation methods for survival deep learning models delineated in Sec. 4. Their respective feature values and observed survival times are denoted in Table A.2. The survival curves predicted by the selected survival NN models are shown in Fig. A.9.
0.20
0.15
Score
The relevance values for each feature are represented by different colors are plotted across time in Figures A.2-A.8 for the selected ob- servations and models, highlighting the temporal dynamics of feature contributions.
0.10
0.05
0.00
1111 A
2
Time
Table A.2. Feature values, observed survival time and event status of two randomly chosen observations (ID and ID 387) from the test set of the simulated datasct with time- independent feature effects
DeepHit
DeesSurv
Figure A. Performance of DeepSurv, CoxTime, and DeepHit models over time measured by Brier score (lower is better; Brier score of 0.255 indicates prediction random).
ID
TIME
STATUS
x1
x2
x3
13
2.665
1
-0.435
0.1162
-0.081
387
0.958
1
2.455
0.2462
-0.043
13

---

Gradient-based Explanations for Deep Learning Survival Models
Coxtine
Incance ID:33
Indence ID: 33?
CoxTime
Intente D:
Trberce ID:
Atritatton S00p0: Crenie
3.5
2.0
Time
Time
DeepHit
Doaphit
Incance ID:31
Indance 337
Infance
Abrbuttin Sibeo- Cronlll
22
GHIt
0.3
0.0
+0.2
00.4
Time
Time
Feature
Feature
DeepSurv
DrepSurv
Attibution 580A Greedt)
1.0
40.
3.0
3.5
Feature
Feature
Figure A.2. Grad(t) relevance curves for selected observations and models trained the time-indepp indeeeddeendent simulation dataset. The relevance values for each feature are represented by different colors (y-axis) and plotted across time (x-axis).
Figure A.3. GxI(() relevance curves the selected observations and models trained the time-independent simulation dataset. The relevance values for cach feature are represented by different colors (y-axis) and are plotted across time (x-axis).
14

---

Gradient-based Explanations for Deep Learning Survival Models
Coxtine
Incarce ID:33
Indence ID: 33?
CoxTime
Intesse D: 13
Tetence ID: 187
Alnitttonn 5004 SC10
0.0
Time
Time
DeepHitt
Doaphit
Ineance ID:31
Indance 337
Infence
Adribaton 5100) SC220
2.2
0.3
0.2
2.0
Time
Time
Feature
Feature
DeepSurv
DrepSurv
Adrilation Shipe 5020
2.5
0.0
Feature
Feature
Figure A.4. SG(L) relevance curves for selected observations and models trained the iine-independent simulation dataset. The relevance values for each feature are represented by different colors (y-axis) and plotted across time (x-axis).
Figure A.5. SGxI(t) relevance curves for selected observa- tions and models trained the time-independent simulation dataset. The relevance values for each feature are represented by different colors (y-axis) and plotted across time (x-axis).
15

---

Gradient-based Explanations for Deep Learning Survival Models
+03
608
-04
Feeture
Feature
Deeplit
Deeplitt
Admibation S08-0 Inttonnt
10.
+02
0.3
Time
Time
Feature
Feature
13
DeepSurv
Insorce 10:33
Instance 397
DeepSurv
Insbace D 13
Irdtncc ID.
Hortubionnin Sipe InCHAURO
0.8
2.4
Time
Time
Feature
Figure A.6. IntGrad(1) relevance curves for the selected obser- vations and models trained on the time-independent simulation dataset. The reference value is the null observation (all feature values set to zero). The relevance values for each feature are rep- resented by different colors (y-axis) and are plotted across time (x-axis).
Figure A.7. IntGrad(t) relevance curves for the selected obser- vations models trained the time-independent simulation dataset. The reference value the mean observation (feature val- ues set to the average over all observations). The relevance values for each feature represented by different colors (y-axis) and are plotted across time (x-axis).
16

---

Gradient-based Explanations for Deep Learning Survival Models
CovTime
1.00
Sunvell Prediction
13.75
CoxTime
relonce E2:33
Attibutton Suiek GraeSHAPHD
0.55
0.00
0.25
-325
0.03
2.50
13 mn 387
Time
Desphit
Feeture
Deeplit
Surveel Predktion
Adtibaden 300 SQEaa CounSHAPR)
0.75
0.50
Time
387
Feature
DeepSurv
1.00
Noriutiinn SR.I GRINEHAPPI
0.25
Sunvval Preditioon
13.75
0.00
-325
0.25
0.00
Feeture
13 387
Figure A.8. GradSHAP(() relevance curves for the selected obser- vations and models trained the ineeeindepnndent simulation dataset. The relevance values for each feature are represented by different colors (y-axis) and are plotted across time (x-axis).
Figure A.9. Predicted survival curves of CoxTime, Deep- Hit and DeepSurv model for randomly chosen observa- tions (blue: ID 13, orange: ID 387). The models unan- imously predict a higher probability of survival any given time point for ID 13.
17

---

Gradient-based Explanations for Deep Learning Survival Models
A.2. Time-dependent Effects
For chosen observation, the hazard function from which the data are generated is of the form:
h(()a:)) = x 8- exp( -321 + 1.72g 2.4x3 + 6 log(t))
(8)
with À = 0.1 and 1.5 and ~ ~1, 1), 22 ~ (O0 1), xa ~ ( 1). To generate the event times ((0) for instance i, the method of (Bender et al., 2005) is applied and the s simsurv package (Brilleman et al., 2020) is used. Observations are artificially censored for { 2 7. The Kaplan-Meier survival curves grouped by low and high values of feature I1 demonstrate the time-dependent nature of its effect. Individuals with high values of si initially show a higher average probability of survival earlier time points (t 2), but their survival probability declines more rapidly over time, leading to lower predicted probability of survival at later time points (1 > 2). A real-world example of such effect could be cancer medication that provides strong early benefits by effectively slowing tumor progression or reducing symptoms. However, over time, the medication's 's efficacy might diminish due to drug resistance or cumulative side effects, resulting in worse long-term outcomes for patients compared to those on lower dosage treatments.
The data is split into training (9, 500 observations) and test set (500 observations) and DeepSurv, CoxTime and DeepHit models with two dense layers of 32 hidden nodes are fit to the training data without tuning, using 500 epochs, early stopping, a batch size of 1, 024 and a dropout probability of 0.1 applied to all layers. For any other hy- perparameters, including the activations, the default values set in the рyсоx (Kvamme et al., 2019) Python package are used, which are based on the default values suggested by (Katzman et al., 2018, 8;; Kvamme et al., 2019; Lee et al., 2018). More details are provided in our code supplement.
Table A.3. Performance metrics for different survival models fitted time-dependent simulation data (C-
index: higher is better, 0.5 indicates random prediction; IBS: lower is better).
MODEL
C-INDEX (t)
IBS (J)
COXTIME 0,85
0.058
DEEPSURV
0.86
0.06
DEEPHIT
0.805
0.095
The models' performance expressed in the Brier score is shown in Fig. A.10 and Table A.3 shows the Concordance index (C-index) and the Integrated Brier Score (IBS) as aggregated performance measures. CoxTime slightly outperforms DeepSurv and DeepHit in the Brier Score, likely because it is able to appropriately capture the violation of the assumption of proportional hazards in the time-dependent effects simulation. However, a superior performance does not necessarily imply that feature effects are captured correctly; this has to be assessed, for instance, using gradient- explanations. DeepHit has a lower C-index and higher IBS, suggesting poorly calibrated probabilistic predictions well as poor discriminatory power compared to DeepSurv and Cox Tome, perhaps result of the Cox-based nature of the simulation.
0.15
1.00
Survival S Probability
0.75
Score
0.10
0.50
0.05
0.25
0.00
0.00
0
2
2
4
Time
Time
CoxTime
DeepHit
DeepSurv
x1 Group
xl_group-Low
x1_ group-High
Figure A.I0. Performance of DeepSurv, CoxTime, and Deep- Hit models over time measured by Brier score (lower is better; Brier score of 0.25 indicates prediction random)
Figure A.II. Kaplan-Meic Meilan survival curves comparing indi- viduals based on high (orange) and low (blue) values of feature in. Each curve represents the estimated survival rate for the corresponding group.
18

---

Gradient-based Explanations for Deep Learning Survival Models
Two observations are randomly chosen illustrate the gradient-based explanation methods for survival deep learning models delineated in Sec. 4. Their respective feature values and observed survival times are denoted in Table A.4. The survival curves predicted by the selected survival NN models are shown in Fig. A.19.
Table A.4. Feature values, observed survival time and event status of two randomly chosen observations (ID 79 and ID 428) from the test set of the simulated dataset with time-dependent feature effects.
ID
TIME
STATUS
X1
X2
X3
X4
We propose several approaches effectively visualize relevance values while incorporating the temporal dimen- sion for terrreeeet--eeeeeeecee methods, beyond merely plotting the computed relevance values for individual fea- tures over time. In -ererenncet---efereenee methods, relevance values explain the deviation between the prediction (i.c., the predicted survival curve) for a selected observation and a chosen reference curve (e.g., the predicted survival curve for an observation where all feature values are set to zero, or where feature values are set to their respective means). To enhance clarity, we can plot the prediction (pred), the reference prediction (pred ref), and their difference (pred di f) along- side the relevance curves, as demonstrated in Figures A.20, A.23, and A.26. Contribution plots (Figures A.21, A.24, A.27) visualize the absolute, normalized feature-wis wise contributions to the prediction to reference difference, with the contributions represented as shaded areas colored by feature. These plots highlight how each feature relevances the prediction-to-referenee difference. The absolute normalized contributions are calculated as
79
4.306
0.194
-0.307
-0.148
0.637
428
1.417
1
0.753
-0.378
-1.15
-0.642
| Σx [
R norm
(9)
and are then plotted in stacked form, maintaining the feature order by using the cumulative across the features. This provides insight into the magnitude of each feature's percentage contribution to the prediction-to-refeenne difference at each point in time. Furthermore, the absolute normalized contributions can be averaged over time to derive time- ime-independent local feature importance measure, which highlights the overall impact of each feature across the entire time period:
R R
Raoem
(10)
which is plotted on the right hand side of Figures A.21, A.24, A.27. The force plots in Figures A.22. A.25, and A.28 are the time-dependent equivalent of the well-established SHAP force plots, illustrating how individual feature contributions combine to explain the cetfictionnt--rffreenee difference. These plots provide detailed breakdown the magnitude and direction of the factors influencing the prediction-to-refereee difference. representative set of equidistant observed survival time points (e.g., 10 points) selected, and the contribution of each feature visualized using stacked bar plots. The direction of each feature's effect emphasized by upturned arrows for positive contributions and downturned arrows for negative contributions. Different colors represent the respective features, and the magnitude of each feature's contribution is displayed as a label within its corresponding bar. The overall prediction-to-rcffe re-ferreec difference is shown as a black line for reference.
19

---

Gradient-based Explanations for Deep Learning Survival Models
Coxtine
Intence D-
Intance ID: -23
ConTiiee
Intence ID 79
Teberce ID: 420
Abrixatonn 500t0: Crentt
0.3
Time
Time
Deephiit
Doaphit
ID:75
Indance 10-423
Infente
Atrbuttnn Sibwo Sibwo Cruatle
0.0
22
0.2
Time
Time
Feature
Feature
DeepSurv
DeepSurv
Albibution 5804 Greddt)
Feature
Feature
Figure 12 Grad(t) relevance curves the selected observa- tions and models trained the dime-dependent simulation dataset. The relevance values for each feature are represented by different colors (y-axis) and plotted across time (x-axis).
Figure A. 13. GxI(() relevance curves for selected observa- tions and models trained the time-dependent simulation dataset. The relevance values for cach feature are represented by diffcrent colors (y-axis) and are plotted across time (x-axis).
20

---

Gradient-based Explanations for Deep Learning Survival Models
Coxtine
Intence DD
Intance DD -23
ConTiie
Inteste 12 79
Tebence ID: 420
Altit 5004 S:100
-03
Time
Time
DeepHiit
Doaphit
Iratnce
Iratance
Infance
0.25
Adribatonn S1ps) S0220
12.55
3.50
-03
Time
Time
Feature
Feature
DeepSurv
DrepSurv
Adrttatton Stipe 5028
604
Feature
Feature
Figure A. SG()) relevance curves for the selected observations and models trained the dimendependent simulation dataset. The relevance values for each feature are represented by diffcrent colors (y-axis) and are plotted across time (x-axis).
Figure A. 15. SGxI(D relevance curves for the selected observa- tions and models trained the time-dependent simulation dataset. The relevance values for cach feature are represented by diffcrent colors (y-axis) and are plotted across time (x-axis).
21

---

Gradient-based Explanations for Deep Learning Survival Models
Attibutton Supe: Intonodd
2.0
Attibaatoon
Festure
Feeture
Desprtit
Deepeit
Irstene
AMRR donn S0808 Incrmodt
03
60.3
32
-0.3
Time
Time
Feature
Feature
DeepSurv
Irstince D: 19
Instance D: 428
DeepSurv
Iustrce (D.
Inspince 425
0.32
Atrbuatin (i00: SMACHAAROO
0.00
-3.5
SEbA)
0.82
3.50
Time
Time
Feature
Fezture
Figure A.16. IntGrad(t) relevance curves for the selected observa- tions and models trained on the time- dependent simulation dataset. The reference value is the null observation (all feature values to zero). The relevance values for each feature are represented by different colors (y-axis) and plotted across time (X-axis).
Figure A.17. IntGrad(t) relevance curves for the selected observa- tions and models trained on the time-dependent simulation datasct. The reference value is the mean observation (feature values set the average over all observations). The relevance values for cach feature represented by different colors (y-axis) and are plotted across time (x-axis).
22

---

Gradient-based Explanations for Deep Learning Survival Models
CoxTimo
Survivl Fredetion
0.75
CoxTime
Atrixdton Sapp Sapp:: GraaSPAPPO
0.0
02
0.03
24
Time
Deephit
Festure
1.00
Deeplit
Adtibation SIEvo SIE00/ GrIdHHAPIH
leszane
UrwaalPredittt
0.2
0.0
0.25
2.1
32
Time
76
428
Festure
12
DeepSurv
DeepSurv
Insture
Inabance
Absbuttonn S1000 GHIEHHAP1
Sunhaal Predicion
0.75
2.00
-3.0
0.03
Time
Feature
428
Figure A.18. GradSHAP(() relevance curves for the selected ob- servations and models trained on the time-dependent simulation dataset. The reference value is the mean observation (feature val- ues set to the average over all observations). The relevance values for each feature represented by different colors (y-axis) and are plotted across time (x-axis).
Figure A9 Predicted survival curves of CoxTime, Deep- and DeepSurv model for randomly chosen observa- tions (blue: ID 79, orange: ID 428). The models uman- imously predict a higher probability survival at any given time point for ID 428.
23

---

Gradient-based Explanations for Deep Learning Survival Models
Atrikatinn Supa InConodi
100
2.5
2.0
Feelure 12
33
Comperison pnd prdd 28
prdd 191
Festure
Despriit
Intence
Deepeit
Instance
Indarce
Adtitd dbn 50000 InChmodd
2.0
Commibution ImGradtd
100
75
05
50
25
Time
Time
Feature
Comgarison
Feature
DeepSurv
Inspance 10.75
Instance 428
DeepSurv
Instance D:
Instaree 1D: 428
1.0
300
2.5
50
25
Time
Time
Festure
prot
Figure A.20. IntGrad(t) relevance curves (yellow, turquoise, blue, purple) and predicted survival curves for the selected observa- tions (ref), predicted survival curve for the reference obscrvation (pred_ref) and their difference (pred_diff) for models trained on the dime-dependent simulation dataset. The reference value the null observation (all feature values set to zero).
Figure A.21. IntGrad(1) contribution plots for the selected observa- tions and models trained on the time-dependent simulation dataset. The reference value is the null observation (all feature values set to zero).
24

---

Gradient-based Explanations for Deep Learning Survival Models
CosTime
Attibationn ImDrodtt
Contibttton ImCoredd
2.0
3.4
3.8
0.3
Testure
c1
Feeture x1
Comperison 2ndd padd
2ndd
Deepriit
Deepeit
68
2.0
044
0.0
2.4
Time
Time
Feature
Feaure
Comperison
DeepSurv
Inspinee 78
Instance
DeepSurv
Insbacco D. 79
Instnce ID:
Commiixtton: Ied sott
2.0
0.5
Time
Time
Festure
Comparson
proddd
Figure A.22. IntGrad(() force plots for the selected observations and models trained on the time-depende dependent simulation datasct. The reference value is the null observation (all feature values set zero).
Figure A.23. IntGrad(1) relevance curves (yellow, turquoise, blue, purple) and predicted survival curves for the selected observa- tions (ref), predicted survival curve for the reference observation (pred_ref) and their difference (pred_diff) for models trained on the time-dependent simulation dataset. The reference value is the null observation (feature values set to zero).
25

---

Gradient-based Explanations for Deep Learning Survival Models
CoxTine
Interse ID 78
Irebicee 420
CoxTiine
Intence 1D: 79
Trstrce ID:
% Corentationn Inttrwd))
25
403
Time
Time
DeepHiit
Doaphit
Instance
Tratunce 420
Instance
% Contrludemn Intioaatd
0.4
Time
Feature
Feature
DeepSurr
DeepSurv
N: Contrbutioee IMCredd
Feature
Feature
Figure A.24. IntGrad(() contribution plots selected observa- tions and models trained the time-dependent simulation dataset. The reference value is the mean observation (feature values set to the average over all observations).
Figure A.25. IntGrad(t) force plots for the selected observations and models trained on the time-dependent simulation datasct. The reference value the mean obscrvation (feature values set to the average over all observations).
26

---

Gradient-based Explanations for Deep Learning Survival Models
CoxTime
Incance ID:15
Indence I0-433
ConTime
Irabence
Indarce ID:433
Admibation Sripp: GHISHAAPH
35
0.0
Time
Time
Fealure
Comparison pred
pr2 prod
Feature
Alti S0NS: GrudS
300
04
Foature
Compansoo ard
prod prod
Feature
DeepSurv
DrepSarv
Abribation SiE- SIEVU GRIESHAP
1.0
300
Conrttation
50
0.0
Festure Q
13
Compartion prté ped_ prot.re
Feature
Figure A.26. GradSHAP(I) relevance curves (yellow, turquoise, blue, purple) and predicted survival curves for the selected obser- vations (ref), predicted survival curve for the reference observation (pred_ref) and their difference (pred_diff) for models trained the dimendependent simulation dataset.
Figure A.27. GradSHAP() contribution plots for the selected ob- servations and models trained the time-dependent simulation datasct.
27

---

Gradient-based Explanations for Deep Learning Survival Models
CoxTime
Instance ID: 79
Instance ID: 428
Contributionn Grad5
0.27
0.2
0.00
0.319
0.08
0.0
0.03
0.07
0.07
0.14
0.34
10:000
03
-0.2
05
-0.00
-029
0.06
-0.4
- 00
0.49
.5.5
-0.6
0
2
4
0
2
Time
F Feature
x1
x2
x3
X4
DeepHit
Instance ID:
Instance ID: 428
0.2
Contributions Grad5
0.09
D.3
0.18
0.1
0.04
0.04
0.63
0.05
0.0
0.01
-0.1
-0002
0.00
+0
04
00.0
04
-0.03
0.00
-0.01
-0.05
0.00
-0.0
2001
3.03
-0.13
0.14
-0.00
-0.
-0.22
0.07
6.05
-0.3
4
6
2
4
Time
F Feature
x1
x2
x3
x4
DeepSurv
Instance 79
Instance ID: 428
Contribution: Grad5
8.13
0.25
0.23
0.2
0.03
0.11
0.0
0,A3
0.03
0.12
0.03
0.04
0.01
0.05
-0.1
0.11
0.08
-000
VOS
0.06
-0.2
0.28
00
-0.4
-0.49
00.06
-0.6
Time
Feature
Figure A.28. GradSHAP(I) force plots for the selected observations and models trained on the time-dependent simulation dataset.
28

---

Gradient-based Explanations for Deep Learning Survival Models
A.3. GradSHAP() SurySHAP(t)
A.3.1. LOCAL ACCURACY
The concept of "local accuracy" originates from Shapley values and refers to the property that individual feature contributions sum up to the difference between the prediction and the average (i.e., marginal) prediction (Lundberg & Lee, 2017):
Σ $ f(x:) - [=)]
(11)
However, in the survival context, this decomposition must be considered for each time point, where the decomposition quantity dynamically changes time since survival functions monotonically decreasing. Krzyzifíski et al. propose time-dependent variation of this measure to account for these dynamics:
E (s()))) ( Esg [f())) Ei= R((1)))) 27
M(I)
(12)
Ei [44)))
This formulation gives greater weight to discrepancies at time points where the decomposition target becomes negligibly small, thus addressing situations where standard local accuracy would be less informative.
In our simulation study, we follow similar setup as described for the time-independent effects (see Sec. A.I). We use p = 20 features with coefficients linearly increasing from censor the event time at 10. The data is split into a training set (1, 000 observations) and a test (100 observations). We train DeepSurv, CoxTime, and DeepHit models with two dense layers and 32 hidden nodes the training data, using 30% validation data and early stopping for up to 500 epochs. For GradSHAP((), we vary the number of integration samples (5, 25, 50). Additionally, for both XAI methods, we apply the methods on all 100 test instances. The results for all model classes are displayed in Fig. A.29. Further details can be found in the code supplement on GitHuu.
and alternating signs, i.e., By = ( 1) - and artificially
Method -
SUnSHAAP(
GradSHAP (5)
GradS HAPIN (25)
GradS (50)
CoxTime
DeepHit
DeepSurv
1e-02
3e-02
3e-02
Local accuracy
1e-02
1e-02
1e-03
3e-03
3e-03
1e-03
1e-04
1e-03
3e-04
3e-04
1e-05
11
IIIIII
II
II0
11
1 IIIII
IIII
11
1
IIIIII
IIII
81
1
0.0
2.5
7.5
10.0
2.5
5.0
10.0
0.0
2.5
7.5
10.0
Time
Figure A.29. Local accuracy (y-axis), measured as the normalized standard deviation of the difference between the black-box model output and the explanation (lower is better), plotted time (x-axis) for SurvSHAP() (purple) and GradSHAP() with varying numbers of integration samples (blue 5, turquoise = 25, yellow = 50). GradSHAP(I) achieves local accuracy comparable to SurvSHAP() while significantly reducing runtime.
A.3.2. RUNTIME
To evaluate the computational efficiency of the proposed feature attribution methods, we compare SurvSHAP() and GradSHAP(t) with varying numbers of integration samples (5, 25, 50) across CoxTime, DeepHit, and DeepSurv models. The survival data is generated analog to the previous section but with varying p and split the data into a training (1, 000 observations) and a test set (100 observations). We train DeepSurv, CoxTime, and DeepHit models with two dense layers
29

---

Gradient-based Explanations for Deep Learning Survival Models
and 32 hidden nodes the training data, using 30% validation split and early stopping for up 500 epochs. To obtain stable time measurements, take the median runtime of 20 repetitions for single trained model and repeat this five times. Additionally, apply the XAI methods on all available test instances. To ensure a fair comparison of runtime performance, we do not employ any parallelization beyond controlling the number of threads. Specifically, we limit the number of torch threads and inter-op threads to 10 each. Further details can be found in the code supplement on GitHub.
Fig. A.30 illustrates the runtime (y-axis) as function of the number of features (x-axis). The results reveal notable differences in computational demands between the methods. SurvSHAP()) consistently exhibits higher runtime across all feature set sizes and model classes, with the computational cost rapidly increasing as the number features grows. This can be attributed to its reliance sampling multiple subsets of features for Shapley value estimation. In contrast, GradSHAP()) demonstrates significantly improved efficiency, particularly when using fewer integration samples, As the number of features increases, GradSHAP(() maintains computational efficiency, outperforming SurvSHAP()) in all configurations.
Notably, increasing the number integration samples from 5 to 50 for GradSHAP() results in higher runtimes but remains computationally more efficient than SurvSHAP() even for large feature sets. These findings underscore the scalability and efficiency advantages of gradient-based attribution methods for survival models when handling nigh-dimensional feature spaces.
Method
HAP
GradSHAP(U) (5)
GradSHAP (25)
GradSHAP(L (50)
CoxTime
DeepHit
DeepSurv
30.0
300
10.0
10.0
Runtime (sec)
100
30
1.0
10
0.1
0
100 150
150
50
100
150
Number features (p)
Figure A.30. Runtime (y-axis) for generating attributions using SurvSHAP(O (purple) and GradSHAP() with varying numbers of integration samples (blue= = S, turquoise = 25, yellow = 50), measured on simulated datasets with an increasing number of features (x-axis). GradSHAP() significantly improves computational efficiency the number of features grows, outperforming SurvSHAP() in all settings. For smaller ler feature sets, GradSHAPG with low integration samples also achieves faster runtimes than SurvSHAP()). The results are averaged over multiple runs (20 per model).
A.3.3. GLOBAL FEATURE RANKINGS
To assess the ability of the attribution methods to capture global feature importance, we evaluate GradSHAP()), SurvSHAP()) and SurvLIME on a simulated dataset with five features of predefined importance (x1 < .2 < 23 < æz < .s). The survival data generated analogously the runtime analysis with = 5, 2, 000 train 300 test samples, and models (CoxTime, DeepHit, DeepSurv) are trained as described in the previous section. Further details can be found the code supplement on GitHub.
Fig. A.31 presents the ranking distribution of features across the test set's 300 predictions for each model. Rankings are derived from the relative importance scores (i.e., the absolute values) assigned to each feature. Both GradSHAP(t) and SurvSHAP(() demonstrate robust performance by consistently maintaining the predefined importance hierarchy across all models. This consistency highlights their ability to capture the underlying feature relationships. In contrast, SurvLIME shows a more uniform distribution of rankings, leading a reduced capacity to differentiate features of varying importance.
30

---

Gradient-based Explanations for Deep Learning Survival Models
Overall, the results emphasize that gradient-based methods like GradSHAP() and sample-based methods like SurvSHAP() well-suited for identifying global feature importance in survival models, whereas SurvLIME may lack the precision required for nuanced analyses.
Features (increasing importance)
x1
x4
GradS HAP (1)
SurvSHAPIt
S SurvLIME
1st
156
94
44
151
97
43
72
39
74
2nd
56
CokTime
3rd
48
96
49
51
92
96
56
62
4th
58
20
36
Importaince ranking
5th
24
35
1st
163
82
51
156
70
50
66
55
59
2nd
DeepHit
3rd
41
61
57
86
50
110
50
62
59
4th
5th
58
250
74
66
2nd
1st
1699
82
168
51
85
63
48
DeepSuuv
3rd
33
62
110
B4
34
113
85
70
56
68
60
46
4th
121
21
5th
53
214
100
200
300
100
200
300
100
200
300
Figure A.31. Comparison of local and global importance rankings for 300 predictions from the CoxTime model (top), DeepHit model (middle) and DeepSurv model (bottom) simulated dataset including five features of increasing importance (21 D2 D3 as). Colors arranged in a gradient from purple to yellow reflect the global ranking of features within each model. GradSHAP()) and SurvSHAPP() perform similarly, consistently retaining the majority of observations for each consecutive feature. This demonstrates superior performance compared to SurvLIME, which produces more uniformly distributed rankings.
A.4. Computational Details
64-bit Linux platform running Ubuntu 22.04 LTS with two AMD EPYC Genoa 9534 64-Core Processors (128 cores, 256 threads total), 1.5 terabytes RAM, and eight NVIDIA RTX 6000 Ada Generation GPUs (each with 48 GB memory) was used for all computations.
31


Reviews:
[{"review_id": "cZbHo63Hme", "reviewer": "Reviewer_xEYK", "summary": "This paper addresses the challenge of interpreting \"black box\" deep learning models used for survival analysis, which predict time-to-event outcomes. The authors introduce a framework for gradient-based explanation methods to capture the time-dependent influence of various features, including those from multi-modal data like medical images and tabular information. They introduce GradSHAP(t), a gradient-based, model-specific counterpart to the model-agnostic SurvSHAP(t). Using both synthetic and real-data experiments, it is shown to be computationally efficient while maintaining accuracy compared to existing approaches.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for your constructive feedback. In response, we conducted additional experiments on the feasibility and computational efficiency of GradSHAP(t) and SurvSHAP(t) on the multi-modal real data example, which we will include in the final paper. Here, GradSHAP(t) took 5 minutes to compute, while we had to abort the computation of SurvSHAP(t) after 10 hours (details below). In light of these results, we argue that runtime is one of the most crucial aspects for post-hoc Shapley value approximation, which is supported by previous literature. Additionally, we want to highlight that local accuracy and importance ranking are already aggregated metrics. The following summarizes all your concerns in-depth:\n\n**R1A1) Limited gain of GradSHAP(t) relative to SurvSHAP(t)?**\n\nThe objective behind GradSHAP(t) is not merely to propose a \"better\" method, but rather a practical one that balances flexibility and computational feasibility. Although differences in local accuracy are visible due to the log scale, they are practically negligible, whereas the runtime improvement of GradSHAP(t) is substantial. This becomes particularly apparent in our additional experiments, including image data, which demonstrate that SurvSHAP(t) quickly becomes computationally infeasible without substantial resources, whereas GradSHAP(t) remains efficient and can be computed on a standard laptop (for further details refer to **R1A3**). Computational runtime is a well-established criterion in post-hoc XAI methods, particularly for SHAP explanations. It is frequently used as a key selling point in the scientific literature when introducing new estimation methods, see e.g., TreeSHAP (Lundberg et al., 2020), FastSHAP (Jethani et al., 2021) or various other SHAP algorithms such as in Ancona et al. (2019) or Chen et al. (2018) and runtime is included as a comparison metric in their SHAP benchmarking suite (Python SHAP package, Lundberg and Lee, 2017).\n\n**R1A2) Limited number of examples/lack of aggregated metrics?**\n\nOur comparison of SurvSHAP(t) and GradSHAP(t) was performed for all three survival DNN classes (see Appendix A.3 for full plot) with varying numbers of input features, averaged over 20 trained models per case. In spite of its name, local accuracy is already a dataset-wide aggregated measure (see Eq. 10, Appendix A.3.1.), which we plot for $p=30$ over the survival time in Fig. 6. Additionally, Figure 7 presents a comparison of global feature importance, particularly importance rankings, which are also aggregated over survival time. Could you clarify if you were referring to something beyond these analyses?\n\n**R1A3) Real data?**\n\nWe performed additional comparisons of SurvSHAP(t) and GradSHAP(t) on real data. On the multi-modal model (including images) for explaining a single instance (Sec. 5.3), we had to abort the computation of SurvSHAP(t) after 10 hours (using 256 threads and 700GB of RAM) with only two reference samples. In contrast, GradSHAP(t) completes in around 8 minutes with 100 reference and 20 integration samples. As an additional comparison, we conducted the same experiment (but ResNet18) on downscaled images (32×32) using a standard ML workstation (48 threads, 256GB RAM). As shown in the table below, SurvSHAP (with 50 samples) takes more than 41 times longer than GradSHAP (n = 25, samples = 50), which is nearly 25 minutes for just a single explanation compared to 36 seconds. GradSHAP achieves this not only faster but also with better instance-wise local accuracy (averaged over survival time $t$, i.e., no aggregation over all instances). \n\nWe agree that this is crucial information for the readers and we will include discussions and visualizations of the computational efficiency of both methods on real data in the final version of the paper. \n\n| Method                                 | Runtime           | \"Instance local accuracy\" (avg. over $t$) |\n|----------------------------------------|-------------------|---------------------------------|\n| GradSHAP(t) (n = 10, samples = 10) | 2.96 sec                     | 0.00108                         |\n| GradSHAP(t) (n = 25, samples = 50) | 36.07 sec                   | **0.00023**                         |\n| GradSHAP(t) (n = 50, samples = 50) | 1 min 18.05 sec         | **0.00021**                    |\n| SurvSHAP(t) (samples = 5)                | 2 min 45.30 sec         | 0.00230                         |\n| SurvSHAP(t) (samples = 25)              | 12 min 34.78 sec       | 0.00044                         |\n| SurvSHAP(t) (samples = 50)              | 24 min 45.93 sec       | 0.00027                         |\n\n\nWe are also happy to provide further clarification on Figure 8, if you further specify which aspects remain unclear from the figure and/or text.", "claims_and_evidence": "The primary claims for novel contributions in the paper are\n1. using existing gradient-based methods to explain survival predictions (specifically, capturing the time-dependence of features)\n2. introducing GradShap(t) and showing that it is computationally efficient while maintaining accuracy with its main competitor SurvShap(t) from a previous work\n\nPrior work has already established the importance time-dependent attribution of features to survival predictions. Claim 1 extends this specifically for gradient-based explanations.\n\nFor Claim 2, the evidence seems preliminary. We seem to lose in terms of the local accuracy (Fig 6). Moreover, I'm not sure how important the corresponding gains in runtime even are, since this is a post-hoc task, not an inference task where runtime is critical. The in-depth evaluation is done on 2 examples presented in Fig 6, and I'd have liked to additionally see aggregate metrics over the dataset. In the end, I'm left thinking it might provide marginal benefits over the existing method SurvShap(t), if at all.", "methods_and_evaluation_criteria": "- The synthetic experiments make sense in that they give us an idea of potential benefits and pave the way for real-data experiments.\n- The real-data experiments are lacking in that the gains relative to the main baseline, SurvShap(t), are unclear. The authors chose 2 examples but even on those, I see gains in runtime (which is not too important for a post-hoc task) but a loss in local accuracy (which is arguably more important for interpretation). I am also unable to parse Fig. 8 to judge whether what the model tells us is indeed sensible.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "I have pointed out the issues with the evaluations used, especially for the real-world data. I think the manuscript should contain not just 2 in-depth examples but also aggregate metrics over the dataset. It should also clearly compare with SurvShap(t) on the same examples and identify where the gains of GradShap(t) are coming from. As a reviewer it is hard for me ascertain based on the limited results here whether GradShap(t) works significantly better and what its benefits over SurvShap(t) are.", "relation_to_broader_scientific_literature": "The work extends interpretability of survival models from model-agnostic methods to gradient-based methods. The hope here is that while model-agnostic methods only deal with the model outputs, the gradient-based methods use the model's \"internals\" to get a better view into how the model is using its features.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strengths: I really enjoyed reading the paper, it is very well written.\n\nWeaknesses: I'd like to see more of\n- explanation of SurvShap(t) and how exactly GradShap(t) gains relative to it\n- experiments with real data that illustrate the above", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fu4HGqrDTt", "reviewer": "Reviewer_xZMF", "summary": "The authors introduce GradSHAP(t), an extension of SurvSHAP(t) that analyzes the gradients to better explain the model’s predictions. The authors also propose extensions of other gradient-focused XAI methods to align with the survival task.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the suggestion to include semiparametric AFT-based survival deep learning models, such as Deep AFT Rank-regression for Time-to-event prediction model (DART). It is an interesting approach, which estimates the survival function in similar fashion to a non-Cox-based version of the DeepSurv model. However, the pre-trained baseline hazard function additionally depends on the output of the base neural network, as highlighted in Eq. 9 in the paper by Lee et al. (2023). Unlike Cox-based methods, such as DeepSurv and CoxTime, this formulation allows for gradient computation on the baseline hazard, which could provide further insights into the differences between Cox and non-Cox methodologies. Given its potential, we aim to explore its integration and explanations in future research.\n\nOur current focus is on methods that are readily accessible to practitioners, particularly those implemented in widely used libraries such as Pycox (Python) and survivalmodels (R). Expanding these packages or developing more comprehensive survival analysis software to also include AFT models such as DART is an important direction, and we are actively working toward addressing this gap.\n\nMore broadly, extending gradient-based XAI methods to AFT-based deep learning models like DART is an exciting avenue for future work. We will highlight this in the paper's future work section.\n\nWe would also like to take the opportunity to highlight that we conducted additional comparisons of SurvSHAP(t) and GradSHAP(t) on real data. In the multi-modal model (Sec. 5.3), SurvSHAP(t) was aborted after 10 hours (256 threads, 700GB RAM) with only two reference samples, while GradSHAP(t) completed in ~8 minutes using 100 reference and 20 integration samples. See our response to Reviewer 1 (**R1A3**) for details.\n\nThank you very much for the positive feedback!", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "GradSHAP(t) (and the other proposed methods) clearly align with the application at hand. Explainability of survival models is critical due to their applications in fields such as healthcare. The temporal aspect creates additional complexity, which motivates the exploration of time-dependent explainability analysis.", "theoretical_claims": "The authors do not provide theoretical claims.", "experimental_designs_or_analyses": "This article relies on synthetic data for a bulk of its experiments. However, given that real world data is often difficult to explain (due to lack of domain knowledge), this is necessary for the purpose of this paper. These results are reinforced by a single real-world dataset, which contains known features of interest.\n\nThe experimental setup is well documented. Also, the motivation for the design decisions made when constructing the synthetic dataset are clearly stated and understandable.\n\nThe authors provide rigorous and convincing analysis of all experiments.", "relation_to_broader_scientific_literature": "The paper is related to both the survival analysis literature and the explainability research literature.", "essential_references_not_discussed": "While the paper considers GradSHAP(t) with respect to proportional hazard (DeepSurv, CoxTime) and discrete time (DeepSurv) models, I would be interested in an additional analysis of Accelerated Failure Time (AFT) model such as DART (Lee et al, 2023).", "other_strengths_and_weaknesses": "The extension of gradient based XAI to survival analysis is novel and has clear potential for future impact. The article is overall written well and all figures are visually digestible.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5opKuHnHMp", "reviewer": "Reviewer_dEe2", "summary": "This paper shows a comparative study on various explanation methods for survival analysis.  While there are several model-agnostic methods to interpret models for survival analysis, this paper considers gradient-based methods.  The applicability of the gradient-based methods is limited to models that can compute gradients (e.g., neural network models), but this paper shows its effectiveness compared to the other methods.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your careful evaluation and suggestions. We acknowledge that our contributions may not have been communicated clearly enough in the original submission. To address this, we will revise the manuscript to better clarify these key contributions:\n\n* Our work follows adaptations common in survival XAI but tackles key challenges beyond straightforward mathematical extensions. E.g., a naive application of these methods on CoxTime is not possible.\n* We introduce tailored visualizations and post-hoc interpretations for gradient-based survival outputs and exemplify the impact of implicit vs. explicit baselines, addressing the recent debate on the disagreement of gradient-based explanations.\n* As a novel contribution, we provide a software package implementing all described gradient-based XAI methods for DeepSurv, DeepHit, and CoxTime.\n\nIn the following, we provide a detailed explanation regarding your feedback:\n\n**R3A1)  Marginal technical contribution?**\n\nOur primary contribution is extending six standard gradient-based explanation methods to time-dependent survival analysis, addressing a crucial gap in survival XAI research. The extensions are far from trivial from both a technical implementation and post-hoc interpretation perspective. For instance, in the CoxTime model time is an input feature, allowing for complex relationships between the artificially created time feature and other features. Thus, applying gradient-based methods naively to the survival function $S(t|x)$ separately at each time point $t$ is not feasible, as time-expanded instances are no longer independent, leading to accumulated gradients from earlier time points. This computational difficulty is not captured in the formal mathematical adaptions. Similar methodological adaptations are standard practice in survival interpretability research (see Kovalev et al., (2020), Krzyzinski et al., (2023); even the time-dependent local accuracy metric in Krzyzinski et al., (2023) follows a similar extension from the original Shapley value axiom.\n\nFurthermore, another major contribution of our work are effective visualization and interpretation techniques for functional outputs tailored to different methods. This is particularly important given the ongoing debate and disagreement regarding gradient-based methods (Sturmfels et al., 2020; Krishna et al., 2023; Koenen et al., 2024). Our work contributes to this discussion by clarifying how implicit or explicit baselines in these methods influence survival explanations and provides practical guidance on selecting appropriate techniques based on their interpretability characteristics.\n\nFinally, we highlight our `R` package, `Survinng`, as an additional contribution. Existing libraries like `Captum` and `innsight` do not natively support survival DNNs, necessitating custom implementations. Our package, which includes all described gradient-based XAI methods for DeepSurv, DeepHit, and CoxTime, will be made available with the final version of this paper.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "No theoretical claims are presented in this paper.", "experimental_designs_or_analyses": "Yes.", "relation_to_broader_scientific_literature": "While Langbein et al. (2024) review many methods to interpret survival models, gradient-based methods are briefly discussed in this review.  This paper focuses more on gradient-based methods, and the contribution of this paper can be seen as extensive experiments on comparative study of gradient-based methods for survival models.", "essential_references_not_discussed": "None.", "other_strengths_and_weaknesses": "A weakness of this paper is that the technical contribution is marginal.  While there are many methods to interpret $y=f(x)$ for the standard regression analysis where $x$ is a feature vector and $y$ is a target value, this paper shows only adaptations of these methods to interpret $p=S(t|x)$ where $t$ is a time point and $p$ is a probability and the adaptations are almost straightforward.  In other words, this paper does not show any novel idea associated with applying the interpretation methods for the standard regression analysis to survival analysis.  (This is in contrast with (Krzyzinski et al., 2023), which proposes a modification of an evaluation metric specifically designed for survival analysis.  The modification is described in the appendix of this paper: from Equation (9) for the standard regression analysis to Equation (10) for survival analysis.)\n\nA strength of this paper can be seen as explicitly providing the adaptations (as summarized in Figure 2) and shows effectiveness of these adapted methods.   The results reported in the experiments (in Section 5) are reasonable and convincing.  For example, this paper shows the effectiveness of using the gradient-based method, GradShap(t), compared with existing model-agnostic methods SurvSHAP(t) and SurvLIME.", "other_comments_or_suggestions": "None.", "questions_for_authors": "None.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Fn5MEv4p4t", "reviewer": "Reviewer_5uUz", "summary": "The paper benchmarks previously proposed gradient-based explanation methods across three previously proposed deep survival analysis methods. Experimental results on synthetic and real-world datasets highlight differences in performance across scenarios.", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable and insightful feedback. Before addressing your concerns in detail, we want to clarify a few crucial aspects and potential misunderstandings:\n* This is **not** a benchmark paper. \n* We do provide the definition and explanation of local accuracy in the appendix. \n* We do provide comprehensive results (as far as meaningful) for **all experiments** and **all methods** and for **aggregated** metrics.  \n* We do compare with SurvLIME (contrary to the point you raised).\n\n**R4A1) What is the objective of this paper?**\n\nThe objective of our paper is explicitly stated to **not** be a benchmark paper comparing different gradient-based explanation methods for survival DNNs (see XAI benchmark papers by Liu et al., 2021 and Agarwal et al., 2022 for classic prediction models). Instead, we adapt these methods for survival DNNs, provide method-specific visualizations and interpretations (addressing the recent disagreement problem), and compare GradSHAP(t) as a flexible model-specific version to SurvSHAP(t), making the calculation of Shapley values for survival DNNs possible (see **R1A3 and R3A1**). \n\n**R4A2) Why no comprehensive (aggregated) evaluations for all experiments? Only synthetic data?**\n\nA comprehensive evaluation across all methods is not feasible because XAI methods pursue different goals and lack definitive \"correct\" explanations. For gradient-based methods, this disagreement problem (Sturmfels et al., 2020; Krishna et al., 2023) origins from varying implicit or explicit baselines, making direct comparisons on a local level unreliable (e.g., Grad(t) measures output-sensitivity, GradxInput(t) attributes implicitly against a zero baseline, etc.). We evaluate the methods as local explanation techniques and highlight their characteristics using the introduced visualizations (see Appendix A.1 and A.2 for our comprehensive results). A meaningful aggregation of results across all instances is only possible for Shapley-based methods (as we did for two measures in Sec. 5.2). However, aggregation compromises the local nature of the explanations as it leads to global measures.\n\n Additionally, we mainly used synthetic data for the comprehensive evaluation since it is the only reliable way to verify – in a controlled environment – if the model identifies truly prediction-relevant local effects based on the method's goal. Instead, a benchmark would focus on comparing methods against each other.\n\n**R4A3) Marginal technical contribution?**\n\nPlease refer to the detailed response for **Reviewer 3 R3A1)**.\n\nWe acknowledge that we may not have effectively communicated our intended contributions (adoptions for survival including challenges, visualizations and package implementation). In the final paper, we will emphasize and explain these contributions better.\n\n**R4A4) Concerns about local accuracy and advantage of GradSHAP(t)**\n    \nThe local accuracy measure for survival outcomes is defined and explained in detail in Appendix A.3.1. We acknowledge the importance of this metric and will incorporate it into the main text. It measures the average decomposition quality of the local attributions of SHAP-based methods (i.e., decomposing $\\hat{S}(t|x)-E(\\hat{S}(t|x))$). The primary advantage of our approach (GradSHAP(t)) lies in its drastically improved runtime efficiency compared to SurvSHAP(t). In addition to the results provided in our paper in Sec. 5.2. Fig. 6 and Appendix A.3.2., we now conducted an additional runtime comparison between SurvSHAP(t) and GradSHAP(t) as discussed in responses to **Reviewer 1: R1A1 and R1A3)**.\n\n**R4A5) Comparison to SurvLIME and other non time-varying methods.**\n\nWe compared GradSHAP(t) with SurvLIME in terms of global feature importance ranking (see Sec. 5.2, Fig. 7, and A.3.3). While SurvLIME estimates local feature importance values, SurvSHAP(t) and GradSHAP(t) provide local attributions. A direct comparison is therefore only meaningful by evaluating the resulting feature ranking across all instances.\n\nIn the related work, we purposefully only explicitly mention survival XAI methods, which are comparable to the gradient-based explanations discussed in the paper. Other survival explainability methods which do not provide time-dependent, local feature attributions are beyond the scope of our paper, including Kovalev et al. (2021) and Utkin et al. (2022), but are discussed in great detail in the referenced comprehensive review by Langbein et al. (2024).", "claims_and_evidence": "- The paper claims an extension of previously proposed GradSHAP to GradSHAP(t) as a key contribution. However, it seems like a straightforward application of GradSHAP to survival outcomes. It is unclear what the actual contributions or challenges unique to survival time GradSHAP(t) address.\n- The paper claims that GradSHAP(t) offers a better balance in computational efficiency and accuracy. However, only Figure 6 is provided to support this claim, and the definition of local accuracy is not provided in the paper. Additionally, it is unclear why alternative approaches such as SurvLIME are not discussed.\n- The paper claims that gradient-based explanation methods effectively identify prediction-relevant features. Unfortunately, most experiments are based on synthetic data, with only Grad(t) and Grad(t) × Input(t) benchmarked on preselected instances. It is unclear why comprehensive evaluations of all gradient-based approaches, including GradSHAP(t), are not provided, with evaluations summarized across all instances.", "methods_and_evaluation_criteria": "- The proposed approach, GradSHAP(t), is a straightforward extension of the previously proposed GradSHAP to survival outcomes.\n- The evaluation criteria on synthetic data is based on two preselected instances, which do not comprehensively capture the variance across test examples. Additionally, it is unclear what local accuracy in survival outcomes entails (Figure 6). \n- The comparisons are not consistent across the deep survival methods and the gradient-based approaches across experimental settings.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "- Given that this is a benchmarking paper, the experimental results are underwhelming. The experimental comparisons are not consistent and seem cherry-picked. I encourage the author(s) to provide extensive analysis and results across all experiments, where all methods are included and results are summarized across all instances.", "relation_to_broader_scientific_literature": "- Gradient-based explanations for deep survival models are an important research area for clinical decision-making", "essential_references_not_discussed": "- The paper should also discuss other non-time-varying survival explanation approaches, e.g., Kovalev et al. (2021) and Utkin et al. (2022).", "other_strengths_and_weaknesses": "The writing could be improved to focus on key contributions. Also, the experimental section is difficult to follow given the inconsistency in the experimental setup.", "other_comments_or_suggestions": "**Minor**\n- Eqn 4: Should be $\\ln$ instead of $\\log$", "questions_for_authors": "- Could you provide comprehensive results for all experiments?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: After the discussion period, three reviewers favored acceptance (2 weak accept, 1 accept) and one favored rejection. In looking at the discussion, I would say that the authors have largely addressed the reviewer concerns, and the dissenting reviewer's concerns have been adequately addressed (in my opinion) by the authors' two responses to this reviewer. I get the impression that one of the major issues with the originally submitted draft was a lack of clarity of technical contributions, which was sufficiently addressed in the discussion period (see author comment "R3A1"); this sort of explanation really needs to be incorporated into the paper and made clear as early as possible. At the moment, I get the impression that some of the added results during the discussion period and some of the clarifications would definitely improve the paper draft but also constitute somewhat significant paper edits that arguably might warrant another round of reviewing. I am recommending a cautious "weak accept".
Decision: Accept (poster)