Temporally-Consistent Survival Analysis
Authors: Lucas Maystre
Daniel Russo
Venue: NeurIPS 2022 Accept (2022)
Keywords: survival analysis
time-to-event
dynamic survival
temporal-difference learning
policy evaluation

Abstract: We study survival analysis in the dynamic setting: We seek to model the time to an event of interest given sequences of states. Taking inspiration from temporal-difference learning, a central idea in reinforcement learning, we develop algorithms that estimate a discrete-time survival model by exploiting a temporal-consistency condition. Intuitively, this condition captures the fact that the survival distribution at consecutive states should be similar, accounting for the delay between states. Our method can be combined with any parametric survival model and naturally accommodates right-censored observations. We demonstrate empirically that it achieves better sample-efficiency and predictive performance compared to approaches that directly regress the observed survival outcome.

Temporally-Consistent Survival Analysis
Lucas Maystre
Daniel Russo Columbia University Spotify djr2174@gsb. columbia. edu
Spotify
Jucasm@apotify.ccm
Abstract
We study survival analysis in the dynamic setting: We seek model the time an event of interest given sequences of states. Taking inspiration from temporal- difference learning, a central idea in reinforcement learning, we develop algorithms that estimate a discrete- timeetttttmeeeeeee survival model by exploiting a condition. Intuitively, this condition captures the fact that the survival distribution at consecutive states should be similar, accounting for the delay between states. Our method can be combined with any parametric survival model and naturally accom- modates right-censored observations. We demonstrate empirically that it achieves better sample-efficiency and predictive performance compared to approaches that directly regress the observed survival outcome.
Introduction
Survival analysis provides a statistical framework for reasoning about the time until an event of interest occurs 220 D It is used in wide range of applications, including modeling patient outcomes in clinical studies 25 2, predicting customer retention in subscription-ber based businesses 14] 77. and many more. In the simplest setting, are given dataset mapping instances, described by a vector of covariates, to observations about time-to-event. Several aspects distinguish survival analysis from simple regression. The event may not always occur within the observed time-window, leading to so-called censored data which contain useful information but need to be treated differently from fully observed data. Furthermore, the focus is often on estimating the rate at which the event occurs, also known as the risk or hazard, as opposed to obtaining pointwise prediction about the time-to-event
Across applications, it is becoming increasingly commonplace collect multiple measurements time. In this dynamic setting, the data consist of sequences of states instead of single, static vector of covariates. From an initial state indicating, for instance, features of a patient and choice of medical treatment, we might observe not just the survival time but rich information on the evolution their health. For registered user of a subscription service, observe not just how long they remain subscriber but also how their usage evolves over time. We ask the question: Can we take advantage of sequential data to improve survival predictions? Existing work has mostly focused on building representations summarizing measurements up to given point in time [34 [2], and on modeling the joint distribution of sequences and time-to-event explicitly [35] [7). In this paper, we contend that there is different, complementary way to capitalize on dynamic survival data.
We study this problem formally through discrete-time model. We begin by assuming that sequences of states follow a Markov chain. We model the event interest as a special terminal state, and we seek to estimate the survival distribution (i.e., the distribution of the hitting time for that terminal state) from any other state. Drawing on ideas from dynamic programming [.) we take advantage of temporal-consistencc condition. The survival distribution at a given state can be written combination of a) the probability of reaching the terminal state at the next step, and b) the survival distribution the next state. Given full knowledge of the Markov transition matrix, the survival distribution from any state can be computed exactly with a simple recursive algorithm.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Building on these insights. we then develop a practical algorithm that fits a parametric survival model by using sample trajectories. Our approach synthesizes camporal-different learning, class of algorithms studied in reinforcement learning 33 6. and survival regression. The algorithm can estimation, and, by exploiting temporal consistency, it can dramatically increase statistical precision in certain cases (Section 4. We also demonstrate our algorithm's 's effectiveness on synthetic and real-world datasets (Section 4.2)
be used as a drop-in replacement to maximum-likelihen
Beyond our algorithmic contributions, we hope our presentation may help bridge an intellectual divide between two academic communities. Our primary focus is on making use of the insights underlying temporal difference learning, which is one of the central ideas in reinforcement learning, within survival analysis. But it is also worth noting that many natural problems in reinforcement learning involve survival outcomes, such as the problem of maximizing the length of a session in a recommender system [9]. It natural to approach such problems way that leverages the decades of modeling insight reñlcced in the survival analysis literature. We sketch such extensions in the supplementary material.
Organization of the paper. In the remainder of this section, we cover related work and set our contributions in context. In Section 2 we introduce key concepts and notation. We then develop our algorithm in Section andd discuss connections between RI and dynamic survival analysis. We evaluate our approach empirically on simulated and real-world data in Section 4 Finally, we discuss future work and conclude in Section
1.1 Related work
Driven by the growing prevalence of longitudinal datasets, there has been significant interest in the problem of dynamic prediction in the literature survival analysis. Landmarking is popular method that consists of regressing the survival outcome on all intermediate states, adjusting the regression target based the time at which the covariates recorded [4 27.. The basic idea underpinning landmarking can be used in combination with methods that build (or learn) feature representations that better capture information about sequence's ' past up given state 34
29
22
13
Another line of
work suggests jointly modeling the sequence dynamics and the survival distribution 355 17 22]
Lee and Whitmore (23) view survival analysis by means of stochastic process reaching a boundary and explicitly consider the implications of the Markov property. Their model is reminiscent of the one we introduce in Section - At their core, all these approaches make use of the observed survival outcome as the prediction target. In contrast, we propose changing the prediction target by exploiting a temporal consistency condition. In that sense, our approach complementary and can be combined with existing dynamic survival models.
The dynamic survival problem we consider in this paper is analogous the so-called policy evaluation problem in reinforcement learning 33 One of the seminal ideas in the field is temporal-differeen this topic. The L.TTD)
learning 2 and our work is considerably influenced by the literature algorithm of Bradtke and Barto 66 is perhaps the closest work to ours; At a high-level, our algorithm survival regression what LSTD least-squares regression. We revisit the connections between our work, policy evaluation, and reinforcement learning more generally in Section 3.4 Most policy realar-valued function (the expected return from a state), but some evaluation schemes estimate authors introduce methods that estimate a full distribution at each state [24 4 12). In contrast to generic distributional RL algorithms, our work focuses on survival prediction. Our specific focus implies a canonical way of modeling the distribution and a canonical way representing the loss function: The algorithm we introduce in Section 3.3 reveals the central role played by the hazard function.
Preliminaries
We consider sequences drawn from Markov chain on a state space X,, with initial distribution Ta (#) and transition probabilities p(x'(z). We assume that X' contains special state of interest, which
we call the terminal state and denote by ø. We are interested in estimating the time T until the
sequence reaches the terminal state from any initial state * Ø. In the literature on stochastic
processes, this is commonly referred to as the hitting time of the terminal state 26 23 In this
paper, we use the terminology of survival analysis and refer to T as the time-to-event We call the

---

probability distribution of T the survival distribution. We describe survival distribution by using three interrelated functions:
probability mass function
[(k]z) P(T mu = =]]
survival function
S(k(z) = P > k xo =x], h(k[2) P T 2g z].
hazard function
is possible to express any one function in terms of any other, for example S(k|z) II
h(k[z)). With slight abuse of notation, we let f(k) Skk)2) = 0 for k 2 L, and
M(0/)) = f(0)=) = { = Ø
{ [ otherwise.
=
5(0)x))
otherwise,
In the remainder of the paper, we estimate survival up to some finite horizon K. That is, we are interested in the hazard probabilities h(l]z),.... h h(K[z) [zx), or equivalently, in ,.. [K)z), with ((K(z) capturing the remaining probability mass.
Datasets of sequences. In practice, will be estimating survival from sequences sampled from the Markov chain. We assume that each sequence & = (Zo. -1, (1,..,, .z) that we observe either a) ends as soon as it reaches the terminal state, b) has not yet reached the terminal state. sequence where It * Ø is called right-censored. We call c = 1[ + Ø the censoring indicator, and the index of the last observed state is the time-to-event or censoring time. We collect observed sequence into dataset 'D = {s,, n = 1,..., .N J. noting that sequences can be of different lengths. Given such a dataset, natural choice for the horizon K is the length the longest sequence.
2.1 Survival regression
In order to achieve better predictive performance given finite data, we often consider parametric approximations to the survival distribution. Generally, the hazard the most natural quantity to model. Assuming that each state x: is described by feature vector Φx E R,,, we approximate the hazard by a function hk(k)))) parametrized by 0, that is a function z through da. Intuitively, if two states x' share similar state representations (i.e., 02 (. then the two states should have similar survival distributions. Specific parametrization choices might link together the survival probabilities across states, across time, both.
The discrete-time Cox proportional-hazaree (PH) model g is widely used in practice. Letting 0 = (a. (3) where a E R K and / E R.,, the Cox PH model is given by
logit(hg 3 d. + akk
where logit(p) loglp/(1 p)) The elements of a correspond the baseline (state-independent) hazard log-odds, and the vector 33 describes the tüme--ndeppenent effect of the features. This model can be extended, c.g., by replacing the linear dependence on de with deep neural network [21]. As an alternative to the Cox PH model, some authors suggest making use of flexible distribution on the positive integers such as the beta- ceta-geometric, and rewrite the parameters of the distribution as linear or non-linear functions of dh [16 I8].
Given dataset D {sw, ) we can learn a parametric approximation by maximizing the likelihood of the model under the data. The maximum-likelihoud estimator (MLE) can be cast as a weighted binary classification problem with cross-entropy loss
20 9.
N
K
MLE E arg min
warld [yn (hg
(I)
where H[pllg] - (1 log(1-g) is the binary cross-entropy, Wns = =ftt, 2 k}. and the
target Vnk (1 k: A G2 = 0) is an indicator variable for the event. The MILE only depends on triplets of the form (No. f, c), i.e., it completely ignores intermediate states. Landmarking extends (T) by effectively augmenting the data to include triplets of the form (æz, e) for <
MMaiin a parallel to reinforcement learning methods, this is similar any- Monte Carlo methods 33
3

---

3 Methodology
In this section, we develop efficient algorithms that estimate the survival distribution from any state. Our approach takes advantage of the following key idea. By using the law of total probability and the Markov property, we can write functions characterizing the survival distribution as recursions. For all k 1, X, have
[(k2z) = Ex pl [ 1|2')}, S(k(z) = Ex 1=) 1|2')].
Intuitively, these identities capture a notion of temporal consistency. For example, the second one states that the probability of surviving k steps starting from given state should be equal to the steps starting from the next state, on average.
probability of surviving
We proceed in three steps. We begin by assuming that the Markov chain dynamics are known and present dynamic programming approach computing the exact survival distribution from every state (Section 3.1). Then, we extend the approach to fit parametric approximation to the survival distribution (Section 3.2 Finally, consider the setting where only have access to finite dataset of sequences, and we design a practical algorithm (Section 3.3 We discuss connections to RL and extensions in Section 3.4
We do so for didactic purposes, and in practice this assumption is not required to use the algorithm we present in Section
For simplicity, we assume that the state space is finite and has cardinality | X.
3.1 Exact solution via dynamic programming
We denote the horizon- K survival distribution from state as the + 1K++))ddimnnioond probability vector
E = [f([[r
f(K[z) S(K[z))
where S(K[z) = 1 - (k)x)) Given any model f(k(z)) (not necessarily matching the true
probability mass function) and the corresponding vector yz [f(1lz) ... f(K(z) S(K[=)).
we define the backshift operator 5 as
t- [f(0/=))
J(K - 11z) 5(K - 1/x)]
where /(0)z)) = 1(x = } and S(K - 1|x) j(K[z) + (KK)). Finally, we define the survival matrix Q" the X (K+1) row-stochastic matrix obtained by stacking rows q. and the transition matrix P as the as the X x X matrix such that Pax' = p(ø' |x). Equipped with these definitions, we can write variant of in matrix form as
P'=
This matrix identity suggests the following iterative scheme. Starting with a matrix Qo formed by an initial guess for the time-to-event probabilities fk(z))) we can iteratively compute
Q++1= = PQ..
(3)
The following proposition shows that applying this mapping K times recovers the true survival distribution Q', no matter what the starting point Qo is.
Propasition 1. For any initial survival distribution fo(k(z) and corresponding row-stochastic matrix Qo. the sequence (Q+);e defined by (3) converges to Q" after K steps.
Sketch of proof. We show, by induction i, that the first i columns of Q: match those of Q* As all survival matrices are now-stochastic, their K + columns must add up to exactly one, and K iterations are sufficient to ensure that Qx = Q*.

---

3.2 Approximate solution with a parametric model
So far, we have represented the survival distributions exactly, at every state, by letting the survival matrix Q range freely over all row-stochastic matrices. Alternatively, we might want to approximate the survival distribution at z by constraining it to a parametric form
[L]()
fa(K(z) So(K[z)]
-
such as the Cox PH model described in Section
In this case, propose to modify 3 by projecting each iterate onto the set Q= (Qa 0 ) of all feasible survival matrices. Given a distribution over states <(x), we define the projection operator II to be a function from R xx(+11) to Q satisfying
IQQ
arg min da (Q, Qa), QoEQ
de, - Σ (())Ml
(4)
where we overload H to denote the categorical cross-entropy [u]]]o] formally, NQ feasible matrix that closest to Q in weighted cross- cross-entropy sense. In the information geometry literature, I1Q is known as the reverse information projection of Q onto Q and closely related to maximum-likelihood estimation [1] Starting from any Qo E 2, we refine a parametric estimate to the survival matrix iteratively as
Σ k= We log ti. In-
K+1
Q++ = IIP
(5)
Whether (5) converges or not remains an open question in the general case, but we identify class of parametric families where convergence is guaranteed.
Assumption A1 (Separable linear model). The parameter set factors as θ = x x ((K)
where each ( is convex. For each x, k there exists a vector &a,k such that logit (ho (k|z)) = &I, ((k) depends on f (0(1), (0(1)...., rank for all k.
only through ((k), Assume En [xx(x)e (Z R has full
Assumption A2 (Non-degeneracy). The population MIEE arg ming (Q", Q9) exists.
Intuitively, a separable model constrains the survival distribution across states but not across time. To understand our second assumption, note that instances of our problem with K = 1 are similar to logistic regression with soft-labels. Logistic regression has degenerate behavior problems where the data are perfectly separable. We rule that out implicitly by assuming finite MLE.
A2 are satisfied, then, for any initial row-stochastic matrix Qu. the sequence (Q.) defined by converges unique fixed point Q after K steps.
Proposition !/A/AA
AM
Sketch of proof. We start by rewriting the projection operator in terms of the hazard function hg. We show by induction
that, for models satisfying A1-A2 there is unique 0 such that he, (k|zr) ha(k 2)) for all & < i and all æ. Given that Q2 is a function of heg we conclude that there is fixed point Q such that Q2 Q for all i K.
Note that Q minimizes the temporal inconsistency (Q, PÂ), which might be different from minimizing the divergence from the true survival matrix Q"
3.3 Temporally survival regression
In most applications, we usually do not have full access the transition probabilities p(æ'[z) |x). Instead, we only get to observe a dataset sequences D {sx n = I,...,N) generated as per Section 2 Building on the ideas developed above, we introduce consisel survival regression (TCSR), a practical algorithm that estimates a survival model from samples. We describe the procedure in Algorithm [
The algorithm takes as input pairs of successive states T ((x,, =m/)) that we obtain by breaking down the N sequences into M Σn, in one-step transitions. Starting with an initial guess for 0, the algorithm then repeatedly solves an optimization problem until convergence. instructive to
compare the optimization problem on Line & to the maximum likelihod estimator (T). Similarly to
5

---

Algorithm 1 Temporally-consister survival regression (TCSR).
Require: dataset of transitions T {(Im mm initial parameters 0.
1: repeat
2: for m 1, ......... M do
3: Vml f- 1 = }
Based on one-step observed outcome.
4: Wml 4-1
5: for k 2, ,,..,, K do
Based on predictions at x'n.
6:
Yink f- h1: 1 1|
7:
Wmmk -- Sg(k - 2 mm)
8:
f- arg ming ΣM Σ K
2k=1 k=1 Wmnk H ymuk (k(ak))))
9:
until 0 has converged
the MLE, our approach casts survival estimation as a weighted binary classification problem. Whereas the MLE regresses hard binary targets that are exclusively based on the observed time-to-event or censoring time, we regress combination of hard targets (whether the sequence terminates the next step, Line 3) and soft targets (predictions at the next state, L)n Similarly, our weights are functions of observations and predictions the next state.
The connection to the MLE has important implications in practice. It means that we can implement TCSR by reusing existing code for survival regression or binary classification. At each iteration, first compute weights and aseudo-targett based on a current estimate of the model, and we can then delegate the optimization problem on Line 8 to existing off fff-the-shell software [9]. It also suggests simple heuristic to reason about the running time: Given existing code that computes the MLE in time T, we expect L. iterations of TCSR take time (M (M/N)LT.
In addition to the perspective provided by the MLE, also possible to view TCSR through the lens of the approximate dynamic programming scheme we have developed in Section 3.2 In fact, TCSR is identical to (5) on a certain empirical Markov chain, as made precise by the following proposition.
Proposition 3. Algorithm is equivalent to the fixed-point iteration chain and state distribution
5 with the empirical Markov
p(x'(a)= = Em 1f2m =I = ////E.... 1mam= = =]
a (σ) = Σm 1Imm= =x)m=x///0 /M)MM.
Sketch of proof. A key ingredient consists of recasting the projection operator in (4) into weighted binary classification problem similar to that of Line in Algorithm| [
By using the fact that [(k[2:
((kk)))S(( ((kkk(1))))) and S(k|z) 1I - [1
-=kk(=)), we can rewrite the categorical cross-entropy
from a distribution z to 8 in terms of binary cross-entropy to the hazard probabilities:
H[2]| = k== log fo(k(æ)) = log So K (r))
Lk-1 26 log (k(x)) Σ 2x -
log[1 (k((k(z)))
EK ( log ho (k|1r)- - (2 K+1 =x/ log(1 - ho (k(x)))
= Eken wk H 110 [x]])]
where wi =k' 2k and ys zx/ Wu. If y we find that y h'(k h'kk- 1|=1) for 2 w1 1, and w/k = S'(k- 22x²)) for k 2 2. The remainder of the proof relies on a linearity property of the cross-entropy.
This connection is helpful to analyze TCSR. For example, the following result is an immediate corollary of Propositions 2 and 3
Corollary 4. If Q satisfies A2 on the empirical Markov chain defined in Proposition 3 then, for any initial parameters Og the sequence (0,)7 defined by successive iterations of Algorithm converges to unique fixed point @ after K iterations.
Note that many parametric families used to model survival in practice not satisfy the assumptions
we require to guarantee convergence. In particular, they are not separable across time and therefore
do not satisfy AT Nevertheless, our experiments suggest that TCSR is very stable in practice, and we have not encountered single case where it failed to converge (see also Section 4.2

---

Continuous state spaces. We began Section by assuming that the state space X' is finite. With Algorithm [] however, this assumption no longer necessary. In practice, by using smooth parametric functions, TCSR works with continuous state representations without any modifications.
3.4 Connections to RL & extensions
Our presentation synthesizes temporal difference learning one of the central intellectual ideas in RL- RL-and survival analysis. The consistency equations ② that form the basis of our approach are closely related to the Bellman equations in Markov decision processes [33 Sec. 3.5]. The fixed-point
iterations and (5) mirror (approximate) dynamic programming approaches to policy evaluation [
Sec. 2.2].
There are also striking parallels between TCSR and the LSTD algorithm of Bradtke and Barto 6 Both algorithms exploit temporal consistency by linking predictions at successive states, and they both solve a sequence of moximum-likellhonn recast as fixed-point iterations on an empirical Markov chain. Whereas LSTD learns a vallu-valuudd function that minimizes squared loss, TCSR learns iustribution-valuee function that minimizes a cross-entropy loss. LSTD can be generalized to account for observations more than one state ahead, variant called LATD(A). We can generalize TCSR in a similar way. providing a unified perspective on Algorithm| and the MILE (T. We sketch such generalization Appendix AA.3
problems on pseudo-observations. They can both be
In RI., the algorithms SARSA and Q-learning naturally extend the insight of temporal-differee learning problems with actions [33] We can provide similar generalization and learn survival distributions that depend on state-action pairs. By interleaving this estimation problem with a policy improvement step, we end up with a policy iteration algorithm that is tailored to survival problems.
We explore these extensions in Appendix A.33
4 Experimental evaluation
In this section we evaluate TCSR (Algorithm) empirically on synthetic and real-world data. Our main focus is data-efficiency: We seek to validate that we can exploit temporal consistency to learn better models with fewer samples. Throughout the section we approximate the survival distribution by learning the parameters 0 (a, () of Cox PH model
logit (k|z)] 8 dz + 0kk
where d₂ is feature vector describing state x. Restricting our attention to this model lets us focus on the impact the learning algorithm, independently of the impact of the parametric form of the model itself. We compare TCSR two alternative approaches to learning a survival model.
Initial state. We regress the observed outcome directly on the initial state of the sequence, by maximizing the logelikeli (I). This approach corresponds the static setting, as it only employs triplets of the form (Zo, t, c) and ignores information contained in the sequence of states. If the sequences of features (Øarg .......) are not Markov, this approach might produce models that are less biased.
Landmarking. We augment the dataset used in the initial state approach by including every interme- diate state and the remaining time-to to-eent or time-to- mene----cnno.nng.. i.e., triplets of the form (ze, t - É, c) for l < t. Parameter inference is carried out by maximizing the likelihood under the augmented dataset. This the dominant approach dynamic survival prediction in the biostatistics literature
27
We provide software library with a reference implementation TCSR the Python programming language well computational notebooks that enable reproducing the results presented in this section at lttpt:tt//// / h/tt:::///ithbb.. wwt.cooo/cooaoe
4.1 Statistical benefits of temporal consistency
We begin by providing concrete generative model that illustrates the benefits of exploiting temporal consistency (Figure [ left). In this problem, we seek determine the impact on survival of different treatments, corresponding to L different initial states. For each treatment, observe exactly
7

---

Estimated vs. true treatment effect
1.0
0.98
Landmarking
0.8
TCSR
Kendall-r
2
0.02
0.6
0.4
Pio
0.04
0.2
P-Pla
L
0.96
0.0
10a
10a
10a
treatments L
Figure I: Example illustrating the statistical benefits exploiting temporal consistency. Left: Markov chain generating the data. Right: performance of two estimators as the number of treatments grows.
10 sequences starting at the corresponding state; sequences that do not reach the terminal state within steps are right-censored. Starting from given initial state æ, the sequence transitions to Pra- From states
state a (healthy) with probability Pra- and to state (sick) with probability and time-to-event is geometrically distributed with probability 0.02 and 0.04, respectively. As such, the survival distribution at each of the treatment states is entirely determined by the transition probabilities {Pra]. which induce a natural ranking over treatments. In simple two-stage problem like this, transitions to state a or b are called surrogate outcomes (28) We set 6. to be the one-hot encoding of the state identifier and learn Cox-PH models by using TCSR and landmarking.
In Figure I (right) we report the correlation between the rankings induced by {Pea} and by the model coefficients { for increasing values of L, averaged over 10 realizations. We observe that TCSR is able to discriminate between the treatments increasingly better as L grows, despite the number of sequences per treatment remaining constant. Intuitively, as L grows, accurate models of survival from states and b by pooling data from all sequences. By leveraging the Markov assumption and regressing predictions at and on the initial states, we remove a large part of the statistical noise in the individual sequences, thus improving our ability to identify the effect of each treatment on survival. Approaches that directly regress the observed time-to-event outcome, such as landmarking, are fundamentally unable to make use of this data-pooling advantage.
are able to learn increasingly
4.2 Empirical performance on three datasets
Next, consider three dynamic survival datasets and evaluate the performance of models on two tasks of practical interest.
1. Rank instances by predicted time-to-event based on the initial state of the sequence, To. To this end, we score sequences by the hazard log-odds 3 d.o and measure the concordance index (CI, see Appendix B.T for precise definition). Random scores achieve CI of 0.5, while CI of corresponds to perfect agreement between scores and observed time-to-event.
2. Estimate the effect of covariates the time-to-event. In the Cox PH model, recall that the regression coefficient {3 captures the linear effect of the ith covariate on the hazard log-odds. We evaluate model by measuring the root-mean-soe mea---uun--- error (RMSE) between the regression coefficients /3 and gold-standard coefficients 3".
These two tasks are particularly relevant to clinical applications. For example, accurately ranking instances by time-to-event could help identify patients which are most at risk early on. In Ap- pendix B.3 we provide results on two additional metrics that capture model's ability to make accurate probabilistic predictions, the predictive log-likelihood and the integrated Brier score [15].
Of the three datasets we consider, two contain real data from clinical studies and are publicly available online [30] The first study tracks survival outcomes from 312 patients diagnosed with PBC, a rare liver disease [25] Patient features include static information such as gender and treatment, and 12 dynamic biomarkers that are measured at study entry and at follow-up visits. The second study follows 467 patients diagnosed with HIV / AIDS (2). In this study, a single dynamic biomarker, the CD4 cell count, is measured repeatedly. The two studies span multiple years, and follow-up visits occur at regular cadence. We discard the time between successive visits and simply consider the

---

Table 1: Summary statistics of three datasets studied in Section 4.2
Dataset
D
Median length
Max length
# static feat.
# dyn. feat.
Frac. censored
PBC2
25
312
5
16
3
12
55. %
AIDS
467
3
59.7%
RW
4
11
0
20
21.9%
PBC2
AIDS
RW
Concordance index
0.8
0.7
0.7
0.6
Initial state
0.60
Landmarking
0.6
TCSR
0.5
4.5
4.0
RMSB
3.5
0.4
3.0
2.5
0.2
50
100
150
200
50
100
150
200
50
100
150
200
# sequences
sequences
sequences
Figure 2: Empirical performance of learning algorithms three datasets. We report the mean and standard deviation over 5 splits. Concordance index: higher is better, RMSE: lower is better.
discrete-time sequences induced by successive measurements. We complement these clinical datasets with synthetic dataset based on 20-dimensional random walk model. At each step, the sequence terminates with a probability that depends on the location 20D space. Summary statistics for all datasets are presented in Table 1 and more details are provided in Appendix B.2
The feature vectors } are obtained by concatenating the static controls with the dynamic features. We define the gold-standard coefficients 8 as the MLE (L) under the full dataset. We train models on increasingly larger subsets of the data. We report the average performance obtained by using 5-fold cross-validation in Figure
We observe that landmarking and TCSR, the methods that make of the full sequences, outperform the static baseline on PBC2 and RW. On AIDS, landmarking performs poorly, while TCSR performs similarly (in terms of CI) or slightly better (in terms of RMSE) than initial state. Recall that this dataset has a single dynamic feature, and thus the information contained in the sequence dynamics is likely limited. Overall, TCSR generalizes systematically better than landmarking, e.g.. necessitating less than 4x fewer samples to reach given performance level on RW and converging strictly better model on PBC2. These findings suggest that the statistical benefits illustrated in Section extend to real-world data and lead to meaningful improvements to predictive performance.
Running time and convergence. In our implementation of TCSR, we delegate Line & in Algo- [ Newton-CC solver provided by the SciPy library BI. For simplicity, we have fixed the number of iterations to 30 across all datasets and experiments. but the algorithm often converges fewer iterations. Interestingly, the actual running time of TCSR is usually significantly less than 30x the running time of landmarking. We believe that this is because initializing the inner opti- mization (Line with the previous iterate accelerates the convergence of the Newton-CG optimizer substantially.

---

Conclusion & future work
Driven by insights from temporal-differene learning, we have introduced new algorithm that estimates a survival model in the dynamic-data setting. Our algorithm takes advantage of a temporal- consistency condition that we know, a priori, should be satisfied by the true survival distribution. Our approach be can combined with existing dynamic survival models and inference software, and it only requires changing the regression targets. As such, we believe that our approach will be of interest to researchers and practitioners alike.
We envision broad directions for future work. The first direction consists of extending our algorithmic approach to accommodate a wider range of use-cases. We believe that the extension to the competing risks setting [ is straightforward, and would simply entail extending the binary problem (surviving or not at each step) to multiclass problem reflecting the multiple possible outcomes. The extension imregulatlyssappled continuous-time sequences also worth considering. Given continuous time, stationary Markov process ztt), we can write temporal cemporal-consistency condition for any time interval A
S(t(x) = Ex'rpa (a)) [S(t A|x')].
where Pa(- (-(r)) is the probability density function of x(t + A) conditional on xxt) = x. We have preliminary ideas on how turn this into practical algorithm, but more research needed. A second direction is to make use of our approach to address sequential decision- decision-making problems that are naturally expressed in terms of maximizing survival. This requires extending our developments to encompass problems with actions, a generalization we sketch in Appendix A.3
We thank Ksenia Konyushkova for feedback on early versions of the idea presented in this paper. We also thank Mounia Lalmas and the anonymous reviewers for their careful review and proofreading.
Acknowledgments.
References
[1] O. Aalen, Ø. Borgan, and H. K. Gjessing. Survival and Event History Analysis: A Process Point of View. Springer, 2008.
[2] D. I. Abrams, A. I. Goldman, C. Launer, J. A. Korvick, J. D. Neaton, L. R. Crane, M. Grodesky, S. Wakefield, K. Muth, S. Kornegay, al. A comparative trial of didanosine zalcitabine after treatment with zidovudine in patients with human immunodeficiency virus infection. New England Journal of Medicine, 330(10):657-62. 1994.
[3] Ö. Asar, J. Ritchie, P. A. Kalra, and P. J. Diggle. Joint modelling of repeated measurement and time-to-event data: An introductory tutorial. International Journal of Epidemiology, 44(1): 334-344, 2015.
[4] M. Bellemare, W. Dabney, and Munos. A distributional perspective on reinforcement learning. In Proceedings of ICML 2017, Sydney, Australia, Aug. 2017.
[5] D. P. Bertsekas and J. N. Tsitsiklis. Neuro- -enro-Dyaamic Programming. Athena Scientific, 1996.
[6] S. J. Bradtke and A. G. Barto. Linear least-squares algorithms for temporal difference learning. Machine Learning, 22:33-57, 1996.
[7] Chandar, B. St. Thomas, L. Maystre, V. Pappu, R. Sanchis Ojeda, T. Wu, Carterette, M. Lalmas, and T. Jebara. Using survival models to estimate user engagement in online experiments. Proceedings of WWW '22, Lyon, France, Apr. 2022.
[8] D. R. Cox. Regression models and life tables. Journal of the Royal Statistical Society, Series (Methodological), 34(2):187-202, 1972.
[9] E. Craig, C. Zhong, and R. Tibshirani. Survival stacking: Casting survival analysis as classification problem. Preprint, arXiv: 13480 [stat.ME], July 2021.
[10] I. Csiszár and F. MarúS. Information projections revisited. IEEE Transactions on Information Theory, 49(6): 1474-11944 2003.
10

---

[11] I. Csiszár and P. C. Shields. Information theory and statistics: A tutorial. Foundations and Trends in Communications and Information Theory, 1(4):417-528, 2004.
[12] W. Dabney, M. Rowland, M. Bellemare, and R. Munos. Distributional reinforcement learning with quantile regression. In Proceedings of AAAI 2018, New Orleans, LA, USA, Feb. 2018.
[13] Deasy, Liò, and A. Ercole. Dynamic survival prediction in intensive care units from heterogeneous time series without the need for variable selection or curation. Scientific Reports, 10(22129), 2020.
[14] S. Fader and B. G. S. Hardie. How to project customer retention. Journal of Interactive Marketing, 21(1):76-90, 2007.
[15] E. Graf, C. Schmoor, W. Sauerbrei, and M. Schumacher. Assessment and comparison of prognostic classification schemes for survival data. Statistics in Medicine, 18(1-118) 2545, 1999.
[16] J. J. Heckman and R. Willis. A beta-logistic model for the analysis of sequential labor force participation by married women. Journal of Political Economy, 85(1):27-58, 1977.
[17] R. Henderson, P. Diggle, and A. Dobson. Joint modelling of longitudinal measurements and event time data. Biostatistics, 1(4):465-48) 2000.
[18] D. Hubbard, B. Rostykus, Y. Raimond, and T. Jebara. Beta survival models. In Proceedings of AAAI Spring Symposium on Survival Prediction 2021, virtual event, Mar. 2021.
[19] E. Ie, Jain, Wang. S. Narvekar, Agarwal, R. Wu, H.-T. Cheng, M. Lustman, Gatto, P. Covington, J. McFadden, T. Chandra, and C. Boutilier. Reinforcement learning for slate- based recommender systems: tractable decomposition and practical methodology. Preprint, 1905v 12767 [cs. LG), May 2019.
[20] J. D. Kalbfleisch and R. L. Prentice. The Statistical Analysis of Failure Time Data. John Wiley & Sons, second edition, 2002.
[21] J. L. Katzman, U. Shaham, A. Cloninger, J. Bates, T. Jiang, and Y. Kluger. DeepSurv: Person- alized treatment recommender system using a Cox proportional hazards deep neural network. BMC Medical Research Methodology, 18(24), 2018.
[22] C. Lee, J. Yoon, and M. van der Schaar. Dynamic- A deep learning approach for dynamic survival analysis with competing risks based on longitudinal data. IEEE Transactions on Biomedical Engineering, 67(1): 2020.
[23] M.-L. T. Lee and Whitmore. Threshold regression for survival analysis: Modeling event times by a stochastic process reaching a boundary. Statistical Science, 21(4):501-513, 2006.
[24] T. Morimura, M. Sugiyama, H. Kashima, H. Hachiya, and T. Tanaka. Parametric return density estimation for reinforcement learning. Proceedings of UAI 2010, Catalina Island, CA, USA, July 2010.
[25] A. Murtaugh, E. R. Dickson, G. M. Van Dam, M. Malinchoc, M. Grambsch, A. L. Langworthy, and C. H. Gips. Primary biliary cirrhosis: Prediction of short-term survival based on repeated patient visits. Hepatology, 20 106, 1994.
[26] R. Norris. Markov Chains. Cambridge University Press, 1997.
[27] L. Parast, L. Tian, and T. Cai. Landmark estimation of survival and treatment effect a randomized clinical trial. Journal of the American Statistical Association, 109(505):344399)) 2014.
[28] R. L. Prentice. Surrogate endpoints in clinical trials: Definition and operational criteria. Statistics in Medicine, 8(4):431-440, 1989.
[29] R. Ranganath, A. Perotte, N. Elhadad, and Blei. Deep survival analysis. In Proceedings of Machine Learning for Healthcare 2016, Los Angeles, CA, USA, Aug. 2016.
11

---

[30] D. Rizopoulos. Joint Models for Longitudinal and Time-to-Evee Data With Applications in R.
CRC Press, 2012.
[31] SciPy 1.0 Contributors. SciPy 1.0: Fundamental algorithms for scientific computing in python. Nature Methods, 17:261-272. 2020.
[32] R. S. Sutton. Learning to predict by the methods of temporal differences. Machine Learning, 3: 9 9-44, 1988.
[33] R. S. Sutton and A. G. Barto. Reinforcement Learning: An Introduction. MIT Press, second edition, 2018.
[34] H. C. van Houwelingen. Dynamic prediction by landmarking in event history analysis. Scandi- navian Journal of Statisties, 34(1):70-85, 2007.
[35] M. S. Wulfsohn and A. A. Tsiatis. A joint model for survival and longitudinal data measured with error. Biometrics, 53:330-339, 1997.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Sections 3.3 and
(c) Did you discuss any potential negative societal impacts of your work? [N/]] Based on our understanding of NeurIPS Ethics Guidelines, there are no outstanding potential negative societal impacts that should be discussed, beyond general & well-documented risks associated with the use of data- predictive methods for decision- makonn..
(d) Have you read the ethics review guidelines and ensured that your paper conforms them? [Yes] Our experiments make use of publicly available data obtained from two clinical studies involving human subjects. These data have been widely analyzed and discussed in the relevant medical communities as well as the statistics and ML communities, and to the best of our knowledge. they have stood up to scrutiny in the 28 years since their release. They do not contain any personally identifiable information.
2. If you are including theoretical results...
(a) Did you state the full set of assumptions all theoretical results? [Yes] We have strived to state all assumptions required to make our results self-contained while minimizing clutter.
(b) Did you include complete proofs of all theoretical results? [Yes] We provide proof sketches in Section 3 the main text and defer the full proofs Appendix A.I in the supplementary material.
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material code and instructions necessary to reproduce all experimental results reported in the paper (ntttts::// http://gtttbb///// com cwt.ub..com/ppptttttict -pweaiy--eeseeeechhepppaeeee
URL)? [Yes] We provide
(b) Did you specify the training details (e.g.. data splits, hyperparameters, how they were chosen)? [Yes] We describe the experimental procedure a high level in Section 4 Some of the details are deferred to the supplementary material and can be found in Appendix[B [ Interested readers also refer to the computational notebooks used to produce the experimental results, as we have made every effort to ensure that they are clear and accessible.
(c) Did you report error bars (e.g., with respect to the random seed after running exper- iments multiple times)? [Yes] The results in Figure I (right) report the mean and standard error over 10 random seeds. The results in Figure report the mean and standard error obtained with 5-fold cross- cross-valdattin
12

---

(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] We include computational details
in Appendix B
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes] We use data collected through two clinical studies [25] 2].
(b) Did you mention the license of the assets? [Yes] The clinical data are available, e.g., through the JM R package that is licensed under GPLv2.
(c) Did you include any new assets either in the supplemental material or as a URL? Yes] We provide an implementation of TCSR as Python library https: // /tte::////ttiii thub com/ rpotafy-researchh tdsurv
(d) Did you discuss whether and how consent obtained from people whose data you'le 're using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
"See: http::/..........ooog/////aaaa/a
13


Reviews:
[{"review_id": "pLrJcjCSI2C", "reviewer": "Reviewer_fEGu", "summary": "The authors propose a novel algorithm to estimate a discrete-time survival model by exploiting a temporal-consistency condition inspired by connections with literature on reinforcement learning. Focusing on survival analysis in the dynamic setting, this work proposes a theoretical framework together with a practical algorithm that fits a parametric survival model by using sample trajectories. At its core, this work argues in favor of the idea that temporal consistency can dramatically increase statistical precision within survival models. Empirically, the authors provide an evaluation of two real-world and two synthetic datasets.\n", "questions": "Mentioned in the main body of the review.", "limitations": "The authors address the limitation of their work at a relatively high level.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Originality: \nTo the best of my knowledge, the authors propose a novel algorithm introducing an interesting methodological advancement in the field of survival analysis. The authors clearly acknowledge related literature by highlighting common challenges in the field and how their contributions are able to fill these gaps. Clear introduction, interesting idea, and nice organization.\nI also appreciated the parallelism with RL methods, both in Section 3.4 and the appendix.\n\nQuality: \nThe authors clearly made an effort to accurately describe the general setting and methodology. Overall, I find the idea quite elegant in its simplicity. Below are some comments/questions which I'd like the authors to address (in no particular order):\n\n- I feel like the empirical analysis is characterized by a rather short discussion as opposed to the methodological contribution which is very clearly explained. Therefore, I believe the manuscript would gain from a qualitative discussion of the results (whenever possible): what should the reader take away from the experiments? how do these results confirm the theoretical contributions? For example, when speaking of quantitative results, the authors don't explain why the performance of TCSR degrades with a larger number of sequences (Figure 2, bottom left). Is this only due to randomness? Why does it seem to be so consistent in the PBC2 dataset? (i.e., across all k-folds). I believe that elaborating on these extreme cases (both positive and negative) can elevate the quality of the entire paper.\n- In Figure 2, the authors show the performance of the three implemented methods across three datasets. Is there a reason why the \"#sequences\" stops at 200? I might be misunderstanding something, but from Table 1, it seems like both real-world datasets could potentially allow for more than 200 sequences (even considering the implemented k-fold cross-validation)\n- In Section 4.2, the authors use the MLE estimate for the regression coefficients as gold-standard, which is clearly a valid approach. To bring this reasoning to an extreme, would it be possible to generate a survival analysis dataset from a set of *known* parameters and check whether these methods are able to converge to the \"absolute-true\" values? If possible, I believe this scenario would greatly benefit the manuscript.\n\nClarity: \nThe paper is well organized and easy to read. All visualizations are clear and effective.\n\nSignificance: \nThe authors address an interesting problem, with novel ideas. The approach shows clear theoretical and practical advantages compared to traditional methods. However, I believe the current version of the paper could be more effective in justifying the contributions through a more detailed experiment analysis.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "aq8qbt5s5LU", "reviewer": "Reviewer_mhhX", "summary": "The authors borrow the idea from temporal difference learning to model sequential survival analysis with the temporal consistency. The authors discretize the time and covariates and prove exact solutions with dynamic programming and also provide approximate solutions with parametric models. The authors show that their method has sample efficiency and better predictive performance on empirical studies. ", "questions": "1. How do the authors handle continuous covariates in the real-world studies?\n2. Survival times are usually continuous. Can the authors' model handle continuous times?\n3. At each time step, we may have a history of covariates (covariates in previous time steps). How do the authors' framework handle the history of covariates? Maybe using the hidden state of a RNN as a continuous state?", "limitations": "The proposed models assume the number of states is finite. This may be my misunderstanding. Feel free to correct me.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The authors provide a framework to model temporal consistency in sequential survival predictions.\n2. The authors describe the problem in a sequential decision way and provide dynamic programming and parametric solutions. I think the solution is elegant. I personally thought about the problem before and got confused on RNN models without temporal consistency. The authors here solve my concerns with a simple setting.\n3. The authors show the benefit on both sample efficiency and better predictive performance on three datasets.\n4. The writing is clear to me.\n\nWeakness:\n1. There is one thing that confuses me. It looks that the authors use finite number of states in model description. But in real world, the features are usually continuous so we have infinite amount of states. How do the authors handle continuous covariates in the real-world studies?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "8q7Z98YiWvy", "reviewer": "Reviewer_VqoZ", "summary": "The authors formulated the dynamic survival analysis problem as modeling the distribution of hitting time for a terminal state given sequences of states. Under the assumption that the sequences of states follow a Markov chain, the survival distribution is determined by the transition matrix and can be computed by a simple recursive algorithm once the transition matrix is known. Without the access to transition matrix, the authors developed a temporally-consistent survival regression (TCSR) algorithm to estimate parametric models given sequences of states. The authors also compared the proposed method with the model only using initial-state information and landmark approach on both synthetic and real-world datasets.", "questions": "Please see the weaknesses above.", "limitations": "Yes, the authors have discussed limitations.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* The authors made a very interesting connection between dynamic survival analysis and reinforcement learning. These insights can help take advantage of algorithms studied in reinforcement learning to estimate survival models.\n* The manuscript is well written and provides the necessary background for general audiences, especially presenting the method step by step from a known transition matrix with nonparametric models, to a known transition matrix with parametric approximation, and finally to an unknown transition matrix. \n* Exploiting the temporal consistency helps improve data efficiency as shown in the experiments.\n\nWeaknesses:\n* The proposed method requires successive measurements of covariates at regular time points and finite state space, which is not often the case in practice. The authors have discussed the potential extension to irregularly-sampled, continuous-time sequences. Could the authors also comment on how generalizable the method is to continuous state space as the covariates can be continuous random variables?\n* Another important approach in dynamic survival analysis, time-dependent Cox regression, is missing in the discussion and comparison. Time-dependent Cox regression is different from both landmarking and joint modeling. Could the authors discuss and compare it with the proposed method?\n* How accurate the unique fixed point $\\bar{Q}$ in Proposition 2 is to the true survival matrix?\n* I don't think the proposed method is completely complementary to existing dynamic survival models. It does make an additional assumption, Markov chain, so as the temporal consistency. If I missed anything here, could the authors explicitly explain what's the impact of the Makov chain assumption on characterizing the survival distribution?\n* Definition of the event time is not clear in line 91: I presume T=k means that the sequence reaches the terminal state at (k+t)-th step given that x_t=x at the t-th step? Because x_t is not defined until line 98.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "59EPw-ax7k", "reviewer": "Reviewer_hyyi", "summary": "This paper presents a new algorithm for survival analysis with time-varying features.  This paper exploits a similarity between survival analysis and reinforcement learning, and the proposed algorithm can be seen as a natural adaptation of the temporal-difference learning in reinforcement learning to the survival analysis.\n", "questions": "I have no questions.\n", "limitations": "Limitations are not clearly stated.  Lines 227-230 in Section 3.3 might describe the limitation, but I could not find any description on limitation in Section 4.\n", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "== Strengths ==\n+ The major contribution of this paper is to establish the connection between the survival analsysis with time-variying features and the reinforcement learning.  By using the connection, this paper shows a new algorithm TCSR, which is based on the temporal-difference learning in reinforcement learning.  I think that this contribution is beneficial for the community of survival analysis.\n+ Related to the previous point, this paper shows a new approach for survival analysis with time-varying features.  Usually, we estimate the outcome for a single state $x$ in survival analysis.  However, this paper shows that it is better to estimate the difference of outcomes between two consecutive states $x$ and $x'$.  This idea is known as temporal-difference learning in reinforcement learning, but it is a new idea in survival analysis.\n\n\n== Weakness ==\n+ Propositions 1-3 supports the superiority of the new algorithm TCSR over the known algorithm (i.e., Landmarking) in the experiments described in Section 4.1.  However, these propositions have less relationship with the experimental results described in Section 4.2, because each state $x$ appears only once in the datasets used in Section 4.2.  The difference comes from the fact that discrete states were used in Section 4.1 while the states are created based on feature vectors with continuous values.  I think, this paper implicitly assumes that, if two states $x$ and $x'$ are similar (i.e., $\\phi_{x} \\approx \\phi_{x'}$), then the true survival distribution $q_{x}$ and $q_{x'}$ are similar.  However, this paper does not state this assumption explicitly.\n+ The experiments described in Section 4.1 can be improved.  Since this paper uses a synthetic dataset in Section 4.1, we can calculate the true probability distribution $q_{x}$ for each state $x$.  Therefore, we can evaluate the outputs of the algorithms by directly comparing with the true probability distribution.  I think that the Kendall-$\\tau$ (and C-index) is an appropriate metric only when we do not know the true probability distribution.\n+ The presentation of this paper can be improved.  For example, $x_{t}$ in Lines 91-92 is not defined (and its definition seems different from $x_{t}$ defined in Lines 97-100).  For another example, $c_{t}$ in Line 122 is undefined, although $c$ in Line 123 is defined in Line 100.  In equations between Lines 51-52 in Section A.1, the $x$ in the third line and $x'_{m}$ in the fourth line are undefined.\n+ I think there are some errors in mathematics.  In Lines 27-28 in Section A.1, the sum over $K=1,2,\\ldots,K+1$ in the second term of the second equality should be $K=2,3,\\ldots,K+1$.  In Lines 28-29 in Section A.1, $w_{xk}(\\theta')$ must be zero when $k \\geq 2$, because $h_{\\theta'}(\\ell | x') = 1$ when $\\ell=0$ by the definition of $h$ in Lines 93-94 in Section 2.\n\n\n== Minor issues ==\n+ It is better to clarify that $T$ in Line 87 is discrete.\n+ I am not sure if it is appropriate to use the terminology \"converges\" in Proposition1 (and Proposition 2).  Proposition 1 only shows how to compute $Q$ from the transition probabilities $p(x'|x)$ by using dynamic programming assuming that we know the true transition probabilities.\n+ There are many grammatical errors (e.g., \"is reaches\" -> \"it reaches\" in Line 99, \"describe\" -> \"describes\" in Line 113, and \"equivalent\" -> \"equivalent to\" in Line 213).\n+ I think that it is better to rephrase \"estimate survival\" with \"estimate survival distribution\" in Line 94.  (Note also that similar phrases appear several times after Line 94.)\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept