Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors formulated the dynamic survival analysis problem as modeling the distribution of hitting time for a terminal state given sequences of states. Under the assumption that the sequences of states follow a Markov chain, the survival distribution is determined by the transition matrix and can be computed by a simple recursive algorithm once the transition matrix is known. Without the access to transition matrix, the authors developed a temporally-consistent survival regression (TCSR) algorithm to estimate parametric models given sequences of states. The authors also compared the proposed method with the model only using initial-state information and landmark approach on both synthetic and real-world datasets.
strengths_and_weaknesses: Strengths:
* The authors made a very interesting connection between dynamic survival analysis and reinforcement learning. These insights can help take advantage of algorithms studied in reinforcement learning to estimate survival models.
* The manuscript is well written and provides the necessary background for general audiences, especially presenting the method step by step from a known transition matrix with nonparametric models, to a known transition matrix with parametric approximation, and finally to an unknown transition matrix. 
* Exploiting the temporal consistency helps improve data efficiency as shown in the experiments.

Weaknesses:
* The proposed method requires successive measurements of covariates at regular time points and finite state space, which is not often the case in practice. The authors have discussed the potential extension to irregularly-sampled, continuous-time sequences. Could the authors also comment on how generalizable the method is to continuous state space as the covariates can be continuous random variables?
* Another important approach in dynamic survival analysis, time-dependent Cox regression, is missing in the discussion and comparison. Time-dependent Cox regression is different from both landmarking and joint modeling. Could the authors discuss and compare it with the proposed method?
* How accurate the unique fixed point $\bar{Q}$ in Proposition 2 is to the true survival matrix?
* I don't think the proposed method is completely complementary to existing dynamic survival models. It does make an additional assumption, Markov chain, so as the temporal consistency. If I missed anything here, could the authors explicitly explain what's the impact of the Makov chain assumption on characterizing the survival distribution?
* Definition of the event time is not clear in line 91: I presume T=k means that the sequence reaches the terminal state at (k+t)-th step given that x_t=x at the t-th step? Because x_t is not defined until line 98.

questions: Please see the weaknesses above.
limitations: Yes, the authors have discussed limitations.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors propose a novel algorithm to estimate a discrete-time survival model by exploiting a temporal-consistency condition inspired by connections with literature on reinforcement learning. Focusing on survival analysis in the dynamic setting, this work proposes a theoretical framework together with a practical algorithm that fits a parametric survival model by using sample trajectories. At its core, this work argues in favor of the idea that temporal consistency can dramatically increase statistical precision within survival models. Empirically, the authors provide an evaluation of two real-world and two synthetic datasets.

strengths_and_weaknesses: Originality: 
To the best of my knowledge, the authors propose a novel algorithm introducing an interesting methodological advancement in the field of survival analysis. The authors clearly acknowledge related literature by highlighting common challenges in the field and how their contributions are able to fill these gaps. Clear introduction, interesting idea, and nice organization.
I also appreciated the parallelism with RL methods, both in Section 3.4 and the appendix.

Quality: 
The authors clearly made an effort to accurately describe the general setting and methodology. Overall, I find the idea quite elegant in its simplicity. Below are some comments/questions which I'd like the authors to address (in no particular order):

- I feel like the empirical analysis is characterized by a rather short discussion as opposed to the methodological contribution which is very clearly explained. Therefore, I believe the manuscript would gain from a qualitative discussion of the results (whenever possible): what should the reader take away from the experiments? how do these results confirm the theoretical contributions? For example, when speaking of quantitative results, the authors don't explain why the performance of TCSR degrades with a larger number of sequences (Figure 2, bottom left). Is this only due to randomness? Why does it seem to be so consistent in the PBC2 dataset? (i.e., across all k-folds). I believe that elaborating on these extreme cases (both positive and negative) can elevate the quality of the entire paper.
- In Figure 2, the authors show the performance of the three implemented methods across three datasets. Is there a reason why the "#sequences" stops at 200? I might be misunderstanding something, but from Table 1, it seems like both real-world datasets could potentially allow for more than 200 sequences (even considering the implemented k-fold cross-validation)
- In Section 4.2, the authors use the MLE estimate for the regression coefficients as gold-standard, which is clearly a valid approach. To bring this reasoning to an extreme, would it be possible to generate a survival analysis dataset from a set of *known* parameters and check whether these methods are able to converge to the "absolute-true" values? If possible, I believe this scenario would greatly benefit the manuscript.

Clarity: 
The paper is well organized and easy to read. All visualizations are clear and effective.

Significance: 
The authors address an interesting problem, with novel ideas. The approach shows clear theoretical and practical advantages compared to traditional methods. However, I believe the current version of the paper could be more effective in justifying the contributions through a more detailed experiment analysis.
questions: Mentioned in the main body of the review.
limitations: The authors address the limitation of their work at a relatively high level.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors borrow the idea from temporal difference learning to model sequential survival analysis with the temporal consistency. The authors discretize the time and covariates and prove exact solutions with dynamic programming and also provide approximate solutions with parametric models. The authors show that their method has sample efficiency and better predictive performance on empirical studies. 
strengths_and_weaknesses: Strengths:
1. The authors provide a framework to model temporal consistency in sequential survival predictions.
2. The authors describe the problem in a sequential decision way and provide dynamic programming and parametric solutions. I think the solution is elegant. I personally thought about the problem before and got confused on RNN models without temporal consistency. The authors here solve my concerns with a simple setting.
3. The authors show the benefit on both sample efficiency and better predictive performance on three datasets.
4. The writing is clear to me.

Weakness:
1. There is one thing that confuses me. It looks that the authors use finite number of states in model description. But in real world, the features are usually continuous so we have infinite amount of states. How do the authors handle continuous covariates in the real-world studies?
questions: 1. How do the authors handle continuous covariates in the real-world studies?
2. Survival times are usually continuous. Can the authors' model handle continuous times?
3. At each time step, we may have a history of covariates (covariates in previous time steps). How do the authors' framework handle the history of covariates? Maybe using the hidden state of a RNN as a continuous state?
limitations: The proposed models assume the number of states is finite. This may be my misunderstanding. Feel free to correct me.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper presents a new algorithm for survival analysis with time-varying features.  This paper exploits a similarity between survival analysis and reinforcement learning, and the proposed algorithm can be seen as a natural adaptation of the temporal-difference learning in reinforcement learning to the survival analysis.

strengths_and_weaknesses: == Strengths ==
+ The major contribution of this paper is to establish the connection between the survival analsysis with time-variying features and the reinforcement learning.  By using the connection, this paper shows a new algorithm TCSR, which is based on the temporal-difference learning in reinforcement learning.  I think that this contribution is beneficial for the community of survival analysis.
+ Related to the previous point, this paper shows a new approach for survival analysis with time-varying features.  Usually, we estimate the outcome for a single state $x$ in survival analysis.  However, this paper shows that it is better to estimate the difference of outcomes between two consecutive states $x$ and $x'$.  This idea is known as temporal-difference learning in reinforcement learning, but it is a new idea in survival analysis.


== Weakness ==
+ Propositions 1-3 supports the superiority of the new algorithm TCSR over the known algorithm (i.e., Landmarking) in the experiments described in Section 4.1.  However, these propositions have less relationship with the experimental results described in Section 4.2, because each state $x$ appears only once in the datasets used in Section 4.2.  The difference comes from the fact that discrete states were used in Section 4.1 while the states are created based on feature vectors with continuous values.  I think, this paper implicitly assumes that, if two states $x$ and $x'$ are similar (i.e., $\phi_{x} \approx \phi_{x'}$), then the true survival distribution $q_{x}$ and $q_{x'}$ are similar.  However, this paper does not state this assumption explicitly.
+ The experiments described in Section 4.1 can be improved.  Since this paper uses a synthetic dataset in Section 4.1, we can calculate the true probability distribution $q_{x}$ for each state $x$.  Therefore, we can evaluate the outputs of the algorithms by directly comparing with the true probability distribution.  I think that the Kendall-$\tau$ (and C-index) is an appropriate metric only when we do not know the true probability distribution.
+ The presentation of this paper can be improved.  For example, $x_{t}$ in Lines 91-92 is not defined (and its definition seems different from $x_{t}$ defined in Lines 97-100).  For another example, $c_{t}$ in Line 122 is undefined, although $c$ in Line 123 is defined in Line 100.  In equations between Lines 51-52 in Section A.1, the $x$ in the third line and $x'_{m}$ in the fourth line are undefined.
+ I think there are some errors in mathematics.  In Lines 27-28 in Section A.1, the sum over $K=1,2,\ldots,K+1$ in the second term of the second equality should be $K=2,3,\ldots,K+1$.  In Lines 28-29 in Section A.1, $w_{xk}(\theta')$ must be zero when $k \geq 2$, because $h_{\theta'}(\ell | x') = 1$ when $\ell=0$ by the definition of $h$ in Lines 93-94 in Section 2.


== Minor issues ==
+ It is better to clarify that $T$ in Line 87 is discrete.
+ I am not sure if it is appropriate to use the terminology "converges" in Proposition1 (and Proposition 2).  Proposition 1 only shows how to compute $Q$ from the transition probabilities $p(x'|x)$ by using dynamic programming assuming that we know the true transition probabilities.
+ There are many grammatical errors (e.g., "is reaches" -> "it reaches" in Line 99, "describe" -> "describes" in Line 113, and "equivalent" -> "equivalent to" in Line 213).
+ I think that it is better to rephrase "estimate survival" with "estimate survival distribution" in Line 94.  (Note also that similar phrases appear several times after Line 94.)

questions: I have no questions.

limitations: Limitations are not clearly stated.  Lines 227-230 in Section 3.3 might describe the limitation, but I could not find any description on limitation in Section 4.
review_title: null

Meta-review: metareview: This paper draws connections between survival analysis where covariates appear over time and td-learning in reinforcement learning. The connection is based on a recursion in the survival distribution when the process satisfies a Markov assumption. The paper then uses this connection to develop a new estimator for survival analysis using dynamic programming. The paper has the potential to connect two disparate communities in ML. From a technical standpoint, I am interested in what happens when the Markov assumption is weakened. 
confidence: Certain
Decision: Accept