Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This work proposes DeepEH, a survival analysis framework that formulates the hazard function as \lambda(t|X) = \lambda_0(t.exp(h_1(x))).exp(h_2(x)). This formulation has the Cox PH and AFT as special cases. The authors estimate h_1 and h_2 by neural networks and provide theoretical support for the consistency and convergence rate of the resulting survival function estimator.
main_review: 1) I saw a few errors in the English and a few typos: "adopt a parametric approaches", "are asymptotically consistent 90 and enjoys fast convergence rates", "advantageousß", "aasumes", "The data sets consists of"

2) What do you mean by "omitted features" in "Keiding et al. [33] also show that, relative to CoxPH, AFT models are more stable when accounting for omitted features."?

3) When you say "Specially, the convergence rates are determined by the intrinsic dimension of the underlying functions rather than the original high-dimension input features." and then go on to say "This, to some extent, explains the great success of deep learning and distinguishes our contributions from existing work.", how is your modeling different from previous survival analysis frameworks that make use of deep learning? In other words, what sets your work apart from other survival analysis algorithms that use deep learning given they are using neural networks too?

4) In section 2.1, when you say "It is difficult to estimate and directly from the likelihood function", it would have been beneficial to add a sentence or two explaining why.

5) When you say "This is the main reason why the Cox proportional hazards model dominates the field of survival analysis for nearly fifty years" what is the main reason of Cox's domination? Optimizing the parametric and non-parametric components separately?

6) In section 3.1, the pseudo-likelihood can be better justified. The intuition behind the formula of the loss is missing. It seems to me that the pseudo-likelihood and the hazard function that is introduced afterwards can switch places. A better storyline would have been: (according to my understanding of what this section is trying to say) Following [52] the hazard function can be approximated by the following formula because ... which would then give rise to a pseudo-likelihood (why is it "pseudo" in the first place?) which is ...

7) In section 4.1, the evaluation criteria can be intuitively more elaborated on. The C-index is a bit explained when you talk about assessing the concordance between rankings but the Integrated Brier Score is not explained much. Giving more intuition to the reader would go a long way in understanding how these criteria are useful in assessing survival estimators.

8) How are you recovering the survival function from the hazard function? Are you using numerical integration methods? 

9) In section 4.2, why DeepHit only performs well when no lower dimensional structures exist for the survival times?

10) Table 1 could use rearrangement. It is difficult to read. Could be decluttered.
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors propose to extend the "Extended Hazards Model" (EH) with deep learning approaches to model right censored time to event data. EH model is a generalization of both the Accelerated Failure Time and the Cox Model Proportional Hazards Model and thus is more flexible than either of the two approaches. They demonstrate empirically that the extended hazards model improves the prediction of survival across multiple datasets in terms of both discriminative performance (C-Index) and Integrated Brier Score on multiple competitive baselines. 


main_review: 
Learning survival models to model right censored datasets with neural networks is a challenging problem. The authors propose to employ the extended hazards model to model survival data. Their motivation is well justified intuitively. The EH is a generalization of both the Cox Proportional Hazards model, and the Accelerated Failure Time model and thus alleviates challenges associated with these two models. 

## Strengths

- Rigorous experimental evaluation across multiple strong datasets and baselines, in terms of both Discriminative Performance and Integrated Brier Score. 
- A major challenge with the CoxPH model is its limitations in handling problems where the PH assumptions are not met. The Extended Hazards approach is well justified intuitively since, it can alleviate these challenges. 
- Some interesting theoretical justification for the use of Deep Learning to learn low dimensional representation to better estimate the time-to-event distribution. 

## Weaknesses:

- Although the paper attempts at providing a theoretical justification, in Sec 3.2. The theoretical results are largely a consequence of the convergence rates and results already established in (Y.-K. Tseng and K.-N. Shu., 2011) and (Zeng and Lin, 2007) making this part of the contribution, much less novel.
- I would thus evaluate this paper as a modeling paper, and not necessary as theoretical contribution. Unfortunately, in terms of modelling the main text of the paper glosses over many important details of the Extended Hazards model which would be required for reproducibility by a reader not deeply entrenched in survival analysis. For example, 

A) it is unclear how Eq. (4) -> Eq. (5), since the authors don't explicitly mention the relationship between the hazard function and the survival function.

B) In the estimate of the Pseudo-Likelihood (post line 165),  it is unclear how the Kernel $\mathcal{K}(\cdot)$ shows up as it has not been referred to before. In any case, the likelihood $\mathcal{L}(\cdot)$ should be explicitly rewritten as a function of the Kernel $\mathcal{K}_b(\cdot)$ with bandwidth $b$. 

C) It is unclear as to how $\Phi(\cdot)$ is computed. (It is an integral over event times) is this computed over a fixed discrete grid or over all observed event times? 

- Finally, I have major concerns about reproducibility of this paper. For example, in the main text and the supplement, the authors do not mention the choice of the  kernel function ($\mathcal{K}(\cdot)$) they use or how the bandwidth is selected. While the number of baselines considered are extensive; there is no discussion about the hyperparameter grid over which the baselines were tuned. 

Overall, the paper proposes an interesting extension to the original idea of extended hazards using Deep Learning, which is welcome. The paper also aims to provide theoretical justification for their choice, however some of the implementation and design details are ignored in the paper calling into question the reproducibility of the proposed approach.


## Other Concerns/Recommendations: 

- Have the authors evaluated performance in terms of Concordance over truncated event times? If yes, they are encouraged to include these results in the supplement to establish how the discriminative performance varies at different time horizons. 
- An important problem that the model aims to alleviate is the strong assumption of proportional hazards. The authors are recommended to include plots of the estimated survival distributions on real world data in the main text to demonstrate the model is able to capture scenarios where the assumption is violated. (One clear situation of this is when the estimated survival curves of two individuals intersect)
- The authors are recommended to keep the theorems in the main text but to move exact details of Section 3.2 to the supplement, and use the extra space to address the concerns raised above. 
- Have the authors compared the performance of their approach to stratified Cox/AFT approaches (separate cox/aft models for different strata) or approaches that involve learning mixtures of Cox/aft models (Discrete mixture regression models for heterogenous time-to-event data: Cox assisted clustering, Kevin H. Eng & Bret M. Hanlon, 2012; Deep Cox Mixtures for Survival Regression, Nagpal et al, 2021; )? If not, can the authors comment when should an EH model be preferred over using stratified Cox models? 
limitations_and_societal_impact: The authors should clearly describe how strong assumptions on the hazard distributions can lead to miss estimation of the risk profile/ trajectory for individuals leading to potentially harmful decision making.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper introduces the Deep Extended Hazard Model as a new approach to learning a survival model, based on learning 2 neural nets, to estimate the 2 terms of the extended hazard function.  It explains how this model extends the EH model, which extends both the Cox proportional hazard model and Accelerated Failure Time.  It then provides first a theoretical analysis, following by an empirical exploration of these models 
main_review: The paper did a nice job of motivating then explaining the various models considered. However, I had a hard time following the theoretical analysis.  Eg. I wish the authors had provided some details of the derivation of Eq1, or a pointer.  I also wondered which parts of Sec 3.2 were general for all neural nets, and what was specific to survival functions, and what just for variants of nn extensions of EH models? Some parts could have been a bit more precise--e.g I assume that \hat-h is based on the sample size n (line 225). Also, I did not understand how the theoretical analysis informed the development of the algorithm, or was used in the analysis of the empirical results.  

They empirically explored many (10) survival models, over 4 different datasets. The authors should have performed some statistical tests, to indicate which of the results were significant.  Given the paper’s concern with high dimensional covariates, it was surprising that the datasets were all low-dimensional.  I wish the authors had empirically explored this issue, by using high dimensional data.

While IBS is considered both discriminative and calibration, it would have been useful to see if each dataset was calibrated--perhaps at the median time.  See the analysis in https://jmlr.csail.mit.edu/papers/v21/18-772.html .


wrt the IBS:  Li258, \hat-G is NOT the KM estimator.  It is the estimator of the conditional survival function of the censoring times calculated using the Kaplan-Meier method.  Is the integral Eq12 done analytically for your models, or estimated empirically? If the latter, are there issues related to the intervals considered?  Is \tau the largest time (censored or not), or the largest uncensored time, or something else? 

The authors should have discussed how they split the data into 5 folds–was this balanced for (censored, death) times? For anything else?

I wish they plotted the results in a figure–perhaps moving Table 1 to the appendix. Similarly, it would be helpful if they had plotted the hazard curves, or the survival curves, to help the reader visualize the results.

 
li55: given the many different ways to evaluate a survival model, what does “improvement” mean?

li77: I assume  “omitted features” means some values are missing for some instances. If so, are these omissions considered  MCAR?

li119: extra β

li277: Why mention competing risks? Isn’t there only a single type of risk?

--------------
My assessment would be better if the paper

 included statistical conclusions

 explicitly connected the theoretical analysis to the algorithm design and/or the empirical results



limitations_and_societal_impact: Predicting a patient's survival time for a given treatment, could help identify the appropriate treatment.  Of course, it could also be used to determine the most toxic poison or disease.  However, I do not think the manuscript needs to discuss this.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: Substituting two general functions for linear risk predictors in the Extended Hazard (EH) model, the DeepEH approximates these two functions via feed-forward neural networks (FNN). The DeepEH may fall into the category of FNN-assisted survival analysis which utilizes FNNs to estimate the hazard/survival function. The shining point of this paper is the established asymptotic properties of estimators from DeepEH. 



main_review: Depending on up-to-date theoretical developments on FNN approximation such as Schmidt-Hieber (2020, Ann. Statist. 48:1875--1897), this submission makes a theoretical contribution to the FNN-assisted survival analysis. We guess this effort is of interest to the community of deep learning research. Also, this paper involves extensive numerical comparison among DeepEH and a dozen competitors.

Correctness: We believe the theories and proofs are generally correct. 

Clarity: Most parts are easy to follow.

limitations_and_societal_impact: The extension from the EH model is natural. In addition, there has been literature that proves the power of FNN from a theoretical point of view, whereas this paper fails to make a review in this regard. Among other works, Schmidt-Hieber (2020) gave an exact upper bound of the approximation error for FNNs involving the least-square loss. Since the DeepEH optimizes a likelihood-based loss, this paper builds up its asymptotic properties by following assumptions and proofs of Theorems 1 and 2 in Schmidt-Hieber (2020) as well as theories on empirical processes.

Additional Feedback: 1) In the manuscript, $\mathbb{P}$ mostly represents a probability but sometimes for a cumulative distribution function (e.g., Eqs. (3) and (4) and L44, all in Appendix), which leads to confusion. 2). The notation $K$ is abused too: it is used both for a known kernel function (e.g., L166) and the number of layers (e.g., L176). 3). What is $K_b$ in estimating baseline hazard (L172)?
review_title: null

Meta-review: This paper has been reviewed by four knowledgeable referees resulting in one accept, two marginal accept and one marginal reject recommendations. Most of the criticism centers on limited novelty and utility of the theoretical portion of this work. Yet, the proposed method appears to stand on its own, yet it has been evaluated on a relatively low dimensional data only. The authors have engaged in discussions with the reviewers that helped resolve some but not all of the stated concerns. All things considered, I recommend this paper for acceptance if space permits its inclusion in the conference agenda. The authors should strive to incorporate all constructive recommendations from the reviewers in the finał camera ready version of their paper.
Decision: Accept (Poster)