Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The authors introduce a deep-learning-based extension to accelerated failure time modeling for survival analysis and introduce a rank-regression-type loss function based on Gehan's rank statistic.
main_review: This is a somewhat confusing paper for several reasons: i) the problem is set as AFT, however, the likelihood (encoded in the residuals), which is key to AFT is not used for optimization, so one could argue that the proposed approach is not an AFT formulation but rather rank regression with censoring; ii) the rank-based loss function in (6) that is very similar to that in Raykar et al. (NeurIPS 2007: On Ranking in Survival Analysis: Bounds on the Concordance Index), which is used in DRAFT but not discussed in the paper; iii) the experiments show that the proposed model is no better than the standard, linear, Cox proportional hazard in terms of C-index which is the most appropriate performance metric considering the proposed model is optimizing for ordering using a rank statistic; iv) the theoretical results for (7) in the linear case, as the authors discuss are not directly generalizable to nonlinear models.

The proposed approach seems to outperform Cox in large datasets. The authors may consider presenting results on additional large datasets to confirm that it is indeed the case, which in turn will make the experiments section stronger.

Note that DRAFT in Figure 1 is not introduced or discussed in the paper. 
summary_of_the_review: The proposed approach presented as a deep-learning extension of AFT optimizes a rank-based loss function very similar to that in Raykar et al. (NeurIPS 2007: On Ranking in Survival Analysis: Bounds on the Concordance Index). The experiments results show that the proposed approach does not outperform standard Cox in 3 out of 4 datasets.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors suggest a neural network based accelerated failure time (AFT) model, as well as an L1-type rank loss, which they argue results in an easy and fast to train model, which is state of the art in terms of two standard evaluation metrics, concordance and integrated Brier score. Following the semi-parametric AFT literature, the baseline hazard is estimated from the data and the full hazard function is parametrized in a way that the parametric part corresponds to a prediction of the failure time. The main innovation of the paper is the use of a novel loss function based on a ranking loss using Gehan’s rank statistic, which while being used in the statistical literature, has not been used in a machine learning setting. 
main_review: Strengths:
 - Use of a (for the machine learning literature) novel loss function, based on Gehan’s rank statistic
 - The paper is mostly clearly written and easy to follow
 - The authors put their method into context well


Weaknesses:
 - Due to semi-parametric nature, prediction only possible up until the last observed patient time.
 - The loss function in Jin et al (2003) is derived for the linear case. While the authors argue for how it could work in the non-linear case, there is no guarantee that the optimal set of parameters corresponds to a zero rank statistic in the non-linear case
 - The loss function is quadratic in the training samples, which could lead to bad performance for large data-sets. The authors however show very fast performance on a large data-set, which makes this a minor point.
 - The authors introduce both a neural network architecture for AFT models and an L1 ranking loss. It is not entirely clear how much influence either of these have on the benchmark results. In the appendix the authors introduce other loss functions, but do not include the trained models based on those losses in the benchmark results. The addition of these in table 2 and 3 would help disentangle the influence of the model architecture and of the loss, which would be interesting.

Other:
 - The loss function is quadratic in the number of patients/intervals, whereas the full likelihood is linear. Given this, the speed on KKBOX seems very surprising and is probably due to a small constant. It would be interesting to see the scaling with the number of patients N with real data with both the ranking and the maximum likelihood loss.
 - The authors could make the connection between $\epsilon$ in equation (3) and $h_0$ in (9) more clear as a step before using the Nelson-Aalen estimator to use a semi-parametric AFT.
 - The authors use “calibrated” when talking about Brier score. However traditionally when talking, especially about AFT models, calibration refers to the difference between real and predicted event times and the Brier score is a mix of calibration and classification. The Brier score is the calibration of the survival distribution over time, so calling it calibration is not incorrect, however within the context of the survival literature somewhat misleading. More specific language would make the paper more readable.
 - An introduction or motivation why the loss function is a good loss function would be beneficial for the understanding of the paper. The authors could elaborate on rank statistics and why they are expected to be more stable. Especially, as e.g. DeepHit talks about ranking when talking about concordance, it would be helpful for reading that ranking here is in reference to a weighted log-rank estimating function.
 - It would be interesting to see how well the model does in calibration of the event times (also with the other loss functions), as this is clinically meaningful.
 - The bold facing in table 2 is not consistent (DeepHit should be bold in one case)
 - In the appendix the authors make it seem that there are no convergence guarantees for MLE based estimators. However, also considering the application to AFT models as a special case, there do exist estimators that have some theoretical convergence guarantees (Tang et al 2020, Survival Analysis via Ordinary Differential Equations). A small discussion thereof could be beneficial.
summary_of_the_review: While there is a large number of machine learning methods for survival, by considering a (for the machine learning literature) novel loss function in the AFT setting, this paper can be seen as a meaningful contribution to the machine learning for survival literature. The theoretical foundation of the loss function is however not perfect and the influence of loss function and architecture are hard to disentangle. Due to the form of the loss function, a formal scaling analysis would also be needed. I would recommend a weak accept. I could be convinced to improve of the score with a more convincing argument why the loss term in the non-linear setting is a good loss term, maybe through some upper bound on the loss term.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes to combine the idea of Gehan’s rank statistic idea on fitting the AFT model, as well as using the deep learning model as a non-linear method for replacing the linear method in the original AFT model. The authors argue they connect Gehan's non-parametric technique with deep learning models. Experiments on various benchmark datasets show that the proposed model is competitive to state-of-the-art baselines.
main_review: Strength:

1. The idea of combining Gehan’s model with non-linear deep learning model is new
2. Performance is stable on several benchmarks.
3. Clear writing

Weaknesses:
1. The intuition of the approach is not quite clear. Does the proposed model focus on reducing computational time or improving the performance, or both?
2. The results do not support the conclusion. The improvements seem not significant compared to baselines.
3. The methodology parts mostly combine another existing model idea into the current AFT model. Hence, the innovation is limited methodological-wise.
summary_of_the_review: While the idea of combining Gehan model with deep learning is new, the results show that the proposed model does not outperform SOTA systems on several benchmarks.
review_title: null

final_score: rating: 1: strong reject
review_content: summary_of_the_paper: This paper proposes a (deep) neural net extension to an existing ranking-based estimator used for the semiparametric AFT model (Jin et al 2003). The resulting method achieves time-dependent concordance index and integrated Brier score values that are competitive compared to baseline deep AFT models.
main_review: I found this paper straightforward to read although I have several large concerns.

My first concern is that the exposition is overly complicated even though the key idea of the paper is extremely simple. The proposed model DART can be derived by just taking the known rank estimator by Jin et al (2003) and swapping out the inner product in their equation (2.1) with a neural net, and then just train using minibatch gradient descent instead of linear programming. In other words, much like how Faraggi and Simon (1995) swapped out the inner product in the semiparametric Cox model to be a neural net to come up with the same neural net model as DeepSurv (Katzman et al 2018), this paper does the same thing with the already existing semiparametric AFT model (which already was known before the Jin et al (2003) paper but the optimization procedures in estimating the regression coefficients were not optimal). From a technical novelty/innovation perspective, there is very little that is conceptually new. Instead, the paper is motivated in a way that spends way too much emphasis/text on describing standard results from survival analysis (e.g., what is a Cox model, what is an AFT model, why one would use standard Cox PH vs AFT, that the Cox model can be made time-dependent, that AFT models can be specified in both parametric or semiparametric forms, etc). I'd suggest perhaps having your background instead focus on existing semiparametric AFT literature, that you're simply doing a straightforward neural net extension (replace inner product with neural net), and comparing your model with DRAFT and DATE (I found Figure 1 very helpful). I'd suggest reducing the amount of text spent on explaining what hazards models are or why one should use hazards models over AFT models as this is a very old debate at this point (from what I can tell it really just depends on the data). Of course, a Weibull regression model is both a hazards model and an AFT model.

My next concern is that the experimental results are inconsistent with existing literature. As is, the experimental results presented make it seem that DART is not a clear winner over top-performing baselines. However, once we consider that some of the numbers are quite off from literature, how good DART is seems even more suspect. For example, your reported DeepHit $C^{\text{td}}$ number for the KKBox dataset is dramatically off from Kvamme et al (2019)---basically according to what they get, DeepHit gets 0.888 which is higher than DART 0.867. Some of your IBS numbers across methods also seem off. Some explanation of these discrepancies would be helpful.

There is another exposition issue: early on in the paper, the text makes it seem like the Cox model does not make a linear assumption whereas AFT does. This isn't true. Cox assumes the partial log likelihood is linear. AFT assumes the log survival time is linear. In other words, both make linearity assumptions, just for different quantities.

References:
- David Faraggi and Richard Simon. A neural network model for survival data. Statistics in Medicine 1995.
- Zhezhen Jin, D. Y. Lin, L. J. Wei, Zhiliang Ying. Rank-based inference for the accelerated failure time model. Biometrika 2003.
- Jared L. Katzman, Uri Shaham, Alexander Cloninger, Jonathan Bates, Tingting Jiang, and Yuval Kluger. DeepSurv: personalized treatment recommender system using a Cox proportional hazards deep neural network. BMC Medical Research Methodology 2018.
- Håvard Kvamme, Ørnulf Borgan, Ida Scheel. Time-to-Event Prediction with Neural Networks and Cox Regression. JMLR 2019.
summary_of_the_review: This paper has extremely limited technical novelty, spends too much text on explaining existing standard results/definitions from survival analysis, and has experimental results that are suspect (inconsistent with existing literature).
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper extends the previously proposed linear semiparametric AFT model based on Gehan’s rank statistic to a nonlinear setup. The nonlinear model termed Deep AFT Rank-regression for Time-to-event prediction model (DART) is parameterized by a neural network. Experimental results across four datasets show a competitive advantage over baselines per concordance index (C-Index), integrated Brier score (IBS), and training time.
main_review: The paper is easy to follow with competitive quantitative results. However, the technical contributions are lacking, there are some misleading statements, and the writing needs improvement. Below are specific examples.

-  Introduction: "The accelerated failure time model (AFT) or accelerated life model relates the logarithm of the failure time linearly to the features." This statement is not necessarily true for parametric AFT other than log-Normal, *e.g.*,  Weibull, Exponential, *, etc.*
-  The $L_{\rm CoxPH}$  (Eq. 2) definition is not correctly specified. Note the risk set $\mathcal{R_i}$ should include censored on non-censored times $T_j$ s.t. $T_j < T_i$
- Sec 3.2: "However, estimating survival quantities (e.g. conditional hazard function) cannot be directly done for AFT-based models." This statement is not necessarily true for some parametric AFT models, *e.g.*, Weibull, Exponential, *, etc.*
- Sec 3.2: While paper claims to make time-to-event  predictions $\hat{T} = \exp(g(X_i, \theta))$ it is not clear why those predictions can not be used to directly estimate $\hat{S}(t| X)$. Instead, the paper proposes a semiparametric conditional hazard transformation similar to CoxPH without providing any justification.
- Sec 4: Is there evidence that supports this statement  "Note that standard concordance index yields identical results with ${\rm C}^{\rm td}$ for AFT-based models "? 
- Sec 5.4:  For fair comparisons, the paper should compare AFT and hazard models using a similar metric, either C-Index or ${\rm C}^{td}$ or both.
- For a comprehensive evaluation, the paper should provide additional qualitative results, *e.g.*, model predictions against ground truth, and calibration curves.

**Minor Issues**
-  Missing $\xi \sim F_{\xi}$ in third term (Eq. 4)
- Introduction: Typo ATF should be AFT
- Sec 4: "well-fitted model yields IBS lower than"  sentence is incomplete

summary_of_the_review: Apart from empirical competitive quantitative results against baselines, the technical contributions are not clear. The semiparametric AFT model based on Gehan’s rank statistic objective function has been previously proposed. Additionally, while the paper claims to make time-to-event predictions, the conditional hazard transformation of model predictions seems to indicate the model predicts hazards instead.
review_title: null

Meta-review: Most reviewers came to the conclusion, that this work lacks novelty and theoretical depth. Further severe concerns about the validity of some statements and about the experimental setup have been raised. The rebuttal was not perceived as being fully convincing, and nobody wanted to champion this paper. 
I share most of these points of criticism. Although there is certainly some potential in this work, I think it is not ready for publication and would (at least) need a major revision.
Decision: Reject