Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: ODEs have recently found their applications in survival analysis where in the case of right-censoring the log-likelihood can be optimized quite straightforwardly using an ODE solver. This work is inspired by an assumption known the clustering literature according to which the decision boundaries do not cross high-density regions in the data space. They use the same intuition in their hazard estimation. They propose a regularizer on the hazard gradient which is the expectation of the hazard gradient with respect to a random variable whose PDF is the normalized survival function. They show that it is equivalent to minimizing the KL divergence between the PDFs of two input vectors which are "close" in the input space.
strength_and_weaknesses: * A major short-coming of this work is being restricted to time-static covariates. There has been recent work on time-varying covariates in survival analysis so considering time-static covariates only comes at the price of usability.


* Since you are proposing a hazard estimator, it would have been beneficial to also provide RMSE results on recovering the true hazard in a dataset. Of course for that you needed to create a synthetic dataset where you simulate survival data using a known hazard function but it would have been beneficial.


* The only non-ODE-solver baseline was CoxPH, whose dependence on input covariates is linear. Comparing against non-linear hazard estimators or even CoxPH extensions whose covariate effect is non-linear would have made the a stronger case of the developed algorithm. Without the mentioned comparisons, only ODE solvers are compared with one another.
clarity,_quality,_novelty_and_reproducibility: The paper is well-written and the authors have made their contributions clear. This work has novelty and was an interesting read. The work seems fairly reproducible given the provided code snippet by the authors.
summary_of_the_review: I find this paper interesting but at the same time I am not fully convinced of its applicability and superiority over baselines. Not sure about applicability because it only incorporates time-static covariates and not sure about superiority given the missing comparisons mentioned above.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper applied the neural ODE method to log trasformed survival rate model expressed by integration of hazard funciton. The penalty proposed is hazrd gradient penalty. The equivalence to KL divergence is proved, and empiral comparison to four other penalties under three criterions demonstrated the benifit of the proposed hazard gradient penalty.   

strength_and_weaknesses: This paper started from a very general form of survival analyses, making the proposed method applicable to broad survival models shown in section 2.1. The alogorithm proposed is computationally efficient than KL convergence by requiring less derivation of intermediant values. Theoretical proof of the equivalence of KL divergence and the hazard gradient method is proved. The experiments performed on three public datasets showed the proposed hazard gradient penalty method perform good under the three adopted criterions. The reviewer like the idea in A2 of feeing scale time to boost training time. 

In A2, it clarifies "calculate C, and AUC at 10%, 20%, ..., 90% event quantiles and average them". This reviewer is wondering the comparision of the proposed method and existing one would be like what if closer quantiles (say each 1% apart), or if higher weight to cetral quantiles. Or say, would it possible the proposed method with estimated hazard change slow in boundary to make calculated mAUC etc look good while not better performance than existing penalties at the central high-density regions.      

clarity,_quality,_novelty_and_reproducibility: This paper presents the related work, the motivation of proposing hazard gradient, and the main steps of how to implement this method clearly. 

This paper's main novelty is proposing using a penalty based on hazard function, while the main compuational framework inherits from Neural ODE (Chen et al 2018).    

The theoritial proof in section 3.2 looks valid. The algorithm presented in Alogorithm 1 and Experiemntal Details A.2 add sense while the reviewer didn't try to programmingly reproduce the result. 

The clarity can be improved in a few detailed places, for example: 
* Section 2: There are numerious splicit survival models. Tang 2022b adopted one form depending on cumulative hazard. The author of this under reviwe paper chose to work under more general survival regression form. Maybe the language "modeling p(t|x) or S(t|x) is challenging" and the footnote can be reworded. 
* Section 4: 1) Specify "our proposed method" is called as ODE+HGP. 2) refer reader to A1 when first metion mC, mAUC and iNBLL. 3) Clarify In table 1 it's mean +/- what is shown, and mention some experimental details in A2.  

summary_of_the_review: This paper proposed hazard gradient penalty to address survial analyses using ODE framework facusing on data point rather than global information. The general presentation is good. The experimental result (if the author can clarify/defend the result with more than only ten quantiles selected) compared to existing method based on the paper showed advantage of implementing this regularizer.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a hazard gradient penalty algorithm for regularising neural ode-based survival models. Additionally, the paper formulates a theoretical connection of the proposed hazard gradient penalty to KL divergence  minimization of  KL ($p(t|x) || p(t| x^{\prime})$), where $x^{\prime}$ are neighbouring data points of $x$. Experimental results on three real-world datasets show performance improvement over baselines per metrics C-Index, negative binomial log-likelihood (NBLL), and AUC. 
strength_and_weaknesses: **Strengths**:
- The paper focuses on survival analysis, an important problem across many domains
- Theoretical analysis of connections of the hazard gradient penalty to KL divergence  minimization of  KL ($p(t|x) || p(t| x^{\prime})$), where $x^{\prime}$, may be of interest to some researchers
- Paper provides a snippet code implementation in JAX

**Weaknesses**:

*Overall Clarity*
-  The title of the paper needs to be more focused since the proposed regularizer  is mostly applicable to neural ODE survival models
- The paper should include a figure, illustrating (i) model parameters $\theta$; (ii) how $h_{\theta}(t|x)$ are learned, and (iii) model outputs at test time.

*Technical Quality*
- The paper claims modeling $p(t|x)$ or $S(t|x)$ is challenging:  This is not necessarily true, since there exist several approaches modeling $p(t|x)$ or $S(t|x)$ including AFT, Lee et al. (2018),  Chapfuwa et al. (2018), *etc.*
- It's unclear how the paper imposes the survival  constraints for learning valid distributions: Survival function is a monotonically decreasing function and $\lim_{t \rightarrow \infty } S(t|x) =0$
- It's unclear what the model outputs at test time.  E.g., Is the model predicting hazard, event times, or survival probabilities?
- Eqn 2: Should be $P(t < T < t + Δt | T> t, x)$
- Eqn 3: What are the inputs to the ODE solver?
- Paper claims modeling the hazard function is better that Tang et al. (2022b)’s cumulative hazard function without providing any justification, *e.g.*, experimental results
- Eqn 5: Should be $t \sim p_{\theta}(t|x)$, it's unclear how the paper samples from the survival function $t \sim S_{\theta}(t|x)$?
- Hazard gradient penalty: It's unclear why the ODESolver is evaluated at varying batch-specific times, instead of shared range $[0, t_{\rm max}]$
- Eqn 7: How is $\lambda$ selected?

*Underwhelming Experiments*
- The performance improvements over the vanilla neural ODE baseline seem marginal:
*  How significant are the claimed improvements
* Table 3: Computational complexity analysis is only provided for one dataset. How does the proposed approach scale with larger datasets (*e.g.*, Support), batch sizes, *etc.*? 
-  Paper should also include qualitative results comparing predicted $h(t|x), S(t|x), p(t|x)$ against ground truth times and baselines
- For completeness, the paper should consider comparisons to other non-ODE baselines, e.g., AFT, Lee et al. (2018),  Chapfuwa et al. (2018), etc.
- Table 2: It seems the model performance is not sensitive to increasing from $M=1$ to $M=10$. Is this expected?

*Novelty*
- Gradient penalty regularisation has been proposed before. An expanded discussion highlighting how the proposed approach is better (or different) than previous approaches, *e.g.*, 1-Lipschitz function constraint in W-GAN is crucial.
clarity,_quality,_novelty_and_reproducibility:  See above.
summary_of_the_review: Some researchers may find the theoretical analysis of the proposed gradient penalty interesting. However, the paper is lacking due to  several technical issues and underwhelming experiments.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The authors propose a hazard gradient penalty on ODE models to ensure the smoothness on the density in survival analysis. The main contribution is this hazard gradient penalty.
strength_and_weaknesses: Strength:
1. The authors propose a novel hazard gradient penalty specifically on survival analysis problems using ODE methods. 

Weakness:
1. The use of the hazard gradient penalty is not well-supported. 
a. The authors claim that 
"The cluster assumption from semi-supervised learning states that the decision boundaries should not cross high-density regions (Chapelle et al., 2006). In a similar vein, hazard functions of survival analysis models should change slowly in high-density regions." I am not sure how the idea of cluster assumption applies here in survival analysis. It does not sound similar to me.
b. The authors somehow prove that their gradient penalty is an upper bound on KL-divergence between p(t|x) and p(t|x'). There are many ways to penalize that. I am not sure why the authors care about the gradient of the hazard. You can actually directly penalize the gradient of p(t|x). You can also model the CDF as an ODE[1] and do not need to model the cumulative hazard. You probably also want to cite [1] since they also talk about using ODE in survival analysis and is very similar to SODEN.
2. Empirical results are not very convincing. The authors do not tune much for the baseline ODE model. But it's actually sensitive to many hyperparams. See the SODEN repo. They random search 100 groups of hyperparams, select the best on the validation and report on the test. If the authors can show that their method is better than the numbers reported in SODEN, it will be more convincing empirically. Some large real world experiments may also be needed (for example, MIMIC, SEER). The current datasets are not very large and do not contain many features which may not be the right test-bed for deep learning.

[1]Derivative-based neural modelling of cumulative distribution functions for survival analysis.

clarity,_quality,_novelty_and_reproducibility: The clarity is okay. The method is new. The method seems to be easy to implement.
summary_of_the_review: It's unclear whether we should use the proposed method. The motivation from ssl is vague and there is obviously another way to penalize the function based on the theoretical claim. So I give "reject".
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This easy to follow paper has been assessed by four knowledgeable reviewers. It presents a novel approach with a somewhat unclear or limited utility and unclear comparison, in terms of empirical performance, to state of the art alternatives. One of the reviewers correctly points out lack of novelty of the proposed gradient penalty regularization scheme. The reviewers also point out that the reported performance improvements over basic alternatives appear marginal. The choice of benchmark datasets is limited and blurs the perception of impact.
Three of the reviewers ranked this work at its current stage below the threshold of acceptance, one considered marginally acceptable. Therefore, I recommend rejection, but I would also like to encourage the authors to address the key deficiencies and resubmit their updated work to a reputable venue.  
justification_for_why_not_higher_score: Reported marginal improvements over of baselines over limited set of benchmarks, limited novelty of some aspects of the contribution.
Most reviewers vote to reject.
justification_for_why_not_lower_score: n/a
Decision: Reject