Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The authors are interested in the setting of predicting discrete outcomes within time intervals of interest (ex. 5-year survival, 10-year survival) from time-to-event data, i.e. the setting in which some observations may be right-censored. The conventional approach relies on inverse probability of censorship weighting (IPCW), which first estimates the distribution over time-to-censorship $p(c|x)$ (for example, via a Kaplan-Meier estimate), then re-weights the data to estimate the distribution over time-to-event p(t|x)$. 

The authors instead propose *Inverse-Weighted Survival Games*, in which they learn *both* the time-to-censorship and the time-to-event distributions in a non-adversarial game, using each to re-weight the other's loss. On the theoretical side, they show that when the loss function is proper, the resulting game has the true distributions as a stationary point. Moreover, when the loss function is the Brier Score, this is the only stationary point. On the experimental side, they run experiments on simulations and public datasets (Survival-MNIST, METABRIC, SUPPORT) and show that IPCW games improve upon NLL-trained models in terms of concordance, KM-weighted log-likelihood, and KM-weighted Brier Score in the data-limited regime.
main_review: I'm a fan of this paper. The idea behind Inverse-Weighted Survival Games is simple and elegant. The background and methods sections are very well-written.

My primary point of concern is with respect to the experiments. I have several questions:

1/ What is the definition of the "NLL"-optimized categorical model? My understanding is that it's a simple optimization of the discrete-time classification model, without accounting for censorship. If so, then without accounting for censorship the metric isn't very useful -- I don't think it'd be worth including in the results.

2/ I'd strongly suggest the authors add as a benchmark a discrete-time categorical model optimized with IPCW via a simple Kaplan-Meier estimate of the censorship distribution $p(c)$. This is important because it's the most common way such models are developed today.

3/ How are the confidence intervals shown in Figures 3-7 derived? 

4/ What's the motivation for presenting the plots as a function of the number of training points? I think the training curves will not be relevant to most readers of the paper -- instead, what they'll care about is end-to-end performance. Instead of Figures 3-7, can we have a table of results measuring BS, BLL, Concordance and Calibration at *convergence*? I'd insist on showing confidence intervals for the above as well, for example by showing the standard deviation of each metric over random samples of the evaluation procedure. 

5/ I'd strongly sugest the authors include a measure of calibration in their evaluation criteria, alongside concordance (which measures discrimination). One example would be L1 calibration error, where censorship is taken into account via a Kaplan-Meier estimate for the observed incidence rate in each bin.

I'd be happy to revise my rating of this paper if the authors are able to address the above questions.

Miscellaneous typos and comments:

1/ L166: missing commas on $[128, 256, 64$]

2/ How many data points are in the simulation dataset?

3/ What is the rate of censorship in each of the datasets?
limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a new training method for discrete-time survival models, where not only a failure model but also a censoring model are simultaneously trained to optimize a set of time-dependent inverse probability of censor-weighting (IPCW) criteria. Discrete-time models trained with different objective functions have been compared on simulated data and multiple real datasets. 
main_review: Strength:
- The new training method that directly optimizes IPCW criteria for discrete-time models is novel and has been shown to have better test BS, BLL, and concordance on simulated data, especially for small sample sizes.
- They show that the IPCW game method has a stationary point at the data-generating distributions and IPCW BS has only one stationary point.
- The paper is generally well written and easy to follow.

Weakness:
- This paper claims that "deep discrete-time models have become a default choice even when data is continuous", this is not true as there are some recent deep continuous-time survival models [1,2] that are shown to have advantages over discrete-time models.  
- The ultimate goal is to obtain an estimated event time distribution that is close to the truth. Different evaluation criteria characterize how far the estimated distribution is from the truth in different perspectives, while the common optimum is expected to be the truth. Why optimizing the BS and BLL is particularly preferable / desired in the context of survival? I find the current argument "a conceptually simple way to evaluate models that is particularly suitable for discrete models is to apply binary classification losses at various time horizons, for instance Brier score (BS) for 5-year survival, or similarly, Bernoulli log-likelihood" is weak. The authors do find some empirical improvement on other criteria when using the new training method (BS, BLL based), but high-level intuition on why it is expected to work better would be appreciated.
- How is 20 selected as the number of categorical bins in experiments? The number of objectives to simultaneously optimize in the proposed method scales linearly with the number of bins, but, in general, discrete-time survival models require enough bins to well approximate a continuous event time. This would affect both the computation complexity and the numerical stability. Additional experiments would be desired.
- The assumption used in building the IPCW objective ($T \perp C| X$) and that used in evaluation metrics ($T \perp C$) for real datasets are inconsistent.
- I am trying to understand why the proposed method works better for small sample sizes. This observation is relatively counterintuitive for me. Because the game method simultaneously models the failure time and censoring time distribution and thereby contains more parameters than the NLL method. But it works better for small sample sizes. 
- References need to be updated such as [1] is still cited as its arXiv version. Please check other citations as well.

References:

[1] H. Kvamme, Ørnulf. Borgan, and I. Scheel. Time-to-event prediction with neural networks and cox regression. Journal of Machine Learning Research, 20(129):1–30, 2019.
[2] Tang, W., J. Ma, Q. Mei, and J. Zhu (2020). SODEN: A scalable continuous-time survival model through ordinary differential equation networks.arXiv preprint arXiv:2008.08637.


-------------------------------------------------------
I thank the authors for their thorough responses. After the discussion, I raised my evaluation.
limitations_and_societal_impact: Limitations have been discussed. More potential limitation: the game method is currently developed for the discrete-time models. But the inverse probability game seems to be general enough for continuous-time survival models as well, which may mitigate the issue of a large number of simultaneous objectives in the IPCW game for discrete-time models.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: In this paper the authors exploit the symmetry present in the survival analysis setting between the roles of C and Y to frame the problem as a game with the parameters as (unique with certain conditions) solution.
The authors prove then provide experiments showing not only that the approach is competitive but can even outperform the standard approach in the low data regime.
main_review: The authors adapt the IPCW framework, which is known to have good generalization properties (see [1]) even when the weights have to be estimated with a twist: instead of estimating S_C first once and for all, they iteratively refine the estimations of S_C and S_Y be exploiting the fact they both symmetrically are the solution to an IPCW ERM problem.


[1]G. Ausset, S. Clémençon, and F. Portier, “Empirical Risk Minimization under Random Censorship: Theory and Practice,”. Available: http://arxiv.org/abs/1906.01908

The main contribution is the proof that for certain losses the solution of interest is indeed a stationary point and that for a particular loss, it is the only saddle point.

The arguments are presented clearly and are convincing but some questions may benefit from more details:

1. The approach proposed is conceptually very similar to the doubly robust approach [2], has the link been studied by the authors?

[2] D. Rubin and M. J. van der Laan, “A Doubly Robust Censoring Unbiased Transformation,” The International Journal of Biostatistics, vol. 3, no. 1, Jan. 2007, doi: 10.2202/1557-4679.1052.

2. More details on the training procedure would be welcome (I personally never managed to achieve convergence using this method.)

Finally, some typos in the manuscript remain, some of them unfortunate (line 244: not invariant => invariant).

Overall I think this is a very interesting work and a worthwhile contribution.
limitations_and_societal_impact: The authors adequately addressed the limitations and potential negative societal impact of their work?
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper introduces a non-adversarial approach for simultaneously learning events and censoring cumulative distribution functions (CDF) at prespecified discrete-time points by optimizing inverse weighted Brier score (BS) and Bernoulli log-likelihood (BLL). Further, the theoretical analysis illustrates that the proposed approach achieves data event and censoring distributions at stationary points. Experimental results on (semi-)synthetic and real-world datasets demonstrate that the proposed achieves better concordance, Brier score, and Bernoulli log-likelihood than an approach that optimizes the standard negative log-likelihood (NLL) given small sample sizes.
main_review: Overall, the paper is well organized and easy to follow.  Moreover, provided theoretical analysis supports the recovery of the data distribution at stationary points. However, below are the key limitations.

**Significance**
- The paper focuses on discrete-time models, which are challenging to scale for large time horizons. Can the proposed approach be adapted to extensive continuous-time survival methods? Also, the paper title needs to be updated to include discrete-time survival since this is the focus of the submission.
- The paper needs to motivate the utility of the proposed metrics (time-varying BS, BLL) relative to standard metrics such as concordance index, relative absolute error, etc. Also, why are integrated BS(BLL) not considered?
- While the BS (or BLL) are proper scoring rules, is there evidence to support that IPCW-BS(BLL) are also proper scoring rules?

**Underwhelming experimental results**
- The paper should also benchmark against other approaches besides negative log-likelihood (NLL), including CoxPH and alternative approaches that optimize BS, such as [1].
- Experimental results on real-world datasets demonstrate that the proposed approach has no clear advantage over the NLL. 
- Also, why are the sample sizes on real-world datasets limited to 200? The SUPPORT dataset has ~10,000 observations.
- The paper should also provide qualitative results,  e.g.,  predicted CDFs against ground truth events or censoring times.

**Below are the key missing references**
- [1]  Avati et al., "Countdown regression: sharp and calibrated survival predictions", In Uncertainty in Artificial Intelligence, 2020.

--------------------------------------------------------------------------------------------------------------------------------------------------------------------------
I thank the authors for addressing my concerns and providing additional experimental results. Hence, I have increased my score.


limitations_and_societal_impact: The discussion needs to include the limitations of discrete-time models and the chosen metrics (BS, BLL).
review_title: null

Meta-review: To be updated - This is one, where I think the score underrates the work and should probably be accepted once reviewers update their scores based on the author response. I just emailed all of the reviewers who have yet to indicate they have read the author response. 
Decision: Accept (Poster)