Inverse-Weighted Survival Games
Authors: Xintian Han
Mark Goldstein
Aahlad Manas Puli
Thomas Wies
Adler J Perotte
Rajesh Ranganath
Venue: NeurIPS 2021 Poster (2021)
Keywords: Survival Analysis
Healthcare
Censoring
Games
IPCW
Brier Score

Abstract: Deep models trained through maximum likelihood have achieved state-of-the-art results for survival analysis. Despite this training scheme, practitioners evaluate models under other criteria, such as binary classification losses at a chosen set of time horizons, e.g. Brier score (BS) and Bernoulli log likelihood (BLL). Models trained with maximum likelihood may have poor BS or BLL since maximum likelihood does not directly optimize these criteria. Directly optimizing criteria like BS requires inverse-weighting by the censoring distribution. However, estimating the censoring model under these metrics requires inverse-weighting by the failure distribution. The objective for each model requires the other, but neither are known. To resolve this dilemma, we introduce Inverse-Weighted Survival Games. In these games, objectives for each model are built from re-weighted estimates featuring the other model, where the latter is held fixed during training. When the loss is proper, we show that the games always have the true failure and censoring distributions as a stationary point. This means models in the game do not leave the correct distributions once reached. We construct one case where this stationary point is unique. We show that these games optimize BS on simulations and then apply these principles on real world cancer and critically-ill patient data.

Weighte Survival Games
Xintian Han
Mark Goldstein
Aahlad Puli NYU
NYU
NYU
xintian. han@nyu. edu
goldstein@nyu.ed
aahlad@nyu.edd edu
Thomas Wies
Adler J. Perotte
Rajesh Ranganath NYU rajeshr@cims www.eeiiiiiinnmmm...m...m. nyu. edu
NYU wiesscs .wr.0ss..yu.ccm nyu.
Columbia University
edu
adler www.rrteeeolummla..eoee
Abstract
Deep models trained through maximum likelihood have achieved state-of-the-art results for survival analysis. Despite this training scheme, practitioners evaluate models under other criteria, such as binary classification losses chosen of time horizons, e.g. Brier score (BS) and Bernoulli log likelihood (BLL). Models trained with maximum likelihood may have poor BS or since maximum likelihood does not directly optimize these criteria. Directly optimizing criteria like BS requires inverse-wer weighting by the censoring distribution. However, estimating the censoring model under these metrics requires inverse- weeerhtng by the failure distribution. The objective for each model requires the other, but neither are known. To resolve this dilemma, we introduce Inverse- Weighted Survival Games. In these games, objectives for each model are built from re- ee-eeighted estimates featuring the other model, where the latter is held fixed during training. When the loss is proper, we show that the games always have the true failure and censoring distributions stationary point. This means models in the game do not leave the correct distributions once reached. We construct one case where this stationary point simulations and then apply these
unique. We show that these games optimize principles on real world cancer and critically-ii -ill patient data.
Introduction
Survival analysis the modeling of time-to-event distributions and is widely used healthcare to predict time from diagnosis to death, risk of disease recurrence, and changes in level of care. In survival data, events, known as failures, are often right-censored, i.e., only a lower bound on the time is observed, for instance, when patient leaves study before failing. Under certain assumptions, maximum likelihood estimators are consistent for survival modeling [Kalbfleisch and Prentice, 2002].
Recently, deep survival models have obtained state-of-the-art results [Ranganath et al., 2016, Alaa and van der Schaar, 2017, Katzman et al., 2018, Kvamme al., 2019, Zhong and Tibshirani, 2019]. Common among these are discrete-time models [Yu et al., 2011, Lee et al., 2018, Fotso, 2018, Lee al.. 2019, Ren al., 2019, Kvamme and Borgan, 2019b, Kamran and Wiens. 2021, Goldstein et al., 2020, Sloma et al., 2021] even when data are continuous because they can borrow classification architectures and flexibly approximate continuous densities [Miscouridou et al., 2018].
Though training is often based on maximum likelihood, criteria such as Brier score (BS) and Bernuulli log likelihood (BLL) have been used to evaluate survival models [Haider et al., 2020]. The and BLL are classification losses adapted for survival by treating the model as binary classifier at various time
Equal Contribution.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

horizons (will the event occur before or after 5 years?) [Kvamme and Borgan, 2019b, Lee et al., 2019, Steingrimsson and Morrison, 2020]. Bs can also be motivated by calibration (section 3) which is valuable because survival probabilities are used to communicate risk [Sullivan et al., 2004]. However BS and BLII are challenging to estimate because they require inverse probability of censor-weighting (IPCW), which depends on the true censoring distribution [Van der Laan et al., 2003].
Though consistent, due finite data, maximum likelihood may lead models with poor BS and BLL. But directly optimizing these criteria is challenging because IPCW estimation requires solving an additional survival modeling problem to estimate the unknown censoring distribution. This poses a re- re-weighting dilemma: each model is required for training the other under these criteria but neither are known.
To resolve the dilemma, we introduce Inverse- Weighted Survival Games for training with respect to criteria such as BS and BLL... We pose survival analysis as a game with the failure and censoring models as players. Each model's loss is built from IPCW estimates featuring the other model. Inspired by game theory [Neumann and Morgenstern, 2007, Letcher et al., 2019], we ask: should the censoring model's 's re- re-weighting role in the failure objective be considered part of the censoring objective? We find the answer be no. In each step of training, each model follows gradients of its loss with the other model held fixed to compute weights.
When the loss is proper (e.g. BS, BLl) [Gneiting and Raftery, 2007], show that games have the true failure and censoring distributions as a stationary point. This means the models in the game do not leave the correct distributions once reached. We then describe one case where this stationary point is unique. Finally, we show that weighte game training achieves better BS and BLL than maximum likelihood methods on simulations and real world cancer and ill-patient data. ?
2 Notation and background on IPCW
Notation. Let T be a failure time with CDF F(I) P(T S (). density f. survival function
G. and model Ge.- This means G(t) P(C > t). Let G(t denote P(C 2 t). We observe features X, time U = min (T. C) and A = 1 [T < C?. For discrete models over K times, let 6rt = Po = t) and lct- = (C
F F. and model Fo,. Let C' be a censoring time with CDF G, density g. G
Models. We focus on deep discrete models like those studied in Lee et al. 20. Kvamme and Borgan [2019b]. The model maps inputs X to categorical distribution over times. When the observations are continuous, a discretization scheme is necessary. Following Kvamme and Borgan [2019b], Goldstein et al. [2020], we set bins to correspond to quantiles of observed times. We represent all times by the lower boundary of their respective interval.
Assumptions. We assume i.i.d. data and random censoring: T .. C| | X [Kalbfleisch and Prentice, 2002]. We also require the censoring positivity assumption [Gerds et al., 2013]. Let f= = d.. Then:
3e s.t. Vx E (1. S mmaxx [(()x) > 0}: G( 2 <>0 0,
Le. is possible that censoring events occur late-enough for to observe failures up until maximum time max Truncating at a maximum time is necessary in practice for continuous distributions because datasets may have samples in the tails, leading practical positivity violations [Gerds et al., 2013]. This truncation happens implicitly for categorical models by choosing the last bin.
In this work, we model the censoring distribution. This task is dual to the original survival problem: the roles of censoring and failure times are reversed. Therefore, to observe censoring events properly, we also require version of eq. (1) to hold with the roles of F and G reversed (appendix A).
IPCW estimators. Inverse probability of censor-weighting (IPCW) is method for estimation under censoring [Van der Laan et al., 2003, Bang and Robins, 2005]. Consider the marginal mean [T]). IPCW reformulates such expectations in terms of observed data. Using IPCW, we can show that:
E [ < CIX P XTIX
T < C GX
AU
E
E
E E
EEE E X CIX
EI
T.C.X
G(U-
Cood is available at
htwp://///nnnnsaaaaaaeeeeeeeeeee Weighted-SSuriivl--GGe
2

---

1.47
0.148
1.46
0.146
1.45
0.114 8
1.44
1.43
nll
0.142
1.42
bs
Jowest bs
Jowest nll
0.140
model
Figure 1: Test NLL and BS for 5 different models, each trained with different learning rate
We derive this fully appendix B. The second equality holds because A means we can identify X[ provided that we know G' and that random censoring and positivity hold.
and
3 Time-dependent survival evaluations
Brier (BS) [Brier and Allen, 1951] is proper for classification, meaning that it has minimum at the true data distribution [Gneiting and Raftery, 2007]. The BS is often adapted for survival evaluations [Lee al., 2019, Kvamme et al., 2019, Haider al., 2020]. For time t, it computes differences between the CDF and true event status at f, turning survival analysis into a classification problem at given time horizon:
BS 0) - (
BS is often used as proxy for marginal calibration error [Kumar et al., 2018, Lee et al., 2019], which measures differences between CDF levels a [0, 1] and observed proportions of datapoints with XX < a [Demler et al., 2015]. This usage of BS stems from its decomposition into calibration plus a refinement (discriminative) term [DeGroot and Fienberg, 1983].
Unfortunately one cannot compute Bs unmodified since 1 [T' < t] is unobserved for point censored before f. IPCW BS [Graf al., 1999, Gerds and Schumacher, 2006] estimates (s( under censoring:
BS(t: 0) Fer (t x))AI {
For (t x) IU
(3)
G(U-13 X)
G(t X)
eq. (3) is equivalent to eq. (2) (appendix C). Negative Bernoulli log likelihood (BII.) is similar, but with log loss (appendix D). BS and BLL. are proper for classification at each time t, so their sum or integral over t is still proper (appendix H).
Proper objectives differ. Though negative log likelihood (NLL.), and BLI. all have the same true distribution at optimum with infinite data, they may yield significantly different solutions in practice. For example, NLL-trained models may not achieve good BS [Kvamee and Borgan, 2019a] In fig. 1, we show test set NLL and BS for models trained with NLL at different learning rates on Gamma-simulated data (described in section 5.1). NLL does not align with models that have low NLL may not have low BS. Model 4 has the lowest NLL but not the lowest BS. When a practitioner requires good performance under or BLL, they should optimize directly for those metrics.
Re-weighting dilemma. Censoring introduces challenges because we must use IPCW estimate BS and BLI.. Crucially, the G' in eq. (3) is the true censoring distribution rather than model, but during training, we only have access models. This poses dilemma: can the models used in re-weighting estimates during training to successfully optimize these criteria under censoring?

---

1.0
1.0
10
G₁(C=1) 1]
0.5
True Paramelers: 0.45
Min Sum: 0.428
10
0.0
0.5
Fo(T = 1)
Fo(T 1)
(a) Contours of summed objective
(b) Gradients of the game
Figure 2: Figure 2(a): the sum of F and G's IPCW BS(1) scores is an improper joint objective for the failure and censoring models. Figure 2(b): in contrast, shown in the gradient field, the one timestep game has unique stationary point at the true data generating distributions.
Inverse-Weighte Survival Games
reasonable attempt to solve the dilemma is to jointly optimize the sum of Fg and Ge losses where each model re-weights the other's loss. The expectation is that both models will improve over training and yield reliable IPCW estimates for each other. Concretely, consider this for eq. (3) plus the same objective with the roles of Fg and Go reversed. Unfortunately. there exist solutions to this optimization problem with smaller loss than for the distributions from which the data was generated, making this summed objective improper for the pair of distributions. In fig. 2, we plot this for IPCW BS(L = 1) for models over two timesteps' as function of each model's single parameter.
To address this phenomenon, we introduce Inverse- Inverse-Weighted Survival Games. In these games, failure player and censor player simultancously minimize their own loss function. The failure and censoring model are featured in both loss functions and playing the game results in trained failure and censoring model. We show experiments that these games produce models with good BS, BLL, and concordance relative those trained with maximum likelihood.
For simplicity, we present the games for marginal categorical models. The analysis can be extended conditional parameterizations with the usual caveats shared by maximum likelihood. Our experiments explore the conditional setting.
4.1 Basic definition of game
We follow the setup in Letcher al. [2019]. differentiable n-player game consists of n players each with loss & and parameter (or state) 0, Player controls only 9; and aims to minimize 4, However, each l, a function of the whole state (0,, 0,, The simultaneous gradient the gradient of the losses w.r.t. each players' respective parameters:
(0)) Ve,l,..... Cx)
The dynamics of the game refers to following -f. The solution concepts in games equilibria (the game analog of optima) and stationary points. One necessary condition for equilibria is finding stationary point O* such that E(0*) The simplest algorithm follows the dynamics to find stationary points and is called simultaneous gradient descent. With learning rate
0 0 (0).
This can be interpreted as each player taking their best possible move at each instant.
"t = 1) is proper for distributions with support over two timesteps because Bs(t = K) for a model with support K timesteps is always 0. the summed equals Bs)
*For two players, when & sometimes referred to as adversarial (e.g. as in Generative Adversarial Networks (GANs) [Goodfellow et al., 2014J). Games with f # -l2 are called non-zero-sum.
E2 the game is called zero-sum and can be written as a minimax game,

---

4.2 Constructing survival games
We specify an Inverse- Weighted Survival Game as follows. First, choose loss L used construct the losses for each player. Next, derive the IPCW form L, that can be used to compute L under censoring: for true failure and censoring distributions F and G*, the losses and L₁ are related through T Foy: G*) T(For) and (.rGGo F*) I(Goc). The loss functions for the two players are defined as:
(0) (r Goc), la(0) A Lr(Geci
(4)
Compared eq. (3), have replaced the true re-wer ne-weighting distributions with models. Finally, the failure player and censor player minimize their respective loss functions w.r.t. only their own parameters:
failure player: min f
censor player: min LG 0g
One example of these games is the IPCW BS(T) game, derived in appendix C. With 111
Fer (1)2 U S
(t) > d)
(0) E
(U-)
Go
(5)
(1) IU S
Go (1)2 (1)
(0) E Før (U)
Fo, (t)
In section 4.3, show that this formulation (fig. 2(b)) has formal advantages over the optimization in fig. 2(a) for particular choices of L.
Multiple Timesteps. The example is specified for given 6, but the games can be designed for multiple timesteps. We use BS for K timestep model to demonstrate. BS(K) (K) is 0 for any model: the left terms contain by and Gog; which are are both when evaluated at K; the right terms, K] is always O. One option is to define summed games with:
1 U >
K-1
K
EF
C
F.
t=1
1=1
The summed game is shown algorithm Alternatively, instead of the sum, possible to find solutions for all timesteps with respect to one pair of models (Fo, Ga). For K-1 timesteps this can be formalized as a --plyyerr game: there is failure player and censor player for the loss at each t:
th failure player: min f,,
tth-censor player: min c
6T.
6ct
We study theory that applies to both approaches in section 4.3, namely that the true failure and censoring distribution stationary points in both types of games. We prove additional properties about uniqueness of the stationary point for a special case the multi aulli-player game in section 4.4. Summed games more stable optimize in practice because they optimize objectives all time steps w.r.t all parameters, while multiplayer games can only improve each time step's 's loss w.r.t. one parameter. We study the summed games empirically in section 5.
Algorithm 1 Following Gradients in Summed Games
Input: Choice of losses Lr, &G, learning rate
?
Initialize: T model parameters 0r and C' model parameters 0g randomly
repeat
Set and DC = 0 for = 1 to K 1 do
gr + dec dd0T
IC = 9c det /d0c
end for
0T +- 0r - 7TT and Ic 9c 79C until convergence Output: 00
5

---

4.3 IPCW games have stationary point at data distributions
Among game's stationary points should be the true failure and censoring distributions.
Proposition Assume = E 0r 302 E O such that F Fez and G = Gog. Assume the game losses Ep, &g are based proper losses L and that the games only computed at times for which positivity holds. Then (07. ( is a stationary point the game eq. (4).
The proof in appendix I. The result holds for summed and multi player games using BS, BLL. or other proper scoring rules such as AUC. When games are built from such objectives, the set of solutions includes (0).; (2))) and models do leave this correct solution when reached. Under the stated assumptions, this result holds for discrete and continuous distributions. However, as mentioned section 2. in practice a truncation time must be picked to ensure the assumptions are met for continuous distributions.
4.4 Uniqueness for Discrete Brier Games
We provide stronger result for the BS game in eq. (5) when solving all timesteps with multi-player games: its only stationary point is located at the true failure and censoring distributions.
Proposition For discrete models over K timesteps, assuming that 0fy > and 0g > 0, the solution (07,, 02) is the only stationary point for the multi-player BS game shown in algorithm 2 for times t {1...., 1}
The proof is in appendix To illustrate this, fig. 2(b) shows that, unlike the minimization in fig. 2(a).
the IPCW BS game moves to the correct solution at its unique stationary point.
Experiments
We run experiments on simulation with conditionally Gamma times, semi-simulated survival dataset based on MNIST, several sources of cancer data, and data on critically-ill hospital patients.
Losses. We build categorical models in ways: the standard NLL method (eq. (8)), the IPCW BS game and the negative IPCW BLL game.
Metrics. For these models we report (uncensored for simulations and Kaplan-Meier (KM)- weighted for real data), BLL. (also uncensored or weighted), concordance which measures the proportion of pairs whose predicted risks ordered correctly [Harrell Jr et al., 1996], and NLL.. We report mean and standard deviation of the metrics over 5 different seeds. In all plots, the middle solid line represents the mean and the shaded band represents the standard deviation.
Model Description. In all experiments except for MNIST, that outputs 20 categorical bins (more bin choices appendix G. For MNIST we first use a small convolutional network and follow with the same fully-connected network.
3-hidden-layyr ReLU network
Model and training details including learning rate and model selection can found in appendix F,
5.1 Simulation Studies
Data. We draw E 32 ~ (O 107) and T ~ Gamma(mean = (+) where 14 a log-linear function of X. The censoring times are also gamma with mean 0.9 * μ4g Both distributions have constant variance 0.05. It holds that 1 C X. Each random seed draws a new dataset.
fig. demonstrates that the games optimize the true uncensored BS, and, though more slowly with respect to training size, log-likelihood does too. The games have better test-set perfor- mance on all metrics for small training size. All methods converge on similar performance when there is enough data (though enough is higeneede on dimensionality and model class).
Results.
'Though often reported, the dime-dependent concordance(t) is proper [Blanche al., 2019].

---

22
0.14
15
0.7
0:10
d
18
Es-jame
1 digamo
920
1000
1620
1200
1520
850
1000
1500
500
1000
1500
(a) Uncensored BS
(b) Uncensored Neg BLL
(c) Concordance
(d) Categorical NLL
Figure 3: Test set evaluation metrics (y-axis) on the Gamma simulation versus number of training points (X-axis) for three methods. Each point in the plot represents the evaluation metric value of fully trained model with that number of training points. Lower is better for all the metrics except for concordance.
Calibration Plot
1.0
R.R.B.A..
xmy
Obeerved Freedional andionalo Cumdalive Riek
0.8
game
0.6
0.4
0.0
0.0
0.2
0.4
1.0
Cumulative risk
Figure 4: Calibration curves [Avati et al., 2019] comparing game-training and NLL-training.
Calibration. We include a qualitative investigation model calibration on the gamma simulation trained with 2000 datapoints. fig. 4 shows that the BS game achieves near- near-perfect calibration while the two likelihood-baase -based methods suffer some error. This is expected since likelihood-based methods do not directly optimize calibration while does (section 3).
5.2 Semi-simulated studies
Data. Survival-MNIST [Gensheimer, 2019, Pölsterl, 2019] draws times conditionally on MNIST label Y. This means digits define risk groups and T 1 X Y. Times within digit i.i.d. The model only sees the image pixels X as covariates so it must learn to classify digits (risk groups) to model times. We follow Goldstein et al. [2020] and use Gamma times. T ~ Gamma (mean 10 (Y + 1)). We set the variance constant to 0.05. Lower digit labels Y yield earlier event times. C is drawn similarly but with mean 9.9 (Y 1). Each random seed draws new dataset.
This experiment demonstrates that better NLL does not correspond to better performance on BS, BLL, and concordance. Similarly to the previous experiment, fig. 5 shows that game methods attain better uncensored test-set BS and BLL on survival-MNIST than likelihood-based training does. The games likewise attain higher concordance. NLL training performs better at the metric it directly optimizes. This experiment establishes that it possible to optimize through deep convolutional models with batch norm and pooling using the game training methods.
Results.

---

1.6
1.6
0.3
12
18200 15002
5200
10000
15000
10300
10000
5000
10000
15200
(a) Uncensored BS
(b) Uncensored Neg BLL
(c) Concordance
(d) Categorical NLL
Figure 5: Test set evaluation metrics (y-axis) on survival-MNIST versus number of training points (X-axis) for three methods. Each point in the plot represents the evaluation metric value of fully trained model with that number of training points. Lower is better for all the metrics except for concordance.
5.3
Real Datasets
Data.
We use several datasets used in recent papers [Chen, 2020, Kvamme al., 2019] and available in the python packages DeepSurv [Katzman et al., 2018] and PyCox [K [Kvamme et al., 2019], and the Survival [Therneau, 2021]. The datasets are:
Molecular Taxonomy of Breast Cancer International Consortium (METABRIC) [Curtis et al., 2012]
Rotterdam Tumor Bank (ROTT) [Fockens al., 2000] and German Breast Cancer Study
Group (GBSG) [Schumacher al., 1994] combined into one dataset (ROTT. & GBSG)
Study Understand Prognoses Preferences Outcomes and Risks of Treatment (SUPPORT) [Knaus et al., 1995] which includes severely ill hospital patients
For more description see appendix For real data, there is no known ground truth for the censoring distribution, which means evaluation requires assumptions. Following the experiments in K Kvamme al. [2019], we assume that censoring marginal estimate with KM to evaluate models."
Results. On METABRIC, games attain lower (better) KM-weighted BS and BLL than NLL-training when the number of datapoints is small, and have better concordance and NLL though they do not directly optimize them. As data size increases, all methods converge to similar performance. On ROTT. & GBSG, the trend similar: games optimize the BS and BLL more rapidly as function of training set size than NLL-training does. Again, all methods converge to similar performance in all metrics when the number of datapoints is large enough. All methods perform similarly on SUPPORT.
Caveats. First, though popular, these survival datasets are -iw-dimensional, any of the objectives can perform well on the metrics with just several hundred points. We see that this distinet from MNIST, where thousands of points were required to improve performance. Second, though possible, may not be true that censoring is marginal on these datasets, which would mean that BS and BLL results only have their interpretation conditional on particular set of covariates. Our method is also correct when censoring marginal though. Lastly, no method stable for metrics, for all training sizes, all seeds for all datascts.
6 Related Work
Nuisance parameters. Under non non-informative censoring, the censoring distribution is unrelated to the failure distribution, but estimating it can help improve learning the failure distribution: here, the censoring distribution a nuisance parameter. Existing causal estimation methods propose two-stage procedures where the first stage estimates the nuisance- parameetrr (e.g. propensity score) and the second stage uses the learned nuisance-parann parameter as-is to define an estimator or loss function for the target parameter (causal effect). [Van Der Laan and Rubin, 2006, Van der Laan and Rose, 201 I, Chernozhukov et al., 2018, Foster and Syrgkanis, 2019]. In this work, we instead show that
'This is also the route taken in the R packages Survival [Therneau, 2021], PEC [Mogensen et al., 2012), and riskRegression [Gerds et al., 2020].

---

2.7
0.90
0.55
24
616
0.99
045
Dogar
(a) K.M-eiihted BS
(b) KM-weighted Neg BLL
(c) concordance
(d) categorical NLL
Figure Test set evaluation metrics (y-axis) on METABRIC versus number of training points (x-axis) for three methods. Each point in the plot represents the evaluation metric value of fully trained model with that number of training points. Lower hetter for all the metrics except for concordance.
2.7
6/1
6/8
0.00
2.3
100
201
108
100
293
100
190
200
109
155
201
(a) KM weighted BS
(b) KM-weighted Neg BLL
(c) concordance
(d) categorical NLL
Figure Test set evaluation metries (y-axis) on ROTT. & GBSG versus number of training points (X-axis) for three methods. Each point in the plot represents the evaluation metric value of fully trained model with that number of training points. Lower better for all the metrics except for concordance.
9221
654
230
922)
153
0.52
532
2.9
270
0290
1190
150
(a) KM-whiihttd BS
(b) KM- weightetted Neg BLL
(c) concordance
(d) categorical NLL
Figure Test set evaluation metries (y-axis) on SUPPORT versus number of training points (X-axis) for three methods. Each point in the plot represents the evaluation metric value of fully trained model with that number training points. Lower better for all the metrics except for concordance.
estimating the target (failure model objective or failure model itself) can benefit from coupled estimation procedure where the nuisance parameter (censoring model) also trained simultaneously. The failure model needs the censoring distribution to compute BS but censoring estimation needs the failure model, and despite this circular dependence, we characterize a case where the game training leads to the true data generating distributions.
Double Robust Censoring Unbiased Transformations. For functions h, Rubin and van der Laan [2007] estimate conditional mean [T, XX under censoring using double-robust estimator: given estimates of the conditional failure and censoring cumulative distribution functions (CDFs) F( and (c) ) the estimator of 1e (T, X] is unbiased when either nuisance CDF correct. However, here we are concerned with estimating a quantity to be used as a loss for learning F. We therefore presumably do not already have an estimate Í to be used in doubly-robust estimator.
Censoring Unbiased Losses for Deep Learning. Steingrimsson and Morrison [2020] build failure model loss functions based on the estimators from Rubin and van der Laan [2007]. Their BS loss extends IPCW BS estimation to the doubly-robust case and to knowledge is the first instance of

---

IPCW-based estimation procedures being used in a general purpose way to define loss functions for deep survival analysis.
However, their censoring distribution estimated once before training and held fixed rather than incorporated into a joint training procedure in the games in this work. The fixed censoring estimate is implemented by KM, which assumes marginal censoring distribution. Making the marginal assumption when censoring is truly conditional should not yield a performant model under the BS criteria since the training objective does not directly estimate or optimize the true BS that would be measured under no censoring. When marginal censoring does hold, KM estimation, which is non parametric, may be a simpler and stable choice versus the game, depending on sample size, data variance, and conditional parameterization assumptions. But since it is in general unknown censoring is marginal, we use conditional models which are also correct under marginal censoring.
7 Discussion
In this work, we propose a new training method for survival models under censored data. We argue that on finite data, it is important to close the gap between training methodology and the desired evaluation criteria. We showed in the experiments that better NLL does not correspond to better performance on BS, BLL, and concordance, all evaluations of interest in survival analysis.
The main trend in our experimental results was that data size matters: smaller meant the game methods performed better than NLL and enough data meant that they perform similarly, which is expected since all objectives are proper. However enough data is hard to define: it depends on dimensionality and on the data generating distribution and model class. It is a great direction to build more precise understanding on how objectives behave differently even when they have the same optimum on infinite data: though likelihood is known to be asymptotically efficient for survival analysis, more analysis is necessary for comparing likelihood and Brier score's 's trade-offs on small sample sizes.
In the experiments, we focus on categorical models. On the other hand, proposition applies to continuous distributions as well, provided that positivity can be satisfied. However, this is rare in practice because most survival data has a final follow-up time, and even before this time there may be very few samples with late times [Gerds et al., 2013]. For this reason, working with continuous distributions requires picking a truncation time and playing games only up to that time.
Evaluation on real data under censoring requires assumptions. It is important to further consider how to better assess test-set performance on metrics such as BS, BLL, and concordance. Because concordance is not proper [Blanche et al., 2019], we do not build objectives from it here, but it too is not invariant to censoring. Regarding games, we showed properties about stationary points. More analysis is necessary to describe important convergence properties of optimizing these games.
Social Impact. Survival models are deployed in hospital settings and have high impact on public health. In this work, we saw benefits of a new training approach for these models, but no training method panacea. Practitioners of survival analysis must take great care to consider various training and validation approaches, as well as consider possible test distribution shifts, prior to deployment.
Acknowledgments and Disclosure of Funding
This work was supported by:
NIH/NHLBI Award R01HL148248
NSF Award 1922658 NRT-HDR: FUTURE Foundations, Translation, and Responsibility for Data Science.
NSF Award 1514422
TWC: Medium: Scaling proof-based verifiable computation
NSF Award 181 1815633 SHF
10

---

References
A. M. Alaa and M. van der Schaar. Deep multi-task gaussian processes for survival analysis with competing risks. In Proceedings of the 31st International Conference Neural Information Processing Systems, pages 1326-23344 Curran Associates Inc., 2017.
P. K. Andersen, O. Borgan, R. D. Gill, and N. Keiding. Statistical models based on counting processes. Springer Science Business Media, 2012.
A. Avati, T. Duan, Zhou, K. Jung, N. H. Shah, and A. Y. Ng. Countdown regression: Sharp and calibrated survival predictions. In A. Globerson and R. Silva, editors, Proceedings of the Thirty-Fifte Conference on Uncertainty in Artificial Intelligence, UAI 2019, Tel Aviv, Israel, July 22-25, 2019, page 28. AUAI Press, 2019. URL http: auai www.aaorr//aaaa.aa..aaaeeaaa papers/ 28. pd..ff
H. Bang and J. M. Robins. Doubly robust estimation in missing data and causal inference models. Biometrics, 61(4))962-9)) 9732 2005.
Blanche, J.-F. Dartigues, and H. Jacqmin- Gadda. Review and comparison of roc curve estimators for time-dependent outcome with marker-dependent censoring. Biometrical Journal, 55(5): 687-704, 2013.
P. Blanche, M. W. Kattan, and T. A. Gerds. The c-index is not proper for the evaluation of of-year predicted risks. Biostatistics. 20(2):347-357, 2019.
G. W. Brier and A. Allen, Verification of weather forecasts. In Compendium of meteorology, pages 841-848. Springer, 1951.
G. H. Chen. Deep kernel survival analysis and subject-specific survival time prediction intervals. In Machine Learning for Healthcare Conference, pages 537-565. PMLR, 2020.
V. Chernozhukov, D. Chetverikov, M. Demirer, E. Duñlo, C. Hansen, W. Newey, and J. Robins. Double/debiased machine learning for treatment and structural parameters, 2018.
C. Curtis, S. P. Shah, S.-F. Chin, G. Turashvili, O. M. Rueda, M. J. Dunning, D. Speed, A. G. Lynch, S. Samarajiwa, Y. Yuan, et al. The genomic and transcriptomic architecture of 2,000 breast tumours reveals novel subgroups. Nature, 486(7403) 2012.
M. H. DeGroot and S. E. Fienberg. The comparison and evaluation of forecasters. Journal of the Royal Statistical Society: Series D (The Statistician), 32(11)2): 32(1-2):12-22, 1983.
O. V. Demler, N. P. Paynter, and N. R. Cook. Tests of calibration and goodness-of of-fil in the survival setting. Statistics in medicine,
2015.
A. Foekens, H. A. Peters, M. P. Look, Portengen, M. Schmitt, M. D. Kramer, N. Brünner, F. Jänicke, M. E. Meijer-van Gelder, S. C. Henzen-Logmans, et al. The urokinase system of plasminogen activation and prognosis in 2780 breast cancer patients. Cancer research. 60(3): 636-643, 2000.
D. J. Foster and V. Syrgkanis. Orthogonal statistical learning. arXiv preprint arXiv:1901.0003 2019.
S. Fotso. Deep neural networks for survival analysis based on a multi-task framework. arXiv preprint 17X.0:8801.05512.. 2018
B. Gensheimer, Michael Fand Narasimhan. A scalable discrete-time survival model for neural networks. PeerJ 7:e6257, 2019.
T. A. Gerds and M. Schumacher. Consistent estimation of the expected brier score in general survival models with right-censored event times. Biometrical Journal, 1806)::022--0000 2006.
T. A. Gerds, M. W. Kattan, M. Schumacher, and C. Yu. Estimating time-dependent concordance index for survival prediction models with covariate dependent censoring. Statistics in Medicine, 32 (13):2173-2184, 2013.
11

---

T. A. Gerds, P. Blanche, T. H. Scheike, R. Mortensen, M. Wright, N. Tollenaar, J. Muschelli, U. B. Mogensen, and Ozenne. Package 'riskregression' 2020.
T. Gneiting and A. E. Raftery. Strictly proper scoring rules, prediction, and estimation. Journal of the American statistical Association, (02(477)335--358 2007.
M. Goldstein, X. Han, A. M. Puli, A. Perotte, and Ranganath. X-cal: Explicit calibration for survival analysis. Advances in Neural Information Processing Systems, 33, 2020.
I. J. Goodfellow, J. Pouget- Abadie. M. Mirza, B. Xu, D. Warde-Farlcy, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial networks. arXiv preprint arXiv:/406.2600 2014.
E. Graf, C. Schmoor, W. Sauerbrei, and M. Schumacher. Assessment and comparison of prognostic classification schemes for survival data. Statistics medicine, 18(17-18)2529-2555 1999.
H. Haider, B. Hoehn, S. Davis, and R. Greiner. Effective ways to build and evaluate individual survival distributions. Journal of Machine Learning Research, 2 (1(85):1--0 16,, 2020.
E. Harrell Jr, K. Lee, and D. B. Mark. Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors. Statistics in medicine, (544)33113377 1996.
H. Hung and C.-T. Chiang. Estimation methods for time-dependent auc models with survival data. Canadian Journal of Statistics, 38(1)88-22) 26, 2010a.
H. Hung and C.-t. Chiang. Optimal composite markers for time-dependent receiver operating characteristic curves with censored survival data. Scandinavian journal of statistics, 37(4))664- 679, 2010b.
D. Kalbfleisch and R. L. Prentice. The Statistical Analysis of Failure Time Data. Wiley Series in Probability and Statistics. John Wiley & Sons, Inc., 2 edition, 2002.
Kamran and J. Wiens. Estimating calibrated individualized survival curves with deep learning. 2021.
J. L. Katzman, U. Shaham, A. Cloninger, J. Bates, T. Jiang, and Y. Kluger. Deepsurv: personalized treatment recommender system using a cox proportional hazards deep neural network. BMC medical research methodology, 18(1):24, 2018.
W. A. Knaus, E. Harrell, J. Lynn, Goldman, R. Phillips, A. F. Connors, N. V. Dawson, W. J. Fulkerson, R. M. Califf, N. Desbiens, et al. The support prognostic model: Objective estimates of survival for seriously ill hospitalized adults. Annals of internal medicine, 122(3):191-203, 1995,
A. Kumar, S. Sarawagi, and U. Jain. Trainable calibration measures for neural networks from kernel mean embeddings. In International Conference Machine Learning, pages 2805-2814, 2018.
H. Kvamme and Ø. Borgan. The brier score under administrative censoring: Problems and solutions. arXiv preprint arXiv:
2019a.
H. Kvamme and Ø. Borgan. Continuous and discrete-time survival prediction with neural networks. 2019b.
arXiv preprint arXiv:
H. Kvamme, Ørnulf Borgan, and I. Scheel. Time-to-event prediction with neural networks and cox regression. Journal of Machine Learning Research, 10(129):1-30, 2019. URL http://jmlr. org/ httpppppaa/v///1112 20/0///2//111444 html.
C. Lee, W. R. Zame, J. Yoon, and M. van der Schaar. Deephit: A deep learning approach to survival analysis with competing risks. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
C. Lee, W. Zame, A. Alaa, and M. Schaar. Temporal quilting for survival analysis. In The 22nd international conference on artificial intelligence and statistics, pages 596-605. PMLR, 2019.
A. Letcher, D. Balduzzi, S. Racaniere, J. Martens, Foerster, K. Tuyls, and T. Graepel. Differentiable game mechanics. The Journal of Machine Learning Research, 20(1):3032-3077 2019.
12

---

X. Miscouridou, A. Perotte, N. Elhadad, and R. Ranganath. Deep survival analysis: Nonparametries and missingness. In Machine Learning for Healthcare Conference, pages 244-256, 2018.
U. B. Mogensen, Ishwaran, and T. A. Gerds. Evaluating random forests for survival analysis using prediction error curves. Journal of Statistical Software, 50(11):1-23, 2012. URL https: // www. www.tssttoff.tt..mmmm wwwpe/ofoottgg////////// org/v50/111.
J. w. Neumann and O. Morgenstemn. Theory of Games and Economic Behavior. Princeton University Press, 2007.
S. Pölsterl. Sebastian pölsterl, Jul 2019. survival-andn -aralllaaalyysi----eeeeeeeee -ww.--rrrree---erningg//e/
URL https://k-d-w.ort http:///.4ow..orrg/oo////1//1/0
R. Ranganath, A. Perotte. N. Elhadad, and D. Blei. Deep survival analysis. arXiv preprint arXiv: 1608. 02188, 2016.
K. Ren, J. Qin, Zheng, Z. Yang, W. Zhang, L. Qiu, and Yu. Deep recurrent survival analysis. In Proceedings of the AAAI Conference Artificial Intelligence, volume 33, pages
2019.
D. Rubin and M. J. van der Laan. A doubly robust censoring unbiased transformation. The international journal of biostatistics, 3(1), 2007.
M. Schumacher, G. Bastert, H. Bojar, K. Hübner, M. Olschewski, W. Sauerbrei, C. Schmoor, C. Beyerle, R. Neumann, and H. Rauschecker. Randomized 2 x trial evaluating hormonal treatment and the duration of chemotherapy in node-positive breast cancer patients. german breast cancer study group. Journal of Clinical Oncology, 12(10):2086225) 2093, 1994.
M. Sloma, Syed, M. Nemati, and K. S. Xu. Empirical comparison of continuous and discrete- time representations for survival prediction. In Survival Prediction-Aggoritmms Challenges and Applications, pages 118-131. PMLR, 2021.
A. Steingrimsson and S. Morrison. Deep learning for survival outcomes. Statistics in medicine, 39 (17):2339-233) 2249, 2020.
L. M. Sullivan, J. M. Massaro, and R. B. D' Agostino Sr. Presentation of multivariate data for clinical use: The framingham study risk score functions. Statistics in medicine, 23110))663111660, 2004.
T. M. Themeau. A Package for Survival Analysis in R, 2021. URL https://CRAN. org/packago-survva
package version 3.2-11.
H. Uno, Cai, L. Tian, and L.-J. Wei. Evaluating prediction rules for t-year survivors with censored regression models. Journal of the American Statistical Association, 102(478):577-537, 2007.
M. Van der Laan and S. Rose. Targeted learning: causal inference for observational and experi- mental data. Springer Science & Business Media, 2011.
M. J. Van Der Laan and D. Rubin. Targeted maximum likelihood learning. The international journal of biostatisties, 2(1), 2006.
M. J. Van der Laan, M. Laan, and J. M. Robins. Unified methods for censored longitudinal data and causality. Springer Science & Business Media, 2003.
M. Wolbers, P. Blanche, M. T. Koller, J. C. Witteman, and T. A. Gerds. Concordance for prognostic models with competing risks. Biostatistics, 15(3):526-539, 2014.
S. Yadlowsky, Basu, and L. Tian. A calibration metric for risk scores with survival data. In Machine Learning for Healthcare Conference, pages 424-450, 2019.
C.-N. Yu, Greiner, H.-C. Lin, and Baracos. Learning patient-specific cancer survival distributions as a sequence of dependent regressors. In Advances in Neural Information Processing Systems, pages 1845-1853, 2011.
C. Zhong and R. Tibshirani. Survival analysis as a classification problem. 119,,, 2019.
arXiv preprint
13


Reviews:
[{"review_id": "e7NeNnGp5wV", "reviewer": "Reviewer_XC9X", "summary": "This paper proposes a new training method for discrete-time survival models, where not only a failure model but also a censoring model are simultaneously trained to optimize a set of time-dependent inverse probability of censor-weighting (IPCW) criteria. Discrete-time models trained with different objective functions have been compared on simulated data and multiple real datasets. ", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n- The new training method that directly optimizes IPCW criteria for discrete-time models is novel and has been shown to have better test BS, BLL, and concordance on simulated data, especially for small sample sizes.\n- They show that the IPCW game method has a stationary point at the data-generating distributions and IPCW BS has only one stationary point.\n- The paper is generally well written and easy to follow.\n\nWeakness:\n- This paper claims that \"deep discrete-time models have become a default choice even when data is continuous\", this is not true as there are some recent deep continuous-time survival models [1,2] that are shown to have advantages over discrete-time models.  \n- The ultimate goal is to obtain an estimated event time distribution that is close to the truth. Different evaluation criteria characterize how far the estimated distribution is from the truth in different perspectives, while the common optimum is expected to be the truth. Why optimizing the BS and BLL is particularly preferable / desired in the context of survival? I find the current argument \"a conceptually simple way to evaluate models that is particularly suitable for discrete models is to apply binary classification losses at various time horizons, for instance Brier score (BS) for 5-year survival, or similarly, Bernoulli log-likelihood\" is weak. The authors do find some empirical improvement on other criteria when using the new training method (BS, BLL based), but high-level intuition on why it is expected to work better would be appreciated.\n- How is 20 selected as the number of categorical bins in experiments? The number of objectives to simultaneously optimize in the proposed method scales linearly with the number of bins, but, in general, discrete-time survival models require enough bins to well approximate a continuous event time. This would affect both the computation complexity and the numerical stability. Additional experiments would be desired.\n- The assumption used in building the IPCW objective ($T \\perp C| X$) and that used in evaluation metrics ($T \\perp C$) for real datasets are inconsistent.\n- I am trying to understand why the proposed method works better for small sample sizes. This observation is relatively counterintuitive for me. Because the game method simultaneously models the failure time and censoring time distribution and thereby contains more parameters than the NLL method. But it works better for small sample sizes. \n- References need to be updated such as [1] is still cited as its arXiv version. Please check other citations as well.\n\nReferences:\n\n[1] H. Kvamme, Ørnulf. Borgan, and I. Scheel. Time-to-event prediction with neural networks and cox regression. Journal of Machine Learning Research, 20(129):1–30, 2019.\n[2] Tang, W., J. Ma, Q. Mei, and J. Zhu (2020). SODEN: A scalable continuous-time survival model through ordinary differential equation networks.arXiv preprint arXiv:2008.08637.\n\n\n-------------------------------------------------------\nI thank the authors for their thorough responses. After the discussion, I raised my evaluation.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations have been discussed. More potential limitation: the game method is currently developed for the discrete-time models. But the inverse probability game seems to be general enough for continuous-time survival models as well, which may mitigate the issue of a large number of simultaneous objectives in the IPCW game for discrete-time models.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "ZFAFgEDWps", "reviewer": "Reviewer_MkMh", "summary": "The paper introduces a non-adversarial approach for simultaneously learning events and censoring cumulative distribution functions (CDF) at prespecified discrete-time points by optimizing inverse weighted Brier score (BS) and Bernoulli log-likelihood (BLL). Further, the theoretical analysis illustrates that the proposed approach achieves data event and censoring distributions at stationary points. Experimental results on (semi-)synthetic and real-world datasets demonstrate that the proposed achieves better concordance, Brier score, and Bernoulli log-likelihood than an approach that optimizes the standard negative log-likelihood (NLL) given small sample sizes.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, the paper is well organized and easy to follow.  Moreover, provided theoretical analysis supports the recovery of the data distribution at stationary points. However, below are the key limitations.\n\n**Significance**\n- The paper focuses on discrete-time models, which are challenging to scale for large time horizons. Can the proposed approach be adapted to extensive continuous-time survival methods? Also, the paper title needs to be updated to include discrete-time survival since this is the focus of the submission.\n- The paper needs to motivate the utility of the proposed metrics (time-varying BS, BLL) relative to standard metrics such as concordance index, relative absolute error, etc. Also, why are integrated BS(BLL) not considered?\n- While the BS (or BLL) are proper scoring rules, is there evidence to support that IPCW-BS(BLL) are also proper scoring rules?\n\n**Underwhelming experimental results**\n- The paper should also benchmark against other approaches besides negative log-likelihood (NLL), including CoxPH and alternative approaches that optimize BS, such as [1].\n- Experimental results on real-world datasets demonstrate that the proposed approach has no clear advantage over the NLL. \n- Also, why are the sample sizes on real-world datasets limited to 200? The SUPPORT dataset has ~10,000 observations.\n- The paper should also provide qualitative results,  e.g.,  predicted CDFs against ground truth events or censoring times.\n\n**Below are the key missing references**\n- [1]  Avati et al., \"Countdown regression: sharp and calibrated survival predictions\", In Uncertainty in Artificial Intelligence, 2020.\n\n--------------------------------------------------------------------------------------------------------------------------------------------------------------------------\nI thank the authors for addressing my concerns and providing additional experimental results. Hence, I have increased my score.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The discussion needs to include the limitations of discrete-time models and the chosen metrics (BS, BLL).", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5 hours"}, "extra_text": {}}, {"review_id": "Uy65A3GriLW", "reviewer": "Reviewer_mDpZ", "summary": "In this paper the authors exploit the symmetry present in the survival analysis setting between the roles of C and Y to frame the problem as a game with the parameters as (unique with certain conditions) solution.\nThe authors prove then provide experiments showing not only that the approach is competitive but can even outperform the standard approach in the low data regime.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The authors adapt the IPCW framework, which is known to have good generalization properties (see [1]) even when the weights have to be estimated with a twist: instead of estimating S_C first once and for all, they iteratively refine the estimations of S_C and S_Y be exploiting the fact they both symmetrically are the solution to an IPCW ERM problem.\n\n\n[1]G. Ausset, S. Clémençon, and F. Portier, “Empirical Risk Minimization under Random Censorship: Theory and Practice,”. Available: http://arxiv.org/abs/1906.01908\n\nThe main contribution is the proof that for certain losses the solution of interest is indeed a stationary point and that for a particular loss, it is the only saddle point.\n\nThe arguments are presented clearly and are convincing but some questions may benefit from more details:\n\n1. The approach proposed is conceptually very similar to the doubly robust approach [2], has the link been studied by the authors?\n\n[2] D. Rubin and M. J. van der Laan, “A Doubly Robust Censoring Unbiased Transformation,” The International Journal of Biostatistics, vol. 3, no. 1, Jan. 2007, doi: 10.2202/1557-4679.1052.\n\n2. More details on the training procedure would be welcome (I personally never managed to achieve convergence using this method.)\n\nFinally, some typos in the manuscript remain, some of them unfortunate (line 244: not invariant => invariant).\n\nOverall I think this is a very interesting work and a worthwhile contribution.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors adequately addressed the limitations and potential negative societal impact of their work?", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "Af4rPdBNm1", "reviewer": "Reviewer_Xu5C", "summary": "The authors are interested in the setting of predicting discrete outcomes within time intervals of interest (ex. 5-year survival, 10-year survival) from time-to-event data, i.e. the setting in which some observations may be right-censored. The conventional approach relies on inverse probability of censorship weighting (IPCW), which first estimates the distribution over time-to-censorship $p(c|x)$ (for example, via a Kaplan-Meier estimate), then re-weights the data to estimate the distribution over time-to-event p(t|x)$. \n\nThe authors instead propose *Inverse-Weighted Survival Games*, in which they learn *both* the time-to-censorship and the time-to-event distributions in a non-adversarial game, using each to re-weight the other's loss. On the theoretical side, they show that when the loss function is proper, the resulting game has the true distributions as a stationary point. Moreover, when the loss function is the Brier Score, this is the only stationary point. On the experimental side, they run experiments on simulations and public datasets (Survival-MNIST, METABRIC, SUPPORT) and show that IPCW games improve upon NLL-trained models in terms of concordance, KM-weighted log-likelihood, and KM-weighted Brier Score in the data-limited regime.", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I'm a fan of this paper. The idea behind Inverse-Weighted Survival Games is simple and elegant. The background and methods sections are very well-written.\n\nMy primary point of concern is with respect to the experiments. I have several questions:\n\n1/ What is the definition of the \"NLL\"-optimized categorical model? My understanding is that it's a simple optimization of the discrete-time classification model, without accounting for censorship. If so, then without accounting for censorship the metric isn't very useful -- I don't think it'd be worth including in the results.\n\n2/ I'd strongly suggest the authors add as a benchmark a discrete-time categorical model optimized with IPCW via a simple Kaplan-Meier estimate of the censorship distribution $p(c)$. This is important because it's the most common way such models are developed today.\n\n3/ How are the confidence intervals shown in Figures 3-7 derived? \n\n4/ What's the motivation for presenting the plots as a function of the number of training points? I think the training curves will not be relevant to most readers of the paper -- instead, what they'll care about is end-to-end performance. Instead of Figures 3-7, can we have a table of results measuring BS, BLL, Concordance and Calibration at *convergence*? I'd insist on showing confidence intervals for the above as well, for example by showing the standard deviation of each metric over random samples of the evaluation procedure. \n\n5/ I'd strongly sugest the authors include a measure of calibration in their evaluation criteria, alongside concordance (which measures discrimination). One example would be L1 calibration error, where censorship is taken into account via a Kaplan-Meier estimate for the observed incidence rate in each bin.\n\nI'd be happy to revise my rating of this paper if the authors are able to address the above questions.\n\nMiscellaneous typos and comments:\n\n1/ L166: missing commas on $[128, 256, 64$]\n\n2/ How many data points are in the simulation dataset?\n\n3/ What is the rate of censorship in each of the datasets?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"ethical_concerns": "No ethical concerns. ", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: To be updated - This is one, where I think the score underrates the work and should probably be accepted once reviewers update their scores based on the author response. I just emailed all of the reviewers who have yet to indicate they have read the author response. 
Decision: Accept (Poster)