Simpler Calibration for Survival Analysis
Authors: Hiroki Yanagisawa
Toshiya Iwamori
Akira Koseki
Michiharu Kudo
Mohamed Ghalwash
Prithwish Chakraborty
Venue: ICLR 2022 Submitted (2022)
Keywords: survival analysis
time-to-event analysis
calibration

Abstract: Survival analysis, also known as time-to-event analysis, is the problem to predict the distribution of the time of the occurrence of an event.  This problem has applications in various fields such as healthcare, security, and finance.  While there have been many neural network models proposed for survival analysis, none of them are calibrated.  This means that the average of the predicted distribution is different from the actual distribution in the dataset.  Therefore, X-CAL has recently been proposed for the calibration, which is supposed to be used as a regularization term in the loss function of a neural network.  X-CAL is formulated on the basis of the widely used definition of calibration for distribution regression.  In this work, we propose new calibration definitions for distribution regression and survival analysis, and demonstrate a simpler alternative to X-CAL based on the new calibration definition for survival analysis.


Under review as a conference paper at ICLR 2022
SIMPLER CALIBRATION FOR SURVIVAL ANALYSIS
Anonymous authors
Paper under double-blind review
ABSTRACT
Survival analysis, also known as time-to-event analysis, is the problem to predict the distribution of the time of the occurrence of an event. This problem has appli- cations in various fields such as healthcare, security, and finance. While there have been many neural network models proposed for survival analysis, none of them are calibrated. This means that the average of the predicted distribution is different from the actual distribution in the dataset. Therefore, X-CAL has recently been proposed for the calibration, which is supposed to be used as regularization term in the loss function of a neural network. X-CAL is formulated on the basis of the widely used definition of calibration for distribution regression. In this work, we propose new calibration definitions for distribution regression and survival analy- sis, and demonstrate a simpler alternative X-CAL based on the new calibration definition for survival analysis.
INTRODUCTION
Survival analysis, also known as time-to-event analysis, is the problem predict the time of the occurrence of an event. In healthcare applications, the event typically corresponds to a death or the onset of disease in a patient. The time between well-defined starting point and the occurrence of the event is called the survival time or failure time. In survival analysis, we usually estimate the dis- tribution of the survival times of patients. Survival analysis has important applications in healthcare as well as various other fields (c.g., credit scoring (Dirick et al. (2017) and fraud detection (Zheng al., ).19)) The recent progress of prediction models for survival analysis has been summarized in survey paper (Wang et al. 2019).
In survival analysis, datasets are often censored, which means that events of interest might not be observed for some instances. This may be due to either the limited observation time window or missing traces caused by other irrelevant events. Typical censored data are right censored data. These are the data points whose exact times of the events are unknown; we know only that the events had not happened up to a certain time. In this paper, we focus on the uncensored data and the right censored data, shown in Figure Here, the event for data point æ; is observed during the period of study and hence this data is categorized as uncensored data. The data points 22 and œ. are categorized right censored data because we did not observe the events during the period of study. The time between a well-defined starting point and the last observation time (e.g., the time of the end of study) is called the censoring time.
One of the classical methods to solve the survival analysis problem is the Kaplan-Meier estima- tor (Kaplan & Meier (958). This is a non-parametric method estimate the distribution of the survival times as survival function S(f), where the value S(t*) for a specific time t" represents the survival rate at time t" (i.e., the ratio of the patients who survived time t*). It is easy estimate the survival function S(t) if the dataset contains only uncensored data points, but the Kaplan- Meeie estimator designed to work for datasets that include censored data. Here, we briefly explain the algorithm of the Kaplan-Meier estimator. Let {4:,/ be the set of distinct times when at least one uncensored event was observed in the dataset. Let d, be the number of (uncensored) events that happened exactly at time t,. and let n, be the number of data points that are known to have survived at time f.. Then, the Kaplan- Meilaan--MMeea--e estimator outputs the survival function
d,
S(t)
II St

---

Under review as a conference paper at ICLR 2022
Start of study
End of study
Uncensored
1
X
Right censored
22
0.04
Right censored
33
4000
Time
Figure 1: Uncensored data and right cen- sored data, where the symbols x show event occurrences.
Figure 2: Survival function S(t) estimated by the Kaplan-Meie estimator for the flchain dataset.
Figure shows example of the survival function S(t) estimated by the Kaplan-Meier estimator for the flchain dataset (Dispenzieri et al. 2012). Here, we can see that the survival rate at time must be zero, but the
t = 2500 is approximately 80%. Note that the true survival rate S(t) at f Kaplan-Meie estimator outputs the survival function S(t) only for time t E [0, tmax]. where max the maximum survival time of the uncensored data points in dataset. This is because we cannot estimate the survival rate S(t) for the time > fmax-
A drawback of the Kaplan-Meier estimator is that it outputs survival function S(t) for the entire population and not for specific patient. Therefore, there have been many algorithms to estimate the survival rate (t) for each patient
enable personalized medicine (Wang et al. 2019). In particular, many neural network models that predict the survival function 5(4)) have
been proposed (Lee et al. 2018 Ren et al. 2019 Zheng et al. 2019 Tjandra al. 202T
1.1 CALIBRATION
When we use a prediction model, should be calibrated. binary classification, this means that a prediction model that outputs a confidence of true label for an input is expected to satisfy the condition that the average of the confidence values over all inputs are equal to the ratio of the data points with a true label in the dataset. For example, if dataset contains 40% of the data points with a true label, then we expect that a calibrated prediction model will output a confidence of 0.4 average. Even though calibration is important for prediction models, and neural network models have been widely used for prediction models, (Guo et al. 2017) showed that neural network models often miscalibrated.
In regression analysis, quantile-based calibration widely used as the definition of calibra- tion (Kuleshov al./ 2018 Song al. 2019 Cui et al.] [2200] Zhao et al.] 2202).. In survival analysis, the definition of calibration is based on this definition for regression analysis. Goldstein al. (2020) showed a method train neural network model achieve calibration by adding new regularizer, X-CAL, to the loss function of the neural network to achieve calibration during the training. This method is in contrast to the widely-used calibration methods for regression analysis such as Platt scaling and isotonic regression, which are post-training methods.
Note that a calibrated model not necessarily useful. For example, in binary classification we can construct trivial calibrated model that always outputs the ratio of true label data points in a dataset as a confidence value. Therefore, a useful prediction model also needs to be sharp, which means that it outputs a confidence value close one zero for each input. To obtain prediction model that is both calibrated and sharp, one approach is to train a neural network with a loss function consisting of one loss term aimed at sharp prediction and another aimed at calibration. X-CAL utilizes this approach in that it is regularizer for calibration and used combination with another loss term that aims at sharp prediction.
2

---

Under review as a conference paper at ICLR 2022
1.2 OUR CONTRIBUTIONS
In this paper, we propose a Kaplan-Meier regularizer as a simpler alternative to X-CAL. An advan- tage of our regularizer is that the obtained prediction model is calibrated for any time / [0, [.a]] Another advantage is that our regularizer does not require any hyperparameter, whereas X-CAL re- quires hyperparameters. The idea behind our new regularizer simple: just reduce the difference between the average predicted survival function and the Kaplan-Meier survival function (see Fig- ure 3.. Our new regularizer based on new definition of calibration for survival analysis, and we discuss its advantages in Section
2 SURVIVAL ANALYSIS
We formally describe the problem settings of regression analysis and survival analysis. In regression analysis, assume that there is unknown probability distribution P the sample space X V, where X' is the domain of features and y/ is the interval [-oc, oo). We refer to the associated random variables with capital letters (i.e., X and Y) and realizations with lower case letters (ær, y) P. We P.
assume that we can obtain independent samples {(ar,, w) from X' x y according to distribution
In survival analysis, we assume that there is an unknown probability distribution Q on the sample space X' T x C. where X' is the domain of features and T and C' are time intervals [0, oo). We refer to the associated random variables with capital letters (i.e., X, T, and C) and realizations with lower case letters (ax t, c) Q. The random variable T corresponds to the survival time (i.e., the time of the occurrence of an event), which might not be observable due to the censoring, and the random variable C' corresponds to the censoring time. We assume that the censoring time do for an uncensored event. The random variable defined by Z min C} corresponds the time of the last observation (i.c., the survival time or the censoring time). Different from the regression analysis, cannot obtain samples {xx, G)) from X' X T x C according to distribution Q due to the censoring in survival analysis. However, we assume that we can obtain independent samples D [ 2t, 6,)) instead, where 6; is binary value indicating if the i-th data point is censored or not. Here an uncensored data point (æ,, zj, 6c) E D satisfies 6, = and z; t and a censored data point (æ.,, Zi, 6.) E D satisfies
0 and z; = C. Hence ò; = 0 means that know only the fact that t; 2 c; and the exact survival time t; is unknown.
The task of survival analysis to predict the probability of an event interest occurring time for æ E X' as a probability distribution function f (t] This function f(t(æ) is often represented in other equivalent forms. For example, it can be represented as its cumulative distribution function (CDF)
F( = 1 [(Tla=)dr
or as the survival function S(I(#)) defined by
S(((ae) F(11z).
Intuitively, F(t æ)) represents the probability of observing the event by time t for and the survival function ((1)2) represents the probability of not-observing the event until time for æ.
Many neural network models have been proposed for survival analysis (e.g., (Lee et al. 2018 Ren eet al., 2019; Zheng et al.) 2019, Tjandra et al.) 2021)). They output the probability distribution the form of (t (f(xx) (T)x), or something equivalent these forms, where is the parameters of the neural network.
3 QUANTILE-BASER CALIBRATION
In this section, we review the definitions calibration for distribution regression and survival anal- ysis in the literature. First, consider the distribution regression whose task is to predict the distribution of the target variable as a CDF Fo(y)z) for x E X, where is the parameters of the prediction model. In distribution regression, quantile-based calibration (Kuleshov al. 2018)) is widely used as the definition of calibration (c.g., (Song et al.) 2019 Cui et al. 2020 Zhao et al. 2020)).
3

---

Under review as a conference paper at ICLR 2022
Definition 3.1 (Quantile-based calibration.) A prediction model Fe (y) for for distributtion regression quantile-calibratee if this equation holds for any quantile level T E [0, 1]:
P (Y () S T) =
If we can compute the inverse of Fo(y)a:), we can rewrite Eq. 1 as Pr-p S Fø (r (x =T
This equation means that, for a random sample (a, v) ~ P. must be at most the r-th quantile
of the predicted CDF (i.c., (r(z)) exactly with probability т. We can rewrite Definition 3.1 in another equivalent formulation: the following equation holds for any subinterval [r1, -2] C [0,
(Fxxx))>-(Fo(>>
E [2)) = 72 T1
This equation means that the quantile level Fo predicted for random sample (æt,) v) P is contained in a subinterval 72] S [0, 1] exactly with probability 72 Ti
On the basis of Definition 3.1 Goldstein et al. (2020) define calibration for survival analysis.
Definition 3.2 (Quantile-based calibration for survival analysis.) A prediction model Fo(t|x)) for survival analysis is quantile-calibral if this equation holds for any quantile level T E [0,
Pr(x, (FFx S =T
3.2 into another equivalent formulation: the following equation holds for any subinterval / -= [1,, 72
We can rewrite Definition
(0, 1):
E I) |1|
(3)
A problem when using Definition 3.2 is that cannot get samples directly from T due to the censoring. As such, we cannot verity Eq. (3) for datasets that include censored data. Goolsttin et al. (2020) resolved this problem by showing how estimate the probability Pr (t) E 7) for any E [c. oo from Pr(Fa(c(æ) E I) for a randomly sampled censored data point (æ, c, 0). Under the assumption that T and C' are independent (i.e., T 1 C X), they show
(r((F(((z)) E (n2 v) 1[0 E 4] (T2 1)
(4)
V where v = (elaz) and [- denotes the step function. By using this estimation, we can compute the left-hand side of Eq. from dataset D that include censored data points.
V
On the basis of Eq. and the approaches deseribed in (Andres et al. 2018 Haider et al. 2020),
Goldstein et al. (0200) proposed metric called distributional calibration (D-CAL), which is defined
as
(D. Σ [[xxxr..o--1[[o(]]])=)]]
IET
where the collection is chosen to contain disjoint contiguous subintervals of C C [0, 1] that cover the whole interval [0, 1].
A prediction model (t)x)) with lower CD-CAL is said to be more calibrated, but we cannot construct a neural network model that directly minimizes D-CAL because CD is dif- ferentiable function due to its step function. Therefore, Goldstein et al. (2000) defined the explicit calibration (X-CAL), an approximation of D-CAL, by replacing the step function with sigmoid function so that X-CAL becomes differentiable function. Moreover, X-CAL is designed to handle a set of data points = ((x,, Zi, b as a mini-batch, which makes it possible integrate X- CAL into the loss function of neural network models because most of those models use a mini-batch training rather than the full batch training. Formally, X-CAL defined as
RX-CAL = EB Σ
(Exx.T.C)_<((F(((T)) X); 171)
TET
where ((2; I, 7 a sigmoid function to approximate the step function D-CAL. (See (Goldstein et al. 2020) for the precise definition of the function Ç.) Here, we abuse notation (X. T, C) ~ B to indicate that we obtain sample data points from mini-batch B (rather than the probability distribution
Q). Note that Goldstein et al. 20200) proposed using X-CAL in the loss function of neural network model as a regularizer, which means that it is intended to be combined with other loss functions. This is because prediction model that aims only calibration is useless (as discussed in Section I1 and the balance between sharpness and calibration must be considered for obtaining a useful prediction model.
4

---

Under review as a conference paper at ICLR 2022
4 VALUE-BASED CALIBRATION
In this section, we propose alternative definitions of calibration, value-based calibration, for dis- tribution regression and survival analysis. Then, the basis of the new calibration definition for survival analysis, we propose our new Kaplan-Meier regularizer, and we discuss its advantages over D-CAL and X-CAL.
We first propose an alternative definition of calibration for distribution regression.
Definition 4.1 Value-based calibration.) A prediction model Fo(y)z) for distribution regression is value-calibrated if this equation holds for any value E [-00, ool:
Exx, X) Pr(xX S v).
This equation means that the average probability of prediction having a value of at most y must be equal to the actual ratio of data points having value of at most y. The difference between Definitions 3.1 4.1 is on the choice the axis. Whereas Definition 3.1] a natural condition for calibration with respect the 7-axis of prediction model Foly))) Definition 4.1 gives natural condition for calibration with respect to the y-axis. In appendix (Section A.Ip. I. we show that a crantile-calibatte model not necessarily a value-can callb-cad model and vice versa.
On the basis of Definition
we define the value-based calibration for survival analysis.
Definition 4.2 (Value-bused calibration for survival analysis.) A prediction model (t(az) for sur- vival analysis is value-calin calubcaaeratee if this equation holds for any time t E [0, tmax), where max is the maximun time of the uncensored data points in the dataset D:
E Bix.T,c))=Fo(((XX PI.(X.T, St).
(5)
Note that we ask Eq. (5) to hold for E [0, fmax] rather than E [0, 20). This is because the prediction model (t ( for survival analysis is usually trained for t [0, [max] and the prediction (a(t)z)) for >> bmax is less accurate.
We can rewrite Definition 4.2 into another equivalent form: the following equation holds for any subinterval [f1, ty] S [0, fmax
((F - (x)) = Pr(x,r,c)-q(tt tz).
This condition exactly matches the explanation of calibration in (Goldstein et al., 2020), i.e., model's predicted number of events within any time interval is similar the observed number. Note also that we can rewrite Eq. (5) into another equivalent form:
E EEX,TC))-S00 X) = P) t).
6
Here, the left-hand side of Eq. 6 changed from the CDF (t function (t and the right-hand side of the equation is also changed accordingly.
in Eq. 55) the survival
On the basis of Eq. (6). we propose our new metric, Kaplan-Meki loss, which is defined
Amaa
fKM (0) = (Exx.T.C)-Ss((11 Pr.x.T.C>~oTT > t))² dt.
In contrast to D-CAL, we can use this loss directly the loss function of a neural network model be- cause this loss differentiable. However, also propose new regularizer RKM(() (0) so that it can be used in a mini-batch training with a set of data points B = =[ zi, oz) d=)
(0) by modifying
(f) EB- L. Lemao
(Exx.T.O)) PI(X.T.C)=BTT > t)) 2))²dd dt.
Again, we abuse notation (X, T) ~ B indicate that we obtain sample data points from mini- batch B (rather than the probability distribution Q). Although used & loss in Eq. 7 1-8). use any other metric to measure the difference between two distributions. Note that we can compute the first term in the parentheses by
E EIX.T.O)
Σ
(1 - (-Fo(t)a:)),
B
(
5

---

Under review as a conference paper at ICLR 2022
S(t)
Kaplan-Meier survival curve
Predicted survival curve
taee
Time t
Figure Our Kaplan-Meier regularizer is designed to reduce the &g loss between the Kaplan-Meie survival curve and the average predicted survival curve.
Figure 4: Each box corresponds to a bin in and each circle corresponds to data point. The arrows show the directions of the moves of the data points during training.
t) by the Kaplan- Meier estimator. Recall that the Haalan-MMiirt estimator is designed estimate Pr(T t) even the dataset contains censored data. Figure B iillustrates the proposed loss (f) and regularizer RKM (f), in which the first term in the parentheses of Eq. ( corresponds to the average pre- dicted survival function (thick curve) and the second term corresponds to the Kaplan-Meier survival curve (red curve) and each equation computes the lg loss between these two curves.
and estimate the second term in the parentheses Pr(X,T,C)=@)
The advantages of our approach can be summarized as follows.
1. Our value-based calibration for survival analysis (Definition is intuitive for practi- tioners in healthcare. Practitioners are often interested in the prediction performance at a specific time t* E (m.x). and Eq. (5) shows natural condition that should hold at time t". In contrast, Eq. ( states nothing about specific time t..
2. Our value-based calibration for survival analysis (Definition 4.2 is defined only for the range [0, fmax]] Regarding the quantile-based calibration, Definition 3.1 means that, if a prediction model (e ax) for distribution regression is quantile-calibrated then the distri- bution (Y\ for (X, Y) P is equal the uniform distribution over [0, 1] ZZhao et al. 2020). However, even if a prediction model F% (t(t))) for survival analysis is quantile- calibrated, the distribution of (T X) usually not equal the uniform distribution when T' refers to the random variable for the survival time of uncensored events. This means that the condition in Definition 3.2 can be satisfied only we can estimate the dis- tribution of Fo X) from the dataset 7) that include censored data by using some method such as Eq. 44. other words, we need to estimate (t xx) for > fmax when we use
Definition 3.2
3. Our Kaplan-Meies loss and regularizer do not require any hyperparameter (other than the batch size [B), whereas D-CAL and X-CAL require the hyperparameters and C.
4. Our Kaplan-Meier regularizer uses a simple & loss and it avoids the binning approach used in X-CAL. In the following, explain the problem in the binning approach. In X-CAL, we count the number of data points in each bin and we update the parameters 0 of a neural network during the training so that the numbers of data points in the bins are balanced. Figure 4 illustrates that, for a bin in which the number of data points exceed the target, the parameters
updated so that the data points in the first half of the bin are pushed to the left and the data points in the second half of the bin are pushed to the right due to the approximation of the step function as the sigmoid function. However, since the numbers of data points in the three bins in the left exceed the target and the numbers of the data points in the two bins in the right are below the target in the case of Figure 4 no data point the three bins in the left should be pushed left and the data points in these bins should be used to fill the two bins in the right. Therefore, we sometimes find difficulties in minimizing X- CAL due to the binning approach. Note that Goldstein al. 2020) showed how to avoid this phenomenon for the left-most and right-most bins, but we cannot avoid this problem for the other bins.

---

Under review as a conference paper at ICLR 2022
444
Deyet
Koglin Maler
250
180
(a) flchain
(b) support
(c) retinopathy
Figure Comparison of average predicted survival functions by using loss functions DeepHit, DRSA, and Survival CRPS with Kaplan-Meies survival curve.
5 EXPERIMENTS
In this section, we compare the performance of our Kaplan-Meier regularizer with X-CAL on real datasets. We show that, by using our Kaplan-Mett Meier regularizer or X-CAL, we can reduce both Kaplan-Meier loss and D-CAL. This means that we can use one of these two regularizers both for quantile-based calibration and value-based calibration and do not need to combine the two regular- izers for two different definitions of calibration.
5.1 NEURAL NETWORK AND DATASETS
We constructed a single neural network for our experiments and combined with various loss func- tions. This neural network was two-layer perceptron with single hidden layer containing 128 neurons, and the number of outputs was 32. The activation function after the hidden layer was the ReLU type, and the activation function at the output node softmax. We used Python 3.7.4 and
PyTorch 1.4.0 for the implementation. The Adam optimizer (Kingma & Ba 2015) was utilized for the training algorithm, with the learning rate set 0.001, the batch size to 1024, and the other parameters to their default values. We run training for 100 epochs.
We used three datasets for survival analysis: two obtained from the packages in R (R Core Team 2016) and one from private data source. Of these former two, one is the flchain dataset (Dispenzieri al. 20122. which was obtained from the 'survival' package and contains 7874 data points (69.95 9% of which are censored), and the other is the support dataset (Knaus et al. 1995), which was obtained from the 'casebase' package and contains 9104 data points (31.9% which are censored). The dataset from the private data source is of patients with retinopathy disease. This dataset contains 6951 data points (33.3% of which are censored).
We split each of the three datasets into the training (80%) and test (20%) sets by using random partitioning. The neural network model was trained using the training set and the experimental results shown in this section (including the Kaplan-Meier survival curves) were obtained using the test set. When we computed the metrics (D-CAL and Kaplan-Meier loss) on test dataset, we used full batch (rather than the mini-batch). Regarding the parameters for D-CAL and X-CAL, used 10000 and the collection C was the set of 20 equally sized bins disjointed over [0, 1].
5.2 RESULTS
Average predicted survival functions without calibration. We compared the average survival functions predicted by the state-of- th- neural networks with the Kaplan-Meier survival curve on each dataset. In our experiments, we used the fixed neural network and the loss functions presented in DeepHit (Lee et al. 2018), DRSA (Ren et al. 2019), and Survival CRPS (Avati et al.) 2019). We used a = σ = 1.0 for DeepHit and a = 0.25 for DRSA as the hyperparameters. Figure shows the average survival functions predicted by using these three loss functions and the Kaplan-Meier survival curve on each dataset. The average of the predicted survival functions should be close to the Kaplan-Meiet survival curve if prediction model is calibrated. However, none of the predicted results was close to the Kaplan-Men Meiper survival curve, which means the loss functions do not achieve value-based calibration.
7

---

Under review as a conference paper at ICLR 2022
the 200
4.4
13
Supln Ualr
250
180
(a) flchain
(b) support
(c) retinopathy
Figure Average predicted survival curves with Kaplan- Meier regularizer with varying
(a) fichain
(b) support
(c) retinopathy
Figure 7: Average predicted survival curves with X-CAL regularizer with varying A.
Calibration with our Kaplan-Meier regularizer and X-CAL. We compared the performances of the proposed Kaplan-Meier regularizer and X-CAL by using each them as regularization term in conjunction with Survival CRPS (Avati et al. 2019). More specifically, used the following loss function for the Kaplan-Man Meier regularizer:
{(O) fs (s-CRrs(0) + XRKM(O),
where À is parameter, and we used the following loss function for X-CAL:
( = Es fs-ccss(0 + XRXx
where À is the parameter. Figures 6 and 7 show results by using these loss functions with various from (10, 103, 10 We can see that the average predicted survival curves are not close to Kaplan-Meier survival curve with small À and therefore the predictions were not calibrated. How- ever, regarding the average predicted survival curves with À = 10 the predicted curves with our Kaplan-Meier regularizer were close to the Kaplan-Meiee Meier survival curves and they calibrated enough. In contrast, the predicted curves with X-CAL were close to the Kaplan-Meier survival curves, but the curves predicted by using Kaplan-Meier regularizer were better than those with X-CAL.
Table the D-CAL and the Kaplan-Meiet loss of these predicted results. We using the Kaplan-Meier regularizer with large À leads to not only reducing the Kaplan-Meier loss but also reducing D-CAL. Similarly, using X-CAL with large À also leads to only reducing D- CAL but also reducing Kaplan-Meier loss. These facts demonstrate that we can use one of these two regularizers (the Kaplan-Meit Meier regularizer or X-CAL) to reduce the two metries (Kaplan-Meier loss and D- D-CAL) for the real datasets, although we theoretically show that a quantile-calibrate model is necessarily a calue-calibrated model and vice versa in the appendix (Section A.1).
here that
RELATED WORK
We use the Kaplan-Meier estimator (Kaplan Meier 1958 in our Kaplan-Meier loss estimate the survival rate S(t) = Pr fi.xT,Cc (Txxx,C))<(( (T t). Our loss function is not restricted to the Kaplan- Meier estimator and we can use any other nonparametric method to estimate the survival rate S()).
For example, use the Meleon-Aalen estimator (Aalen 1978, Nelson 1969 1972) instead.
8

---

Under review as a conference paper at ICLR 2022
Table D-CAL and Kaplan-Meier loss for the various combinations of dataset, regularizer, and X
Kaplan-Meier regularizer (proposed)
10
A
10s
À 10a
D-CAL Kaplan-Meier loss
0.1181
0.0587 0.0705
0.0023
0.1110
0.0004
fichain
X-CAL (Goldstein et al. 2020)
À = 10
== 10
À = 10
D-CAL
0.1176 0.1110
0.0588 0.0730
0.0005
Kaplan-Meier loss
0.0049
Kaplan-Meier regularizer (proposed)
10
x= 10
À = 10a
D-CAL Kaplan-Meier loss
0.2948 0.0914
0.0938 0.0167
0.0089
0.0009
support
X-CAL (Goldstein et al.
2020
A = 10
X = 10
X = 10a
D-CAL Kaplan-Meier loss
0.0839 0.0889
0.0186 0.0426
0.0008
0.0030
Kaplan-Meier regularizer (proposed)
10
x= 103
10a
D-CAL Kaplan-Meier loss
0.0800
0.0505
0.0010
0.1731
0.0074
0.0004
retinopathy
X-CAL (Goldstein al. 2020)
= 10
103
10
D-CAL Kaplan-Meier loss
0.0783
0.0218
0.0003
0.1733
0.0524
0.0005
Regarding another calibration for survival analysis, there was study on I-Calibration (Haider al., 2020). In 1- I-Calibration, we consider the calibration of single point of the predicted distribution, but the entire distribution not evaluated for calibration. We also note that the distributional divergence for calibration (DDC) Kamran & Wiens (202T) is similar to D-CAL. The difference between DDC and D-CAL are that D-CAL uses a binning approach and DDC uses the Kullback Leibler divergence to measure the distance between the predicted survival function and the uniform distribution.
7 CONCLUSION
In this work, we presented new definitions of calibration, value-based calibration, for distribution regression and survival analysis. On the basis of the new calibration definition for survival analysis, then proposed
metric called Kaplan-Meier loss for value-based calibration. The results of experiments showed that it can be used as the regularizer of a loss function in a neural network model for calibration, and it can be seen simpler alternative X-CAL.

---

Under review as a conference paper at ICLR 2022
8 REPRODUCIBILITY STATEMENT
We summarize our efforts for reproducibility. The details of our Kaplan-Meier loss and regularizer described in Section 4 In addition, describe the algorithm of the Kaplan-Meier estimator in Section [ to make our paper self-contained, although the Kaplan-Meier estimator is a famous algorithm survival analysis. Regarding our experiments. we described all of the details about our Python environment and the parameters used in Section [ We used three datasets, and two of them are publicly available datasets. Although we do not attach our source code, we believe that readers of our paper can easily reproduce our results, because our algorithm is easy to implement and influence of the undescribed factors (e.g., choice the random seed) can be ignored (e.g., the conclusion of our paper does not change even if the numbers in Table I fluctuate by 20%)
REFERENCES
Odd Aalen. Nonparametric inference for family of counting processes. The Annals of Statistics, 6 (4):701-726, 1978.
Axel Andres, Aldo Montano-Loza, Russell Greiner, Max Uhlich, Ping Jin, Bret Hoehn, David Bigam, James Andrew Mark Shapiro, and Norman Mark Kneteman. A novel learning algorithm to predict individual survival after liver transplantation for primary sclerosing cholangitis. PLoS One, (3(3)=e0133523, 2018.
Anand Avati, Tony Duan, Sharon Zhou, Kenneth Jung, Nigam H. Shah, and Andrew Y. Ng. Count- down regression: Sharp and calibrated survival predictions. In Proceedings of UAI 2019, pp- 145-155, 2019.
Peng Cui, Wenbo Hu, and Jun Zhu. Calibrated reliable regression using maximum mean discrep- ancy. In Proceedings of NeurIPS 2020, 2020.
Lore Dirick, Gerda Claeskens, and Bart Baesens. Time to default in credit scoring using survival analysis: benchmark study. Journal of the Operational Research Society, 68(6):652-665, 2017.
Angela Dispenzieri, Jerry A. Katzmann, Robert A. Kyle, Dirk R. Larson, Terry M. Therneau, Colin Colby, Raynell Clark, Graham P. Mead, Shaji Kumar, L. Joseph Melton III, and S. Vincent Rajkumar. Use of nonclonal serum immunoglobulin free light chains to predict overall survival in the general population. Mayo Clinic Proceedings, 87(6):517-523, 2012.
Mark Goldstein, Xintian Han, Aahlad Manas Puli, Adler Perotte, and Rajesh Ranganath. X-CAL: Explicit calibration for survival analysis. In Proceedings of NeurlPS 2020, 2020.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. In Proceedings of ICML 2017, pp. 1321-1330, 2017.
Humza Haider, Bret Hochn, Sarah Davis, and Russell Greiner. Effective ways to build and evaluate individual survival distributions. Journal of Machine Learning Research, 21(85):1-63, 2020.
Fahad Kamran and Jenna Wiens. Estimating calibrated individualized survival curves with deep learning. In AAAI 2021, 2021.
Edward L. Kaplan and Paul Meier. Nonparametric estimation from incomplete observations. Journal of the American Statistical Association, 53(282):457-44 481, 1958.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Proceedings of ICLR 2015, 2015.
W. A. Knaus, F. E. Harrell Jr., Lynn, L. Goldman, R. S. Phillips, A. Connors Jr., N. V. Dawson, W. J. Fulkerson Jr., R. M. Califf, N. Desbiens, P. Layde. R. K. Oye, P. E. Bellamy, R. B. Hakim, and D. P. Wagner. The SUPPORT prognostic model. objective estimates of survival for seriously ill hospitalized adults. study understand prognoses and preferences for outcomes and risks of treatments. Annals of Internal Medicine, 122(3):191-200 1995.
Volodymyr Kuleshov, Nathan Fenner, and Stefano Ermon. Accurate uncertainties for deep learning using calibrated regression. In Proceedings of ICMI. 2018, pp. 2796-2804, 2018
10

---

Under review as a conference paper at ICLR 2022
Changhee Lee, William R. Zame, Jinsung Yoon, and Mihaela der Schaar. Deephit: A deep learning approach to survival analysis with competing risks. Proceedings of AAAI-18, pp- 2314-2321, 2018. 2018.
Wayne Nelson. Hazard plotting for incomplete failure data. Journal of Quality Technology, 1:27-52, 1969.
Wayne Nelson. Theory and applications of hazard plotting for censored failure data. Technometrics, 14(4):945 966, 1972.
R Core Team. R: A Language and Environment for Statistical Computing. R Foundation for Statis- tical Computing, Vienna, Austria, 2016. Utthttts: /tt:://wwwww.o pro.epprooee..o.. org/
Kan Ren, Jiarui Qin, Lei Zheng, Zhengyu Yang, Weinan Zhang, Lin Qiu, and Yong Yu. Deep recurrent survival analysis. In Proceedings of AAAI-19, pp. 4798 4805, 2019.
Hao Song, Tom Diethe, Meelis Kull, and Peter Flach. Distribution calibration for regression. In Proceedings of ICML 2019, pp. 5897-5906, 2019.
Donna Tjandra, Yifei He, and Jenna Wiens. hierarchical approach to multi-event survival analysis. In Proceedings of AAAI 2021, pp- 591-599, 2021.
Ping Wang, Yan Li, and Chandan Reddy. Machine learning for survival analysis: survey. ACM Computing Surveys, 5 51(6):1-36. 2019.
Shengjia Zhao, Tengyu Ma, and Stefano Ermon. Individual calibration with randomized forecasting. In Proceedings of ICML 2020, pp. 11387-11397, 2020.
Panpan Zheng, Shuhan Yuan, and Xintao Wu. Safe: neural survival analysis model for fraud early detection. In Proceedings of AAAI-19, pp. 1278-1285, 2019.
A APPENDIX
A.
RELATIONSHIIP BETWEEN TWO DEFINITIONS OF CALIBRATION
We show that Definition 3.1 and Definition 4.1 are not equivalent. To see this, suppose that we have a probability distribution P on X' x )r where X' = y = [0, 1]. such that the corresponding random Y.
variables and Y are the uniform random variables on [0, 1] that satisfy
A quantile-cllibrated model is not value-calibrated model. Fly[z) defined by
We consider a prediction model
0
(x 0),
F Fzy(z)
y
(0 < x 1),
(x 1).
Since the distribution F the uniform distribution [0, 1). this prediction model F(ylæ) is quantile-calibraeed by Definition 3.1 However, since we have
E. Ex F(y X) = 0.5
and
Pr(XY) Y S y) = y for any y E 10,,),, the prediction model F(y(æ) does not satisfy the condition of calibration in Definition 4.1
value-calibrated model is not quantile-clibrrted model. We consider a prediction model F(y[z) defined by
F(y(z) = 0 (x y)
(x < v)
Then this prediction model F(ylæ) is value-calibrated by Definition 4.1
because we have
Bxx,y)>PF))XX)
Pr(x,r)>( Y S y)
for any y E [0, 1]. However, since F (yl(z) = for any sample (z, y) ~ P, the prediction model
(yylz)) does satisfy the condition of calibration in Definition 3.1


Reviews:
[{"review_id": "dJH8WYepI7B", "reviewer": "Reviewer_xdbC", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Title:\n    > \"simpler\" compared to what? Also, what does \"simple\" mean in this context? \"easier to optimize\"?\n\n\nAbstract:\n    > In your abstract and even at the beginning of the introduction, it is not obvious what you are trying to estimate. Survival function? hazard? What is your neural network trying to predict? \n    > You put much more emphasis on X-CAL rather than your own work. I believe it would have been better to talk about your work in this paper.\n\n\nSection 1:\n    > first paragraph: you talk about survival analysis, then talk about application in healthcare, then again you talk about survival analysis and after than about its application in healthcare and other fields. The flow fluctuates. Also, you just talk about the existence of a survey paper without mentioning what to take away from it.\n    > third paragraph: The connection between KM and your work at this point is not properly explained given you are delving into it. After that, you only discuss other models that are patient-specific. But the connection to the main paper is still not clear. Your work, as far as I understood, is not making KM patient specific, it is about using it as a regularizer. \n\n\nSection 1.1:\n    > first paragraph: please define what you mean by confidence. \n\n\nSection 1.2:\n    > same as the title, 'simpler' in what sense? \n    > \"an advantage ... is calibrated for any time y \\n [0, t_{max}].\" why is this an advantage?\n\n\nSection 2:\n    > at the end, also a branch of work has been dedicated to hazard estimation rather than survival function estimation.\n\n\nSection 3 and 4:\n    > prior work seems to be mixed up with your work. It would have been more organized to first talk about prior work (calibration definition and KM and things like that) and then move to your own work. This way they can be mixed up. Or talk about your work first and then the connection to the prior work. \n\n\nSection 3:\n    > after equation 1, maybe worth mentioning this is due to the CDF function being nondecreasing and if invertible, the inequality does not flip\n    > last paragraph: explaining the work already done in Goldstein et al. it should be more aligned towards your work and how it is different from the prior work, not explaining somebody else's work. \n\n\nSection 4:\n    > several definitions of calibration make it confusing which one is the one you are considering. Same goes for Section 3 vs Section 4. \n    > at equation 8 and still not sure what your model is. Is your neural network predicting the parameters of a known distribution? It is not obvious to me how you compute equation 8 given the model is not known but you talk about mini-batch training which means your model should be defined concretely already.\n    > The advantages of the approach: advantage 1: why should your regularization term or even the calibration be 'intuitive'? either your definition is well-calibrated, ready to be used, or not.\n\n\nSection 5:\n    > In the only paragraph for this section, it seems like the calibration definition is relative and there is no better way of defining it.\n    > How about discrimination results? You are regularizing the model based on KM which is not input dependent. I am curious to see how it affects the model discrimination or 'sharpness' as mentioned in the paper.\n\n\nSection 5.2:\n    > You are including a KM based regularization in your model so the significance of average survival curve being similar to the KM curve needs to be explained. Also in the last paragraph, what is the take-away? Which of the two regularizers prove to be more useful?", "summary_of_the_paper": "This paper first defines a new definition of calibration which, in contrast to the one used in prior work, is confined to the maximum observed time in the data. They then propose a KM regularizer for making sure their survival curves are calibrated: they are closer to KM curves. They also claim that their regularization, unlike prior work, does not require any hyperparameters and avoids the binning approach.", "summary_of_the_review": "The paper needs organization and the significance needs to be more clear. Also, I am skeptical about how much better it performs over the baseline given the authors only give results on their defined calibration rather than how discriminative the model remains. When optimizing for curves similar to KM, it is expected to obtain them as such. How would the other metrics change?", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "ZxTiLKVcWMj", "reviewer": "Reviewer_sM7d", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper explores calibration in survival analysis, which is an important yet relatively under-explored problem. However, I found several issues with this submission. Here is a summary: \n\n*The paper overlooks several works in survival analysis including an important baseline*\n- A Kaplan-Meier calibration objective for survival analysis has been proposed before [1]. Therefore the paper (including experimental results) should be positioned relative to [1]\n- Prominent survival analysis literature before neural networks have been completely ignored, *e.g.*, cox proportional hazards [2], accelerated failure time times [3], *etc.*\n- Several works (including neural network-based) that focus on hazards (instantaneous risk) predictions have been ignored, including [2, 4].\n- While the paper references S-CRPS, the paper overlooks that the S-CRPS objective focuses on sharp and calibrated predictions. A discussion connecting their proposed objective to the S-CRPS thresholded calibration with Brier scores is important since Brier scores are prominently used as a calibration metric in survival analysis.\n\n*Weak experimental evaluations*\n- The paper's focus on only calibration metrics is concerning since calibration alone does not yield accurate predictions. Other accuracy-based metrics such as concordance index and relative absolute error should be included. This is important since calibration and accuracy are orthogonal concerns.\n- As expected, it seems the Kaplan-Meier regulariser and X-CAL obtain perfect calibration at very large $\\lambda$. At such a large $\\lambda$, the model survival predictions will be inaccurate. Therefore, the paper should provide sensitivity analysis on how accuracy is impacted at varying $\\lambda$.\n- Why is the empirical Kaplan-Meier compared to estimated average survival functions, instead of an estimated Kaplan-Meier from predictions, since the datasets are right-censored?\n- Since X-CAL has a slight competitive advantage. Why should users adopt the proposed Kaplan-Meier regulariser?\n\n*Misleading statement*\n-  Abstract: \"While there have been many neural network models proposed for survival analysis, none of them are calibrated.\" This is a very strong and misleading statement. Depending on the calibration objective, some methods are calibrated, *e.g.*, S-CRPS, SFM in [1], *etc.*\n\n*Redundant content*\n- In general, writing can be improved by minimizing the discussions focused on *regression* as they may serve as a distraction since survival analysis is completely different from regression.\n\n*Minor Issues*\n- There are several typos, I encourage the authors to proofread carefully\n\n**Missing references**\n- [1] Chapfuwa et al., \"Calibration and Uncertainty in Neural Time-to-Event Modeling\", IEEE Transactions on Neural Networks and Learning Systems, 2020.\n- [2] Cox, “Regression models and life-tables,”  Breakthroughs in Statistics, 1992.\n- [3]  Wei, “The accelerated failure time model: A useful alternative to the cox regression model in survival analysis,” Statistics in medicine, 1992\n- [4] Katzman et al., \"Deep survival: A deep cox proportional hazards network\", BMC Medical Research Methodology, 2018", "summary_of_the_paper": "The paper proposes a Kaplan-Meier calibration regulariser for training survival (right-censored) datasets for calibrated predictions. Experimental results of the proposed approach against previously proposed  X-CAL regulariser on three datasets per metrics: D-CAL and Kaplan-Meier loss indicate that X-CAL has a competitive advantage.", "summary_of_the_review": "The paper overlooks several prominent works in survival analysis, including a baseline closely related to their proposed Kaplan-Meier regulariser. Additionally, the paper's focus on calibration alone in the experimental results is problematic since calibration and accuracy are equally important but orthogonal metrics.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "5S5VG5x78C5", "reviewer": "Reviewer_KZxz", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I found this paper enjoyable to read and the proposed method to be extremely intuitive and well-presented. Essentially the regularizer proposed is taking advantage of a consistency check, i.e., a reconstructed version of KM based on a user-specified survival model should match up with the classic KM estimator. The proposed KM regularizer is much simpler to describe than X-CAL and one doesn't have to deal with the clunky binning or censorship handling issues in the case of X-CAL. As an added benefit (one that perhaps is under-emphasized in the paper), the KM estimator is theoretically very well-understood with known rates of weak & strong convergence (e.g., Foldes & Rejto 1981a/b, Gill 1983, Chen & Lo 1997), known confidence interval construction (e.g., standard Greenwood formula). Of course, KM is also widely used/well-understood by practitioners.\n\nI think this paper's main weakness is in the thoroughness of the experiments:\n- It would be helpful to provide insights about various standard survival models that are used (beyond just picking a few SOTA approaches). For example, with the addition of the KM regularizer to different models, do we get some insight on which models tend to be better calibrated? Here, I think trying more models would be helpful. I realize that some thought would need to go into how to make the comparison in some sense fair across the models as different models' loss functions could have scales that are a bit different, so the same value of $\\lambda$ across models doesn't correspond to the same regularization amount across models. Basically, the main question is whether some models just really struggle to be calibrated and one must use a very large amount of KM regularization, vs whether some models are just better-calibrated to begin with and very little if any KM regularization is needed.\n- I think it would be helpful providing a baseline comparison against, say, random survival forests where the estimators at the leaves are KM estimators (rather than Nelson-Aalen estimators as in the original RSF paper), where you don't try to regularize RSF but you just show how far it is from KM. Similarly, seeing how close the standard Cox model's averaged individual survival curves compares with KM could also be helpful as a baseline comparison.\n- If the survival model that we use is just the KM estimator so that we ignore input feature vectors entirely, then it would be perfectly calibrated according to the KM loss (if I'm understanding things correctly). Of course, this is a bit unsatisfying as the KM estimator does not provide individual-level information. By increasing the amount of KM regularization, we're making individual survival curves closer to the population KM average. This could mean that individual-level predictions are no longer as good. It would be helpful perhaps showing the tradeoff between a standard accuracy metric (e.g., time-dependent concordance or time-dependent AUC) vs the amount of KM/X-CAL regularization?\n- I'd prefer having some sort of experimental repeats to get error bars of sorts on the experimental results (e.g., Figures 5-7, Table 1).\n\nReferences:\n- Kani Chen, Shaw-Hwa Lo. On the rate of uniform convergence of the product-limit estimator: strong and weak laws. The Annals of Statistics 1997.\n- Antonia Foldes, Lidia Rejto. Strong uniform consistency for nonparametric survival curve estimators from randomly censored data. The Annals of Statistics 1981a.\n- Antonia Foldes, Lidia Rejto. A LIL type result for the product limit estimator. Zeitschrift für Wahrscheinlichkeitstheorie und Verwandte Gebiete 1981b.\n- Richard Gill. Large Sample Behaviour of the Product-Limit Estimator on the Whole Line. The Annals of Statistics 1983.", "summary_of_the_paper": "This paper proposes a new calibration loss motivated by a simple observation: the average of individual estimated survival curves should be close to the marginal nonparametric estimate given by the Kaplan-Meier estimator. This calibration loss readily supports mini-batching, introduces no new hyperparameters during the optimization, and is shown to also be effective at encouraging an existing calibration metric D-CAL to be small.", "summary_of_the_review": "The proposed KM regularizer looks very promising and is quite intuitive. While the experiments presented so far are instructive, they could be substantially more thorough to help us understand the pros vs cons of using more KM regularization, and also what we can say about how well-calibrated different standard survival models are.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "4QsV_WmMQOt", "reviewer": "Reviewer_MTaz", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "=== Strengths\n\nOne of the main strengths of this work is the novelty, yet relative simplicity, of the proposed approach. Defining calibration of the learned survival function with respect to the actual Kaplan-Meier (or similar) curve is reasonable. I am not an expert in this area, but to the best of my knowledge, it is also novel.\n\nI did not thoroughly verify the propositions and assertions of probability, though they seem to follow from definitions. Thus, the derivation of the proposed regularization terms appears sound.\n\n=== Weaknesses\n\nThe empirical evaluation comparing the proposed and existing approaches is rather limited. First, the dataset characteristics which result in performance differences of the two approaches is not explored at all. For example, on the “flchain” dataset, the proposed approach results in an order of magnitude improvement for the KM-loss compared to X-CAL. On the other hand, for “retinopathy,” there is hardly any difference. It would be helpful to understand the characteristics of these datasets (such as amount of censoring, spread of survival times, types or amount of features, etc.) which lead to different relative performance. If needed, synthetic datasets could be created to control various factors of the data, such as the rate of censoring. Second, it would be very useful to also report some measures of prediction accuracy; as the authors highlight, both calibration and sharpness are important. Third, cross validation or some other approaches should be used to provide some measure of statistical significance for the work.\n\nThe clarity of the paper could be improved. While the overall structure and flow of the paper is reasonable, Sections 3 and 4 are quite dense. As a reader less familiar with this area, I would have prefered to avoid the various reformulations in the middle of the text; rather, I believe starting with and sticking with the forms (possibly refering to the equivalences given in Section 2) finally used in the regularizers would have been easier to follow.\n\nThe reproducibility and value of resources provided by the authors is clearly limited. The proposed approach itself seems straightforward enough to implement, as the authors say; however, considering the numerous steps in the method, the lack of reproducibility diminishes the impact of the work.\n\n=== Typos, etc.\n\nThe font size in Figures 5, 6, and 7 should be increased.\n\n“for the real datasets” -> “for real datasets”\n\nIn the [Kamran and Wiens] reference: “In AAAI” -> “In Proceedings of AAAI”", "summary_of_the_paper": "In this work, the authors propose a novel approach for learning calibrated predictions for survival analysis and similar tasks. The intuition of the approach is that the average probability that a prediction has a value less than or equal to $t$ should approximately equal the number of observations with value less than or equal to $t$. The authors then derive a differentiable regularization term using this idea and Kaplan-Meier curves. A small set of empirical evaluations suggest the proposed approach modestly outperforms another recent survival analysis calibration method according to some metrics.", "summary_of_the_review": "Time-to-event predictions accounting for censored data is a wide-spread problem. Considering the relative simplicity, such as lack of hyperparameters, of the proposed approach, I believe it has the potential to have modest impact in several applied areas. On the other hand, the empirical support for the work is rather limited, so more evidence is likely needed before wider adoption and impact.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: A number of suggestions have been given about the manuscript. The evaluation raised questions about clarify, placement with respect to other approaches, choices for the design, etc. There are no immediate replies from authors, so I hope the suggestions are useful for future work.
Decision: Reject