Year: 2021

Abstract:
Survival analysis, also known as time-to-event analysis, is the problem to predict the distribution of the time of the occurrence of an event. This problem has applications in various fields such as healthcare, security, and finance. While there have been many neural network models proposed for survival analysis, none of them are calibrated. This means that the average of the predicted distribution is different from the actual distribution in the dataset. Therefore, X-CAL has recently been proposed for the calibration, which is supposed to be used as a regularization term in the loss function of a neural network. X-CAL is formulated on the basis of the widely used definition of calibration for distribution regression. In this work, we propose new calibration definitions for distribution regression and survival analysis, and demonstrate a simpler alternative to X-CAL based on the new calibration definition for survival analysis.

heading: 1 INTRODUCTION
text: Survival analysis, also known as time-to-event analysis, is the problem to predict the time of the occurrence of an event. In healthcare applications, the event typically corresponds to a death or the onset of disease in a patient. The time between a well-defined starting point and the occurrence of the event is called the survival time or failure time. In survival analysis, we usually estimate the distribution of the survival times of patients. Survival analysis has important applications in healthcare as well as various other fields (e.g., credit scoring (Dirick et al., 2017) and fraud detection (Zheng et al., 2019)). The recent progress of prediction models for survival analysis has been summarized in a survey paper (Wang et al., 2019).
In survival analysis, datasets are often censored, which means that events of interest might not be observed for some instances. This may be due to either the limited observation time window or missing traces caused by other irrelevant events. Typical censored data are right censored data. These are the data points whose exact times of the events are unknown; we know only that the events had not happened up to a certain time. In this paper, we focus on the uncensored data and the right censored data, as shown in Figure 1. Here, the event for data point x1 is observed during the period of study and hence this data is categorized as uncensored data. The data points x2 and x3 are categorized as right censored data because we did not observe the events during the period of study. The time between a well-defined starting point and the last observation time (e.g., the time of the end of study) is called the censoring time.
One of the classical methods to solve the survival analysis problem is the Kaplan-Meier estimator (Kaplan & Meier, 1958). This is a non-parametric method to estimate the distribution of the survival times as a survival function S(t), where the value S(t∗) for a specific time t∗ represents the survival rate at time t∗ (i.e., the ratio of the patients who survived at time t∗). It is easy to estimate the survival function S(t) if the dataset contains only uncensored data points, but the Kaplan-Meier estimator is designed to work for datasets that include censored data. Here, we briefly explain the algorithm of the Kaplan-Meier estimator. Let {ti}ki=1 be the set of distinct times when at least one uncensored event was observed in the dataset. Let di be the number of (uncensored) events that happened exactly at time ti, and let ni be the number of data points that are known to have survived at time ti. Then, the Kaplan-Meier estimator outputs the survival function
S(t) = ∏ i:ti≤t ( 1− di ni ) .
Figure 2 shows an example of the survival function S(t) estimated by the Kaplan-Meier estimator for the flchain dataset (Dispenzieri et al., 2012). Here, we can see that the survival rate at time t = 2500 is approximately 80%. Note that the true survival rate S(t) at t =∞ must be zero, but the Kaplan-Meier estimator outputs the survival function S(t) only for time t ∈ [0, tmax], where tmax is the maximum survival time of the uncensored data points in a dataset. This is because we cannot estimate the survival rate S(t) for the time t > tmax.
A drawback of the Kaplan-Meier estimator is that it outputs a survival function S(t) for the entire population and not for a specific patient. Therefore, there have been many algorithms to estimate the survival rate S(t|x) for each patient x so as to enable personalized medicine (Wang et al., 2019). In particular, many neural network models that predict the survival function S(t|x) have been proposed (Lee et al., 2018; Ren et al., 2019; Zheng et al., 2019; Tjandra et al., 2021).

heading: 1.1 CALIBRATION
text: When we use a prediction model, it should be calibrated. In binary classification, this means that a prediction model that outputs a confidence of a true label for an input is expected to satisfy the condition that the average of the confidence values over all inputs are equal to the ratio of the data points with a true label in the dataset. For example, if a dataset contains 40% of the data points with a true label, then we expect that a calibrated prediction model will output a confidence of 0.4 on average. Even though calibration is important for prediction models, and neural network models have been widely used for prediction models, (Guo et al., 2017) showed that neural network models are often miscalibrated.
In regression analysis, quantile-based calibration is widely used as the definition of calibration (Kuleshov et al., 2018; Song et al., 2019; Cui et al., 2020; Zhao et al., 2020). In survival analysis, the definition of calibration is based on this definition for regression analysis. Goldstein et al. (2020) showed a method to train a neural network model to achieve calibration by adding a new regularizer, X-CAL, to the loss function of the neural network to achieve calibration during the training. This method is in contrast to the widely-used calibration methods for regression analysis such as Platt scaling and isotonic regression, which are post-training methods.
Note that a calibrated model is not necessarily useful. For example, in binary classification we can construct a trivial calibrated model that always outputs the ratio of true label data points in a dataset as a confidence value. Therefore, a useful prediction model also needs to be sharp, which means that it outputs a confidence value close to one or zero for each input. To obtain a prediction model that is both calibrated and sharp, one approach is to train a neural network with a loss function consisting of one loss term aimed at sharp prediction and another aimed at calibration. X-CAL utilizes this approach in that it is a regularizer for calibration and is used in combination with another loss term that aims at sharp prediction.

heading: 1.2 OUR CONTRIBUTIONS
text: In this paper, we propose a Kaplan-Meier regularizer as a simpler alternative to X-CAL. An advantage of our regularizer is that the obtained prediction model is calibrated for any time t ∈ [0, tmax]. Another advantage is that our regularizer does not require any hyperparameter, whereas X-CAL requires hyperparameters. The idea behind our new regularizer is simple: just to reduce the difference between the average predicted survival function and the Kaplan-Meier survival function (see Figure 3). Our new regularizer is based on a new definition of calibration for survival analysis, and we discuss its advantages in Section 4.

heading: 2 SURVIVAL ANALYSIS
text: We formally describe the problem settings of regression analysis and survival analysis. In regression analysis, we assume that there is an unknown probability distribution P on the sample space X ×Y , where X is the domain of features and Y is the interval [−∞,∞]. We refer to the associated random variables with capital letters (i.e., X and Y ) and realizations with lower case letters (x, y) ∼ P . We assume that we can obtain independent samples {(xi, yi)}ni=1 from X ×Y according to distribution P . In survival analysis, we assume that there is an unknown probability distribution Q on the sample space X × T × C, where X is the domain of features and T and C are time intervals [0,∞]. We refer to the associated random variables with capital letters (i.e., X , T , and C) and realizations with lower case letters (x, t, c) ∼ Q. The random variable T corresponds to the survival time (i.e., the time of the occurrence of an event), which might not be observable due to the censoring, and the random variable C corresponds to the censoring time. We assume that the censoring time is∞ for an uncensored event. The random variable defined by Z = min{T ,C} corresponds to the time of the last observation (i.e., the survival time or the censoring time). Different from the regression analysis, we cannot obtain samples {(xi, ti, ci)}ni=1 from X ×T ×C according to distributionQ due to the censoring in survival analysis. However, we assume that we can obtain independent samples D = {(xi, zi, δi)}ni=1 instead, where δi is a binary value indicating if the i-th data point is censored or not. Here an uncensored data point (xi, zi, δi) ∈ D satisfies δi = 1 and zi = ti and a censored data point (xi, zi, δi) ∈ D satisfies δi = 0 and zi = ci. Hence δi = 0 means that we know only the fact that ti ≥ ci and the exact survival time ti is unknown. The task of survival analysis is to predict the probability of an event of interest occurring at time t for x ∈ X as a probability distribution function f(t|x). This function f(t|x) is often represented in other equivalent forms. For example, it can be represented as its cumulative distribution function (CDF)
F (t|x) = ∫ t 0 f(τ |x)dτ
or as the survival function S(t|x) defined by S(t|x) = 1− F (t|x).
Intuitively, F (t|x) represents the probability of observing the event by time t for x and the survival function S(t|x) represents the probability of not-observing the event until time t for x. Many neural network models have been proposed for survival analysis (e.g., (Lee et al., 2018; Ren et al., 2019; Zheng et al., 2019; Tjandra et al., 2021)). They output the probability distribution in the form of fθ(t|x), Fθ(t|x), Sθ(T |x), or something equivalent to these forms, where θ is the parameters of the neural network.

heading: 3 QUANTILE-BASED CALIBRATION
text: In this section, we review the definitions of calibration for distribution regression and survival analysis in the literature. First, we consider the distribution regression whose task is to predict the distribution of the target variable as a CDF Fθ(y|x) for x ∈ X , where θ is the parameters of the prediction model. In distribution regression, quantile-based calibration (Kuleshov et al., 2018) is widely used as the definition of calibration (e.g., (Song et al., 2019; Cui et al., 2020; Zhao et al., 2020)).
Definition 3.1 (Quantile-based calibration.) A prediction modelFθ(y|x) for distribution regression is quantile-calibrated if this equation holds for any quantile level τ ∈ [0, 1]:
Pr(X,Y )∼P(Fθ(Y |X) ≤ τ) = τ. (1)
If we can compute the inverse of Fθ(y|x), we can rewrite Eq. (1) as Pr(X,Y )∼P(Y ≤ F−1θ (τ |X)) = τ. This equation means that, for a random sample (x, y) ∼ P , y must be at most the τ -th quantile of the predicted CDF (i.e., F−1θ (τ |x)) exactly with probability τ . We can rewrite Definition 3.1 in another equivalent formulation: the following equation holds for any subinterval [τ1, τ2] ⊆ [0, 1]: Pr(X,Y )∼P(Fθ(Y |X) ∈ [τ1, τ2]) = τ2 − τ1. This equation means that the quantile level Fθ(y|x) predicted for a random sample (x, y) ∼ P is contained in a subinterval [τ1, τ2] ⊆ [0, 1] exactly with probability τ2 − τ1. On the basis of Definition 3.1, Goldstein et al. (2020) define calibration for survival analysis.
Definition 3.2 (Quantile-based calibration for survival analysis.) A prediction model Fθ(t|x) for survival analysis is quantile-calibrated if this equation holds for any quantile level τ ∈ [0, 1]:
Pr(X,T ,C)∼Q(Fθ(T |X) ≤ τ) = τ. (2)
We can rewrite Definition 3.2 into another equivalent formulation: the following equation holds for any subinterval I = [τ1, τ2] ⊆ [0, 1]: Pr(X,T ,C)∼Q(Fθ(T |X) ∈ I) = |I| = τ2 − τ1. (3) A problem when using Definition 3.2 is that we cannot get samples directly from T due to the censoring. As such, we cannot verify Eq. (3) for datasets that include censored data. Goldstein et al. (2020) resolved this problem by showing how to estimate the probability Pr(Fθ(t|x) ∈ I) for any t ∈ [c,∞] from Pr(Fθ(c|x) ∈ I) for a randomly sampled censored data point (x, c, 0). Under the assumption that T and C are independent (i.e., T ⊥ C |X), they show
Pr(Fθ(t|x) ∈ I) = (τ2 − v)1[v ∈ I]
1− v + (τ2 − τ1)1[v < τ1] 1− v , (4)
where v = Fθ(c|x) and 1[·] denotes the step function. By using this estimation, we can compute the left-hand side of Eq. (3) from dataset D that include censored data points.
On the basis of Eq. (3) and the approaches described in (Andres et al., 2018; Haider et al., 2020), Goldstein et al. (2020) proposed a metric called distributional calibration (D-CAL), which is defined as
`D−CAL(θ) = ∑ I∈I ( E(X,T ,C)∼Q1[Fθ(T |X) ∈ I]− |I| )2 ,
where the collection I is chosen to contain disjoint contiguous subintervals of C ⊆ [0, 1] that cover the whole interval [0, 1].
A prediction model Fθ(t|x) with a lower `D−CAL(θ) is said to be more calibrated, but we cannot construct a neural network model that directly minimizes D-CAL because `D−CAL(θ) is not a differentiable function due to its step function. Therefore, Goldstein et al. (2020) defined the explicit calibration (X-CAL), an approximation of D-CAL, by replacing the step function with a sigmoid function so that X-CAL becomes a differentiable function. Moreover, X-CAL is designed to handle a set of data points B = {(xi, zi, δi)}bi=1 as a mini-batch, which makes it possible to integrate XCAL into the loss function of neural network models because most of those models use a mini-batch training rather than the full batch training. Formally, X-CAL is defined as
RX−CAL(θ) = EB∼Q ∑ I∈I ( E(X,T ,C)∼Bζ(Fθ(T |X); I, γ)− |I| )2 ,
where ζ(z; I, γ) is a sigmoid function to approximate the step function in D-CAL. (See (Goldstein et al., 2020) for the precise definition of the function ζ.) Here, we abuse notation (X,T ,C) ∼ B to indicate that we obtain sample data points from mini-batchB (rather than the probability distribution Q). Note that Goldstein et al. (2020) proposed using X-CAL in the loss function of a neural network model as a regularizer, which means that it is intended to be combined with other loss functions. This is because a prediction model that aims only at calibration is useless (as discussed in Section 1) and the balance between sharpness and calibration must be considered for obtaining a useful prediction model.

heading: 4 VALUE-BASED CALIBRATION
text: In this section, we propose alternative definitions of calibration, value-based calibration, for distribution regression and survival analysis. Then, on the basis of the new calibration definition for survival analysis, we propose our new Kaplan-Meier regularizer, and we discuss its advantages over D-CAL and X-CAL.
We first propose an alternative definition of calibration for distribution regression.
Definition 4.1 (Value-based calibration.) A prediction model Fθ(y|x) for distribution regression is value-calibrated if this equation holds for any value y ∈ [−∞,∞]:
E(X,Y )∼PFθ(y|X) = Pr(X,Y )∼P(Y ≤ y).
This equation means that the average probability of prediction having a value of at most y must be equal to the actual ratio of data points having a value of at most y. The difference between Definitions 3.1 and 4.1 is on the choice of the axis. Whereas Definition 3.1 gives a natural condition for calibration with respect to the τ -axis of a prediction model τ = Fθ(y|x), Definition 4.1 gives a natural condition for calibration with respect to the y-axis. In appendix (Section A.1), we show that a quantile-calibrated model is not necessarily a value-calibrated model and vice versa.
On the basis of Definition 4.1, we define the value-based calibration for survival analysis.
Definition 4.2 (Value-based calibration for survival analysis.) A prediction model Fθ(t|x) for survival analysis is value-calibrated if this equation holds for any time t ∈ [0, tmax], where tmax is the maximum time of the uncensored data points in the dataset D:
E(X,T ,C)∼QFθ(t|X) = Pr(X,T ,C)∼Q(T ≤ t). (5)
Note that we ask Eq. (5) to hold for t ∈ [0, tmax] rather than t ∈ [0,∞]. This is because the prediction model Fθ(t|x) for survival analysis is usually trained for t ∈ [0, tmax] and the prediction Fθ(t|x) for t > tmax is less accurate. We can rewrite Definition 4.2 into another equivalent form: the following equation holds for any subinterval [t1, t2] ⊆ [0, tmax]:
E(X,T ,C)∼Q(Fθ(t2|X)− Fθ(t1|X)) = Pr(X,T ,C)∼Q(t1 ≤ T ≤ t2). This condition exactly matches the explanation of calibration in (Goldstein et al., 2020), i.e., a model’s predicted number of events within any time interval is similar to the observed number. Note also that we can rewrite Eq. (5) into another equivalent form:
E(X,T ,C)∼QSθ(t|X) = Pr(X,T ,C)∼Q(T > t). (6) Here, the left-hand side of Eq. (6) is changed from the CDF Fθ(t|X) in Eq. (5) to the survival function Sθ(t|X), and the right-hand side of the equation is also changed accordingly. On the basis of Eq. (6), we propose our new metric, Kaplan-Meier loss, which is defined as
`KM(θ) = ∫ tmax 0 ( E(X,T ,C)∼QSθ(t|X)− Pr(X,T ,C)∼Q(T > t) )2 dt. (7)
In contrast to D-CAL, we can use this loss directly in the loss function of a neural network model because this loss is differentiable. However, we also propose a new regularizerRKM(θ) by modifying `KM(θ) so that it can be used in a mini-batch training with a set of data pointsB = {(xi, zi, δi)}bi=1:
RKM(θ) = EB∼Q ∫ tmax 0 ( E(X,T ,C)∼BSθ(t|X)− Pr(X,T ,C)∼B(T > t) )2 dt. (8)
Again, we abuse notation (X,T ,C) ∼ B to indicate that we obtain sample data points from minibatchB (rather than the probability distributionQ). Although we used `2 loss in Eq. (7)–(8), we can use any other metric to measure the difference between two distributions. Note that we can compute the first term in the parentheses by
E(X,T ,C)∼BSθ(t|X) = 1 |B| ∑
(xi,zi,δi)∈B
(1− Fθ(t|xi)),
and we can estimate the second term in the parentheses Pr(X,T ,C)∼Q(T > t) by the KaplanMeier estimator. Recall that the Kaplan-Meier estimator is designed to estimate Pr(T > t) even if the dataset contains censored data. Figure 3 illustrates the proposed loss `KM(θ) and regularizer RKM(θ), in which the first term in the parentheses of Eq. (7)–(8) corresponds to the average predicted survival function (thick curve) and the second term corresponds to the Kaplan-Meier survival curve (red curve) and each equation computes the `2 loss between these two curves.
The advantages of our approach can be summarized as follows.
1. Our value-based calibration for survival analysis (Definition 4.2) is intuitive for practitioners in healthcare. Practitioners are often interested in the prediction performance at a specific time t∗ ∈ [0, tmax], and Eq. (5) shows a natural condition that should hold at time t∗. In contrast, Eq. (2) states nothing about specific time t∗.
2. Our value-based calibration for survival analysis (Definition 4.2) is defined only for the range t ∈ [0, tmax]. Regarding the quantile-based calibration, Definition 3.1 means that, if a prediction model Fθ(y|x) for distribution regression is quantile-calibrated, then the distribution of Fθ(Y |X) for (X,Y ) ∼ P is equal to the uniform distribution over [0, 1] (Zhao et al., 2020). However, even if a prediction model Fθ(t|x) for survival analysis is quantilecalibrated, the distribution of Fθ(T ′|X) is usually not equal to the uniform distribution when T ′ refers to the random variable for the survival time of uncensored events. This means that the condition in Definition 3.2 can be satisfied only if we can estimate the distribution of Fθ(T |X) from the dataset D that include censored data by using some method such as Eq. (4). In other words, we need to estimate Fθ(t|x) for t > tmax when we use Definition 3.2.
3. Our Kaplan-Meier loss and regularizer do not require any hyperparameter (other than the batch size |B|), whereas D-CAL and X-CAL require the hyperparameters γ and C.
4. Our Kaplan-Meier regularizer uses a simple `2 loss and it avoids the binning approach used in X-CAL. In the following, we explain the problem in the binning approach. In X-CAL, we count the number of data points in each bin and we update the parameters θ of a neural network during the training so that the numbers of data points in the bins are balanced. Figure 4 illustrates that, for a bin in which the number of data points exceed the target, the parameters θ is updated so that the data points in the first half of the bin are pushed to the left and the data points in the second half of the bin are pushed to the right due to the approximation of the step function as the sigmoid function. However, since the numbers of data points in the three bins in the left exceed the target and the numbers of the data points in the two bins in the right are below the target in the case of Figure 4, no data point in the three bins in the left should be pushed left and the data points in these bins should be used to fill the two bins in the right. Therefore, we sometimes find difficulties in minimizing XCAL due to the binning approach. Note that Goldstein et al. (2020) showed how to avoid this phenomenon for the left-most and right-most bins, but we cannot avoid this problem for the other bins.

heading: 5 EXPERIMENTS
text: In this section, we compare the performance of our Kaplan-Meier regularizer with X-CAL on real datasets. We show that, by using our Kaplan-Meier regularizer or X-CAL, we can reduce both Kaplan-Meier loss and D-CAL. This means that we can use one of these two regularizers both for quantile-based calibration and value-based calibration and do not need to combine the two regularizers for two different definitions of calibration.

heading: 5.1 NEURAL NETWORK AND DATASETS
text: We constructed a single neural network for our experiments and combined it with various loss functions. This neural network was a two-layer perceptron with a single hidden layer containing 128 neurons, and the number of outputs was 32. The activation function after the hidden layer was the ReLU type, and the activation function at the output node was softmax. We used Python 3.7.4 and PyTorch 1.4.0 for the implementation. The Adam optimizer (Kingma & Ba, 2015) was utilized for the training algorithm, with the learning rate set to 0.001, the batch size to 1024, and the other parameters to their default values. We run training for 100 epochs.
We used three datasets for survival analysis: two obtained from the packages in R (R Core Team, 2016) and one from a private data source. Of these former two, one is the flchain dataset (Dispenzieri et al., 2012), which was obtained from the ‘survival’ package and contains 7874 data points (69.9% of which are censored), and the other is the support dataset (Knaus et al., 1995), which was obtained from the ‘casebase’ package and contains 9104 data points (31.9% of which are censored). The dataset from the private data source is of patients with retinopathy disease. This dataset contains 6951 data points (33.3% of which are censored).
We split each of the three datasets into the training (80%) and test (20%) sets by using random partitioning. The neural network model was trained using the training set and the experimental results shown in this section (including the Kaplan-Meier survival curves) were obtained using the test set. When we computed the metrics (D-CAL and Kaplan-Meier loss) on a test dataset, we used full batch (rather than the mini-batch). Regarding the parameters for D-CAL and X-CAL, we used γ = 10000 and the collection C was the set of 20 equally sized bins disjointed over [0, 1].

heading: 5.2 RESULTS
text: Average predicted survival functions without calibration. We compared the average survival functions predicted by the state-of-the-art neural networks with the Kaplan-Meier survival curve on each dataset. In our experiments, we used the fixed neural network and the loss functions presented in DeepHit (Lee et al., 2018), DRSA (Ren et al., 2019), and Survival CRPS (Avati et al., 2019). We used α = σ = 1.0 for DeepHit and α = 0.25 for DRSA as the hyperparameters. Figure 5 shows the average survival functions predicted by using these three loss functions and the Kaplan-Meier survival curve on each dataset. The average of the predicted survival functions should be close to the Kaplan-Meier survival curve if a prediction model is calibrated. However, none of the predicted results was close to the Kaplan-Meier survival curve, which means the loss functions do not achieve value-based calibration.
Calibration with our Kaplan-Meier regularizer and X-CAL. We compared the performances of the proposed Kaplan-Meier regularizer and X-CAL by using each of them as a regularization term in conjunction with Survival CRPS (Avati et al., 2019). More specifically, we used the following loss function for the Kaplan-Meier regularizer:
`(θ) = `S−CRPS(θ) + λRKM(θ), where λ is a parameter, and we used the following loss function for X-CAL:
`(θ) = `S−CRPS(θ) + λRX−CAL(θ), where λ is the parameter. Figures 6 and 7 show results by using these loss functions with various λ from {10, 103, 105}. We can see that the average predicted survival curves are not close to the Kaplan-Meier survival curve with small λ and therefore the predictions were not calibrated. However, regarding the average predicted survival curves with λ = 105, the predicted curves with our Kaplan-Meier regularizer were close to the Kaplan-Meier survival curves and they are calibrated enough. In contrast, the predicted curves with X-CAL were close to the Kaplan-Meier survival curves, but the curves predicted by using our Kaplan-Meier regularizer were better than those with X-CAL.
Table 1 shows the D-CAL and the Kaplan-Meier loss of these predicted results. We can see here that using the Kaplan-Meier regularizer with large λ leads to not only reducing the Kaplan-Meier loss but also reducing D-CAL. Similarly, using X-CAL with large λ also leads to not only reducing DCAL but also reducing Kaplan-Meier loss. These facts demonstrate that we can use one of these two regularizers (the Kaplan-Meier regularizer or X-CAL) to reduce the two metrics (Kaplan-Meier loss and D-CAL) for the real datasets, although we theoretically show that a quantile-calibrated model is not necessarily a value-calibrated model and vice versa in the appendix (Section A.1).

heading: 6 RELATED WORK
text: We use the Kaplan-Meier estimator (Kaplan & Meier, 1958) in our Kaplan-Meier loss to estimate the survival rate S(t) = Pr(X,T ,C)∼Q(T > t). Our loss function is not restricted to the KaplanMeier estimator and we can use any other nonparametric method to estimate the survival rate S(t). For example, we can use the Nelson-Aalen estimator (Aalen, 1978; Nelson, 1969; 1972) instead.
Regarding another calibration for survival analysis, there was study on 1-Calibration (Haider et al., 2020). In 1-Calibration, we consider the calibration of a single point of the predicted distribution, but the entire distribution is not evaluated for calibration. We also note that the distributional divergence for calibration (DDC) Kamran & Wiens (2021) is similar to D-CAL. The difference between DDC and D-CAL are that D-CAL uses a binning approach and DDC uses the Kullback Leibler divergence to measure the distance between the predicted survival function and the uniform distribution.

heading: 7 CONCLUSION
text: In this work, we presented new definitions of calibration, value-based calibration, for distribution regression and survival analysis. On the basis of the new calibration definition for survival analysis, we then proposed a new metric called Kaplan-Meier loss for value-based calibration. The results of experiments showed that it can be used as the regularizer of a loss function in a neural network model for calibration, and it can be seen as a simpler alternative to X-CAL.

heading: 8 REPRODUCIBILITY STATEMENT
text: We summarize our efforts for reproducibility. The details of our Kaplan-Meier loss and regularizer are described in Section 4. In addition, we describe the algorithm of the Kaplan-Meier estimator in Section 1 to make our paper self-contained, although the Kaplan-Meier estimator is a famous algorithm in survival analysis. Regarding our experiments, we described all of the details about our Python environment and the parameters used in Section 5. We used three datasets, and two of them are publicly available datasets. Although we do not attach our source code, we believe that readers of our paper can easily reproduce our results, because our algorithm is easy to implement and influence of the undescribed factors (e.g., choice of the random seed) can be ignored (e.g., the conclusion of our paper does not change even if the numbers in Table 1 fluctuate by 20%).

heading: A APPENDIX
text: A.1 RELATIONSHIP BETWEEN TWO DEFINITIONS OF CALIBRATION
We show that Definition 3.1 and Definition 4.1 are not equivalent. To see this, suppose that we have a probability distribution P on X × Y , where X = Y = [0, 1], such that the corresponding random variables X and Y are the uniform random variables on [0, 1] that satisfy X = Y .
A quantile-calibrated model is not a value-calibrated model. We consider a prediction model F (y|x) defined by
F (y|x) = { 0 (x = 0), y (0 < x < 1), 1 (x = 1).
Since the distribution F (Y |X) is the uniform distribution on [0, 1], this prediction model F (y|x) is quantile-calibrated by Definition 3.1. However, since we have
E(X,Y )∼PF (y|X) = 0.5 and Pr(X,Y )∼P(Y ≤ y) = y for any y ∈ (0, 1), the prediction model F (y|x) does not satisfy the condition of calibration in Definition 4.1.
A value-calibrated model is not a quantile-calibrated model. We consider a prediction model F (y|x) defined by F (y|x) = {
0 (x > y), 1 (x ≤ y).
Then this prediction model F (y|x) is value-calibrated by Definition 4.1, because we have E(X,Y )∼PF (y|X) = y = Pr(X,Y )∼P(Y ≤ y)
for any y ∈ [0, 1]. However, since F (y|x) = 1 for any sample (x, y) ∼ P , the prediction model F (y|x) does not satisfy the condition of calibration in Definition 3.1.

Review summaries:
This paper first defines a new definition of calibration which, in contrast to the one used in prior work, is confined to the maximum observed time in the data. They then propose a KM regularizer for making sure their survival curves are calibrated: they are closer to KM curves. They also claim that their regularization, unlike prior work, does not require any hyperparameters and avoids the binning approach.

In this work, the authors propose a novel approach for learning calibrated predictions for survival analysis and similar tasks. The intuition of the approach is that the average probability that a prediction has a value less than or equal to $t$ should approximately equal the number of observations with value less than or equal to $t$. The authors then derive a differentiable regularization term using this idea and Kaplan-Meier curves. A small set of empirical evaluations suggest the proposed approach modestly outperforms another recent survival analysis calibration method according to some metrics.