Proper Scoring Rules for Survival Analysis
Authors: Hiroki Yanagisawa
Venue: Submitted to ICLR 2023 (2023)
Keywords: scoring rules
survival analysis
time-to-event analysis

Abstract: Survival analysis is the problem of estimating probability distributions for future events, which can be seen as a problem in uncertainty quantification. Although there are fundamental theories on strictly proper scoring rules for uncertainty quantification, little is known about those for survival analysis. In this paper, we investigate extensions of four major strictly proper scoring rules for survival analysis. Through the extensions, we discuss and clarify the assumptions arising from the discretization of the estimation of probability distributions. We also discuss the relationship between the existing algorithms and extended scoring rules, and we propose new algorithms based on our extensions of the scoring rules for survival analysis.


Under review as a conference paper at ICLR 2023
PROPER SCORING RULES FOR SURVIVAL ANALYSIS
Anonymous authors Paper under double-blind review
ABSTRACT
Survival analysis is the problem of estimating probability distributions for future events, which can be seen problem in uncertainty quantification. Although there are fundamental theories on strictly proper scoring rules for uncertainty quantification, little is known about those for survival analysis. In this paper, we investigate extensions of four major strictly proper scoring rules for survival analysis. Through the extensions, we discuss and clarify the assumptions arising from the discretization of the estimation of probability distributions. We also dis- cuss the relationship between the existing algorithms and extended scoring rules, and we propose new algorithms based on our extensions of the scoring rules for survival analysis.
INTRODUCTION
The theory of scoring rules is fundamental theory in statistical analysis, and it is widely used in uncertainty quantification (see, e.g., Mura et al. (2008); Parmigiani & Inoue (2009); [enedetti 2010):[Schlag al. (2015)). Suppose that there is a random variable y whose cumulative distribu- tion function (CDF) is Fy. Given an estimation Fy of Fy and single sample y obtained from Y, scoring rule ((Fy, y) is function that returns an evaluation score for Fy based on y. Since Fy is CDF and y is single sample of Y, it is not straightforward to choose an appropriate scoring rule S(Fy,)) v). The theory of scoring rules suggests strictly proper scoring rules that can be used to recover the true probability distribution Fy by optimizing the scoring rules. This theory shows that there are infinitely many strictly proper scoring rules, and examples of them include the pinball loss. the logarithmic score, the Brier score, and the ranked probability score (see, e.g., Gneiting & Raftery (007) for the definitions of these scoring rules).
Survival analysis, which is also known as time-to-event analysis, can be seen a problem in uncer- tainty quantification. Despite the long history of research survival analysis (see, e.g., Wang al. 2019) for a comprehensive survey), little is known about the strictly proper scoring rules for survival analysis. Therefore, this paper investigates extensions of these scoring rules for survival analysis.
Survival analysis is the problem of estimating probability distributions for future events. In health- care applications, or the onset of disease). The time between a well-defined starting point and the occurrence of an event is called the survival time or event time. Survival analysis has important applications in many fields such as credit scoring (Dirikk et al. 2017) and fraud detection (Zheng et al. 2019) as well as healthcare. Although we discuss survival analysis in the context of healthcare applications, we can use the extended scoring rules for any other applications.
event usually corresponds an undesirable event for a patient (e-g.. death
Datasets for survival analysis are censored, which means that events of interest might not ob- served for a number of data points. This may be due to either the limited observation time window or missing traces caused by other irrelevant events. In this paper, we consider only right censored data, which is widely studied problem setting in survival analysis. The exact event time of right censored data point is unknown; we know only that the event had not happened up to a certain time for the data point. The time between well-defined starting point and the last observation time of right censored data point is called the censoring time.
One of the classical methods for survival analysis is the Kaplan-Meier estimator (Kaplan & Meier 1958]. It is a non non-norameee parametric method for estimating the probability distribution of survival times as survival function s(t), where the value x(() represents the survival rate at time & (i.e., the ratio of

---

Under review as a conference paper at ICLR 2023
the patients who survived at time t). By definition, x<0) = 1 and x(() is monotonically decreasing function.
Since there are many applications that require estimate of the survival function for each patient rather than the overall survival function x(() for all patients, many algorithms have been proposed. In particular, many neural network models have been proposed (e.g.. (Lee al. 2018 Avati et al.,
2019
Ren et al. 2019 Kamran & Wiens) 2021 Tjandra et al.) (021)).
A problem with these neural network models is that most of them not based on the theory of scoring rules except for (Rindt et al. 2022). Since we cannot directly use a known scoring rule due censoring in survival analysis, the state-of-the-art neural network models for survival analysis use their own custom loss functions instead. Even though these custom loss functions can be seen as variants of known scoring rules, they are not proven to be strictly proper for survival analysis in terms of the theory of scoring rules.
We review variants of scoring rules used in survival analysis with respect to the four major strictly proper scoring rules.
Pinball loss. Portmoy's 's estimator (Portnoy] 2003). which is a variant of the pinball loss, has been used in quantile gngression-based survival analysis (Portnoy 2003 Neocleous al.] 2006 Pearce et al. 2022). However, it unknown if Portnoy's estimator is proper or not.
Logarithmic score. Rindt et al. (2022) proved that a variant of the logarithmic score is strictly proper for survival analysis. This variant has been used in the loss function of many neural network models (e.g., (Lee et al. [0088] Avati et al.) 2019 Ren et al., 2019: Kamran & WWinns 2021; Kvamme & Borgan| [021 Tjandra al. 2021 ). However, most of them
202I]Tjandra
use this variant in part of the loss functions, and these loss functions are used without the proof of properness.
Brier score. The IPCW Brier score (Graf al. 1999) and integrated Brier score (Graf et al.
1999) are widely used in survival ana iallyis (e.g.. (Kvamme et al. 2019; Haider et al.,
2020
Han al. 2021 Zhong et al. 2021 I) as variants of the Brier score. However, Rindt et al.
(2022) show that neither of them are not proper in terms of the theory of scoring rules.
Ranked probability score. Variants of the ranked probability score have been proposed in (Avati et al. 2019) Kamran& & Wiens 2021), but (Rindt et al. 2022) show that they not proper in terms of the theory of scoring rules.
Our contributions. We analyze survival analysis through the lens of the theory of scoring rules. First, we prove that Portnoy's estimator, which is extension of the pinball loss, is proper under certain conditions. This result underpins the grid-search algorithm (Portnoy) 2003; Neocleous et al.) 2006) and the CQRNN algorithm (Pearce et al. 2022). which is based on the expectation maxi- mization (EM) algorithm. Second, we show another proof for an extension of the logarithmic score. This scoring rule has already been proven be strietly proper in (Rindt al.. 2022), but our proof clarifies the implicit assumption in the proof. Third, we show that there are two other proper scor- ing rules for survival analysis under certain conditions by extending the Brier score and the ranked probability score. By using these extended scoring rules, we construct two new algorithms by using the EM algorithm.
RELATED WORK
Survival analysis has been traditionally studied under the proportional hazard assumption. Its sem- inal work is the Cox model (Cox) 1972). and many other prediction models have been proposed under this strong assumption. See, e.g., Wang et al. (2019) for comprehensive survey of the pre- diction models based on this assumption. Since do not require the theory of scoring rules under this assumption, we consider survival analysis without this assumption. Note that most of the state- of-the-art art neural network models for survival analysis do not this assumption.
Regarding evaluation metrics for survival analysis, the concordance index (C-index) (Harrell et al, 1982) has been widely used under the proportional hazard assumption. Some variants of the C- index (Antolini et al., 2005; Uno et al. 201T) are proposed for survival analysis without the pro- portional hazard assumption. However, they proven not be proper in terms of the theory of
2

---

Under review as a conference paper at ICLR 2023
74
F(1)
73
72
(tt)
71
Time
Time t
mmax
0
C1
C2
C3
C1 S5
(a) Quantile regression
(b) Distribution regression
Figure 1: Two types of discretization of probability distribution F(t) with B
scoring rules (Blanche et al., 2018; Rindt al. 2022). Therefore, we do not use these variants of the C-index in this paper. We also note that [Sonabend et al. (2022) discuss the problems of using these variants of the C-index in survival analysis.
PRELIMINARIES
We define notation here before showing the extensions of the scoring rules for survival analysis. Unless otherwise stated, we consider a single patient æ, and let T and C' be random variables for the event time and censoring time of this patient, respectively. Let T and e C be samples obtained from I" and C, respectively. We assume that t and are positive real values (i.c., t E R+ and c E IR +)+ ). In survival analysis, we can observe only the minimum 2 min(t, c}. and we use 5 = 1(t I c) to indicate whether represents the true event time (i.e., 5 = 1 means is uncensored, and z = f) or z represents the censoring time (i.e., 8 = 0 means 2 censored, and = = c). this paper, pair of samples (t, is often represented as pair of values (z, 6) to emphasize that we can observe only one of and c. We assume that there exists &max > 0 such that 0 < 2 S 2max, which means that have prior knowledge that is most Emax. Let F(t) be the CDF of T. which is defined as F(t) = Pr(T < t). By the definition of F(t), we have F(0) = 0, and we can represent the probability that the true event time between t and l2 by Pr(ta < T (2) F(12) F(1)).
Survival analysis is the problem of estimating the (() of the true CDF F(I). For simplicity, assume that both Ft)) and Ft)) are monotonically increasing continuous functions. This means that F(t.) < F(22) holds if and only if 0 t 12 < This assumption enables us to calculate F(t) for any time 0 S & < oo and to calculate F (r) for any quantile level 0 S T S 1. When we estimate (() by using a neural network, we usually discretize
F(I) along with the p-axis or the t-axis as shown in Fig. 1 In quantile regression-based survival analysis, F(t) is discretized along the p-axis, F- (Tx) is estimated for 0 To TI < < TB-1 тв 1, and assume that (ra) 0 and (TB( 2max- In distribution regression-based survival analysis,
p == f() discretized along the t-axis, F(C.) is estimated for Co < Ç <.. < <B- < CB
Emax, and we assume that F(Go) = and F(CB)) 1.
Throughout this paper assume that the censoring time and the event time are independent of each other given a feature vector of patient x. This assumption is widely used in survival analysis, and this assumption is represented
Assumption 3.1. T TL C|X.
Note that he Kaplan-Meier estimator (Kaplan & Meier 1958), which is classical non-parametric method for survival analysis, uses this assumption. D-calibration (Haider et al.) 2020), which is one of the widely used metries in survival analysis, also uses this assumption. We can find examples of the other stronger assumptions (e.g., unconditionally random right censoring) used in survival analysis in (Pengl 2021)
3

---

Under review as a conference paper at ICLR 2023
4 PROPER SCORING RULES FOR SURVIVAL ANALYSIS
We briefly review the theory of scoring rules for uncertainty quantification. Let Y be a random variable, and let Fy(y) be its CDF, which is defined as (y())) Pr(Y S y). A scoring rule is function S(Fr, 2) that returns real value for inputs Fy and y, where Fy is an estimation of Fy and y is sample obtained from Y. In this paper, we consider negaaively-oriennted scoring rules. Therefore, the inequality S(F),)) < S(F2. v) means that Fi is better estimation than F22 We can interpret the scoring rule ((yr, y) as penalty function for the misestimation of Fy for sample y-
The proper and strictly proper scoring rules are defined by using the expected score of a scoring rule, which can be written as
S(Py; = (S(Cy
Definition 4.1. A scoring rule ((yy v) proper if ((P: Y) 2 S(Fr;Y) holds.
Definition 4.2. A scoring rule S(Fy, v) is strictly proper if ((y;;Y) 2 S(Py;Y) holds and the equality holds only when Fy = Fy.
These definitions ask a proper scoring rule satisfy that the score S(fy, y) for estimation Fy is always at least ((Fy, y) for true CDF Fy, and the score is minimized if Fy = Fy. This a natural property that any scoring rule should satisfy, and this means that we recover the true Fy if we can minimize the score of a strictly proper scoring rule. The theory of scoring rules shows that there infinitely many strictly proper scoring rules (see, e.g.. Gneiting & Raftery (007
Now we extend these definitions of the proper and strictly proper scoring rules for survival analysis. In survival analysis, the inputs of a scoring rule S(,, (z, 6)) are changed from Fy and y to F and (z, The proper and strictly proper scoring rules defined by using
Š(F;T,C) = E((.))<(T,c) [S( (2,
Definition 4.3. A scoring rule ((F, ;,, (z, 6)) is proper Šf;; 2 S(F;T,C) holds.
Definition 4.4. A scoring rule 5 (z, 6)) is strictly proper if S(F T,C) Z $(F;T, C)) holds and the equality holds only when F = F.
Now we investigate the extensions of the scoring rules for survival analysis. In Sec. we consider quantile regression and survival analysis based on quantile regression. In Secs, we consider distribution regression and survival analysis based on distribution regression.
4.1 EXTENSION OF PINBALL LOss
We first review quantile regression (Koenker & Bassett 1978 Koenker & Hallock 2001). Let Y be real-valued random variable and Fy be its CDF. In quantile regression, estimate the 7-th quantile of Y, which can be written as
Fy (T) = inf (v()22)
The pinball loss (Koenker & Bassett 1978). which is also known as the check function, a widely used scoring rule. The pinball loss for an estimation Fy of Fy and a quantile level is defined as
(0-T)(P-'66) v) if if (T) 2 y, -
Spinban(Fr,, pr (1 (T),))
(I)
if if (T) <
Note that the pinball loss with 7 0.5 is equivalent the mean absolute enror (MAE) and it can be used to estimate the median (i.e., 0.5-th quantile) of Y. This means that the pinball loss is generalization of MAE for any quantile level r E [0, 1]. Note also that we include the quantile level T in the notation Spinball Fy y; T) to clarify that this scoring rule receives 7 as an input.
is known that the pinball loss is strictly proper (see e.g.. (Gneiting & Raftery 2007) )) which means
that we have
E Eyyy (Prinban (f.yy:)) 2 (Spinball (Fr, y5 T)];
4

---

Under review as a conference paper at ICLR 2023
and the equality holds only when (r) F7 by Definition 4.2 sion can be formulated as the problem of computing
Therefore, quantile regres-
arg min E [Spimhan( (By, v)]
Fy
As an extension of the pinball loss for quantile regression based survival analysis, Portnoy's 's estima- for is proposed in Portnoy (2003), which is defined as
Pr (F(F-1(T),2) up, [F-
if = 1, if 6 = 0, + (1- (1
Sportnoy (F, (2. ();
(2)
where A- is the pinball loss defined in Eq. 1 w is weight parameter to control the balance between two pinball loss terms, and Zoe is any constant such that Zoo > 2max- In Portnoy's estimator, we can set an arbitrary constant 0 I w I 1 for the parameter w if To > where To Pr(t I c) = F(c),
but we have to set w = (c(F(c) < F(t) < 7/C > c) = (r /o)/(1 - Ta) otherwise (i.e., To S T). Since we do not know the true value To F(c), we have to resolve this problem to use this estimator.
Before showing how to resolve this problem, we prove that this estimator is proper under the con- dition that w is correct. Note that this is the first result for the quantile regression- survival analysis in terms of the theory of scoring rules.
Theorem 4.5. Pormoy's 's estimator is proper under the condition that w is correct.
Proof. The proof is given in Appendix A.1
This theorem means that the crucial part of Portnoy's estimator is to set an appropriate value for w, and this theorem ensures that we can recover the true probability distribution F-1 by minimizing Eq. (2) if w is correct.
Now we discuss how to set parameter w in Portnoy's 's estimator. First, we emphasize that we cannot avoid the dependence on F(c) in the definition of any of the scoring rules for survival analysis due the discretization of F. Even if we know the true value F-1(7:) for all {T.) we cannot compute F(c) because e is not always contained in (F- == The best we can do is to find quantile levels 7; and Ti+1 such that (T)( < c S (-((+++1 by using the assumption that F is a monotonically increasing function. Note that, even if could find such 7; and Ti+1, we would not be able to calculate some important probabilities such as Pr(c < t S F- = Ti+1 F(c). Therefore, we usually mitigate this problem by using large B, which enables us to assume, for example, F(T++1) - F(T.) = O for all i.
Even if we use a large B assume that we can find the quantile level T/. such that c F-" for any c, the problem that we do not know the true F remains. One of the approaches to tackling this problem the grid search algorithm [Portnoy] 2003 Neocleous et al. 2006). this algorithm, we use a sufficiently large B, and we estimate F (7) of (7:) in the increasing order of i 0, 0,1,...,, B. Suppose that we have estimated (Tr) and we are going to estimate F (Ty() The key idea of this algorithm is that we can find T/d E {T such that ~ F-'(F') if Te = F(e) < Tj. If we can find such Th, we estimate w by using T x Te- we cannot find such The this algorithm assumes that To > Tj and we use an arbitrary constant 0 S w < 1. Portnoy (2003) discuss that this algorithm is analogous to the Kaplan-Meier estimator, and their theoretical analysis (Portnoy) 2003; Neocleous et al. 2006) proves that Portnoy's estimator combined with linear regression can recover the true probability distribution F.
As for another approach, Pearce al. (0022) propose the CQRNN algorithm, which combines neural network and the EM algorithm. Unlike the grid search algorithm, this algorithm estimates (T)) simultaneously by using a neural network. This algorithm starts with an arbitrary initial estimation F, and the parameter w is estimated by using F. Then, this algorithm updates F by using the estimation á of w. and it repeats this alternative estimation of f and t until these values converge. This EM algorithm can be implemented for "free" according to (Pearce et al.) 2022), which means that we can implement it casily in the computation of the loss function of neural network training algorithm and we do not need to construct two separate neural network models for estimating F and w. The experimental evaluation in (Pearce et al. (2022) shows that the CQRNN algorithm performs the best among the quantile regression- survival analysis models.
5

---

Under review as a conference paper at ICLR 2023
4.2 EXTENSION OF LOGARITHMIC SCORE
While we estimate (T)) in quantile regression, we consider distribution regression, in which we estimate (F(G)) For distribution regression, the logarithmic score (Good 1952 is known as one of the strictly proper scoring rules, and is defined as
B-1
(log(F, ,0: (2) = - Σ
1(4 < S S411)
F(G))
==0
Σ
1(s y (411
(3)
i=0
where f; = ((4+1) F(G) for i 0,1,..., B 1.
We extend this logarithmic score for distribution regression- survival analysis as
SCeu log (f, (z, 8); w. ())))n))
B-1
- - <<<<>>>< G++1) (olog S: + (1- + (1 w) (1 - P(G++ .(4)
1=0
where Pr(c < S4tll (F(G++1) (c)))((1 F(c)). I this scoring rule is equal to Eq. (3). Similar to Portnoy's estimator, we cannot set the parameter w of this scoring rule because do not know F((++1) and F(c).
Even though we do not know the correct w, we prove that this scoring rule is strictly proper if the parameter is correct.
Theorem 4.6. The scoring rule Scen- Log (F, 1),5), (2)) is strictly proper if w is correct.
Proof. The proof is given in Appendix A.2
Similar to Portnoy's estimator, we can use both the grid-search algorithm and an EM algorithm similar to the CQRNN algorithm to estimate w. In addition, we show another simpler approach by using the observation that R 0 if B is large. If B is large, 1 F(c) is usually much larger than F 1(++1) (c) (see Fig. 2 a)), and hence we have w = (F((++1) F(c))/(1 F(c)) 0.
Therefore, obtain simpler variant of SCen-log by setting 0:
Scen Sten-simple--og( (F,
B-1
-
Σ 1(s 2 (ōlog ft + log(1 1 - F(GH+2))
(5)
Furthermore, by increasing B to infinity (i.e., B + oc), we obtain the continuous version of this scoring rule:
SCen-cont- (F,,0 - df di
-1 (1- (())),
which is equal to the extension of the logarithmic score that is proven to be strictly proper in (Rindt)
et al. 2022). This clarifies that the proof in (Rindt et al. 2022) implicitly assumes that is suffi- ciently large.
4.3 EXTENSION OF BRIER SCORE
In distribution regression, the Brier score BBiee 1950) is also known strictly proper scoring rule, which is defined as
B-1
Sprier (f,y: f = Σ (1(G < < - -f)²²
(6)
2=0

---

Under review as a conference paper at ICLR 2023
1
F(c)
F(c)
F(S) - F(c)
F(S42)) - F(c)
Time /
Time t
0
G++1
0
(a) Sog
(b) SCea - -Binaryin Brier
Figure 2: Illustrations of computations of w
where J; = ((G+1) - ((G) for i = 0...... B - 1.
We extend this Brier score for distribution regression- paaressio--based survival analysis as
B-1
SCen-Brier (F, (F, (w: Eo (ci) = Σ
w;([ fi)² + (1 - -w:)ff).
(7)
1=0
where
if 1 and (++1 <==t
if 5 = 1 and Ci < <== 1 Gi+1
W/:
0
if z S &
( F(C)
if 8 0 and ( <
Gi+1
fa/(1 - ((c)
if 8 0 and G+1 < 2 = c.
If 8 = it is easy see that Eq. is equal to Eq. 66
We prove that this scoring rule is strictly proper if the set of parameters {w:)er) B-1 is correct.
Theorem 4.7. The scoring rule Scen-Brier (F, (); (wi) {()) strictly proper if wy is correct for all i.
Proof. The proof is given Appendix
We the EM algorithm similar the CQRNN algorithm estimate However, unlike Port- noy's 's estimator and the extension of the logarithmic score, we cannot use the grid-search algorithm in this extension of the Brier score because w; depends on f, such that
Note that each we in this scoring rule is close to zero B is large and = 0. However, since wys are designed satisfy Σg We 1, we cannot use the approximation w; for this scoring rule.
4.4 EXTENSION OF RANKED PROBABILITY SCORE
The ranked probability score (RPS) is also known strictly proper scoring rule (see e.g., Gneiting
& Raftery 2007)). is defined
B S Brier (f, Ci),
SRPS
Σ
(8)
where Spinary-Brier Briir the binary version of Sprier (Eq. (6)) with single threshold Ç:
Spinary Brier Brier (F, y) (1 S () - 1)².
(9)
We extend this scoring rule for survival analysis:
B-1
Seen BT 1 (G) = Σ
Scen- Bina-Bi Huuurr--Brierfee (F, F, (2, 8); wi, Ci),
(10)

---

Under review as a conference paper at ICLR 2023
where Scea- Biinay- Brier is the binary version of SCen-Briet Brier (Eq. with single threshold Ç:
F(()2
if > < if 0 = 1 and 2 = t S C
Scen Biito-Binar---ee Brier (2,
(1 -F(C))2
w(1 - F(G))2 + (1-w)F((3)
if Ō 0 and
S C,
where w -( F(c)).
Since this scoring rule is just the sum of the binary version of Brier scores, it is straightforward to prove this theorem.
Theorem 4.8. The scoring rule (Cen-RPs((F (z, 6); (w; ) ( is strictly proper if w; is correct for all i.
Note that the scoring rule Scen -Cea-Binary-Brier is analogous to Portnoy's estimator. The scoring rule Hein-BBnary-Baae Brier is designed to estimate FC), where ( is an input, and we use F(c) and Ç to set w, whereas Portnoy's estimator is designed to estimate f- (r), where T is an input, and we use F(c) As these two scoring rules similar, can both the grid-search algorithm and an EM algorithm similar to the CQRNN algorithm for Scen RPS-
and
Unlike Sen-log defined in Eq. 4 the parameter of the scoring rule Scen SCen-Binary -Brier is usually not close to zero, because Ç and с are usually not close to each other as shown in Fig. ((b). We note that the parameter of Portnoy's estimator is also not close zero for similar reason.
5 EVALUATION METRICS FOR SURVIVAL ANALYSIS
While we have discussed the extensions of the scoring rules as loss functions, we can use strietly proper scoring rules also for evaluation metries. If use a nonproper scoring rule Snomproper as an evaluation metric, a neural network model can find f such that
E (T,,)) (Soopprrprr (F, F, (z, 5)) ((,,))(TTc)) (S (F, (2,
by using Snon-proper for the loss function. This suggests that we should avoid nonproper scoring rules as evaluation metrics, because we may obtain over-optimized estimation F, which has lower score than F in terms the evaluation metric Snonproper-
Among the extensions of the scoring rules for survival analysis, we can use only Scen SCen-simple-bl simple-log (Eq. (5) as an evaluation metric for survival analysis, because the other scoring rules depend on the parameter w or {wi) 1 Note that this scoring rule Scen SCon-simple-les is valid only if B is sufficiently large. In Appendix B we conducted experiments on choosing an appropriate B, and the
results suggested using 16.
Regarding calibration metrics for survival analysis, while D-calibration (Haider et al. 2020) is widely used, we propose another metric for calibration, KM-calibration. We define this metric
B-1
dKM--aatKs Pavgi ()) dxx.(1l]1 Pawy (G) = Σ (pi log pt pt g:),
where is is the survival function estimated by using the Kaplan-Meier estimator (Kaplan & Meier 1958,, Favz is the average of the estimated CDFs of all patients. p; = x(G+1) - k(G),, and yi (1 Favg(Gi)) (In this computation, we assume that k((B) 0.) This metric is the Kullback-eebler divergence between x(t) and the average of the estimated survival function 1 1- -Fvg (4). This metric is based on the observation that the model's predicted number of events within any time interval should be similar to the observed number (Goldstein al.. 2020).
(1 F avv Eeg((++1))
We note here that calibration is particularly important for survival analysis especially in healthcare applications. we use a prediction model that is miscalibrated, the predictions obtained from the miscalibrated model would be pessimistic or optimistic medical doctors were to make decisions on patient treatments on the basis of such miscalibrated prediction model, the treatments could be harmful for patients because of the pessimistic or opti- mistic predictions. Calster et al. (2019) extensively discuss the importance of calibration in survival analysis.
whole compared with the actual ones.
8

---

Under review as a conference paper at ICLR 2023
Table 1: Prediction performances (lower is better) of extended scoring rules with B 32
Metric
Loss Function
flchain
prostateSurvival
support
SCen #Cen-log-sime -igmle
Con-log
1.5054 = 0.0508
1.3608 0.0295
1.8307 0.0452
Sen Cen-Brier Brier
1.5137 = 0.0557
1.3680 + 0.0291
1.8467 1
SCen RPS
1.6737 = 0.0821
1.4821 = 0.0639
2.1036 =
Portnoy
1.6641 0.0518
1.4352 = 0.0420
2.0645 10645=:.000 0.0455
D-calibration
Con-log
0.0003 = 0.0001
0,0001 = 0.0000
0.0063 = 0.0009
Cen-Brie Brier
0.0004 = 0.0002
0.0001 = 0.0000
0.0071 = 0.0009
S Cen-RI'S RPS
0.0005 = 0.0003
0.0010 + 0.0005
0.0045 + 0.0011
Portnoy
0.0071 = 0.0031
0.0055 - 0.0041
0.0237 + 0.0037
KM-calibration
SCen-log
0.0206 0.0049
0.0312 0.0084
0.0299 = 0.0115
SCen Brier
0.0268 = 0.0071
0.0324
0.0090 0.0090
0.0492 + 0.0125
SCen RPS
0.1553 0.0349
0.5931 = 0.3846
0.2668 = 0.1192
Sportnoy
0.0434 + 0.0067
0.1895 + 0.1413
0.0809 + 0.0381
EXPERIMENTS
In our experiments, we used three datasets for the survival analysis from the packages in R R Core Team (2016): the fichain dataset Dispenzieri al. (2012), which was obtained from the 'sur- vival' package and contains 7874 data points (69.9% 9% of which are censored), the prostateSurvival dataset (Lu- Yao et al. 2009), which was obtained from the 'asaur' package and contains 14294 data points (71.7% of which are censored), and the support dataset Knaus et al. (1995), which was obtained from the 'casebase' package and contains 9104 data points (31.9% of which are censored).
We compared the prediction performances of the extended scoring rules: SCen-log (Eq. (4). SCen- Brier (Eq. (7)). SCen-Serrrrr RPS (Eq. (TO), and SPortnoy (Eq. (2). We used a neural network model with B = 32 to estimate F, and we combined it with the EM algorithm to estimate w or [w:),--' o1 This means that we used the CQRNN algorithm (Pearce et al., 2022). which is the state-of-the- art model for quantile regression- bbaeed survival analysis, for SPortwoy- We used Scen llo---o---- simple (Eq. ( as a metric for discrimination performce. We used D-calibration (Haider al. (0200) and KM-calibration as metrics for calibration performance, where we used 20 bins of equal-length for D-calibration.
Pable[[soowe the results, and each number shows the mean and standard deviation of the measure- ments five-fold cross validation. These results showed that SCen-log and SCen-Brier performed the best. Note that the former one is almost equal to the variant of the logarithmic score used in Rindt et al. 2022), and that the latter one is our new extension of Brier score. Compared to these two scoring rules, the prediction performance of SCen-RPS and Sportnoy and these results seemed to be not close to the true probability distribution, even though we prove that they conditionally proper scoring rules. It is considered that the estimation ve of parameter w by the EM algorithm was not accurate enough to converge to the true probability distribution for SCen- RPS and Sportnoy. As we illustrate Figure 2 the parameter w of SCen SCen-RPS RPS (and Sportnoy) is usually not close to zero unlike SCen and this fact indicates that it was difficult to find correct for these two scoring rules.
worse than expected
7 CONCLUSION
We have discussed the extensions of the four scoring rules for survival analysis, and we have proved that these extensions are proper if the parameter or [w;) B-1 is correct. By using these scoring rules, we present neural network models combined with the EM algorithm to estimate the parameter, and our experiments showed that the models with Scen and Scen-Bria performed the best. In addition, we clarified the hidden assumption in the proof of the variant of the logarithmic score for survival analysis (Rindt et al. 2022),, which suggests to sufficiently large B when use it as an evaluation metric. We believe that our approach of extending scoring rules for survival analysis can be used for many other known strictly proper scoring rules.
9

---

Under review as a conference paper at ICLR 2023
REFERENCES
L. Antolini, P. Boracchi, and E. Biganzoli. A time-dependent discrimination index for survival data. Statistics in Medicine, 24(24):397-33444 2005.
A. Avati, T. Duan, S. Zhou, Jung, N. Shah, and A. Y. Ng. Countdown regression: Sharp and calibrated survival predictions. In Proceedings of UAI 2019, pp. 145-155, 2019.
R. Benedetti. Scoring rules for forecast verification. American Meteorological Society, 138(1): 203-211, 2010.
Blanche, M. W. Kattan, and A. Gerds. The c-index not proper for the evaluation of t-year predicted risks. Biostatistics, 20(2):347-3577 2018.
G. W. Brier. Verification of forecasts expressed in terms of probability. Monthly Weather Review, 78(1):1-3, 1950.
B. V. Calster, D. J. McLemnon, M. van Smeden, L. Wynants, and E. W. Steyerberg. Calibration: the Achilles heel of predictive analytics. BMC Medicine, 17:230, 2019.
D. R. Cox. Regression models and life-tables. Journal of the Royal Statistical Society, Series B, 34 (2):187-220) 1972.
L. Dirick, G. Claeskens, and B. Baesens. Time to default in credit scoring using survival analysis: benchmark study. Journal of the Operational Research Society, 68(6):652-663 262522666,-06 2017.
A. Dispenzieri, J. A. Katzmann, R. A. Kyle, D. R. Larson, T. M. Therneau, C. L. Colby, R. J. Clark, G. P. Mead, S. Kumnr, L. Melton III, and V. Rajkumar. Use of nonclonal serum immunoglobulin free light chains to predict overall survival in the general population. Mayo Clinic Proceedings, 1776)35 2012.
Gneiting and A. E. Raftery. Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association, 102(477):359-37) 2007.
M. Goldstein, X. Han, A. M. Puli, A. Perotte, and R. Ranganath. X-CAL: Explicit calibration for survival analysis. In Proceedings of NeurIPS 2020, pp. 18296-18307, 2020.
I. J. Good. Rational decisions. Journal of the Royal Statistical Sociery. Series B (Methodological), 14(1):107-114, 1952.
E. Graf, C. Schmoor, W. Sauerbrei, and M. Schumacher. Assessment and comparison of prognostic classification schemes for survival data. Statistics in Medicine, 18( 18(17-18):529-2544 1999.
H. Haider, B. Hoehn, S. Davis, and R. Greiner. Effective ways to build and evaluate individual survival distributions. Journal of Machine Learning Research, 21 21(5)):1-66 2020.
X. Han, Goldstein, Puli, T. Wies, Perotte, and R. Ranganath. Inverse-weighted survival games. In Proceedings NeurIPS 2021, pp. 12160-172,,0 2021.
F. E. Harrell, R. M. Califf, D. B. Pryor, K. L. Lee, and R. A. Rosati. Evaluating the yield of medical tests. Journal of the American Medical Association, 247(18):2543 -246. 1982.
Kamran and J. Wiens. Estimating calibrated individualized survival curves with deep learning. In Proceedings of AAAI 2021, pp. 240-248, 2021.
E. L. Kaplan and P. Meier. Nonparametric estimation from incomplete observations. Journal of the American Statistical Association, 53(282):457-45 481, 1958.
W. A. Knaus, F. E. Harrell, Jr., J. Lynn, L. Goldman, R. S. Phillips, A. Connors, Jr., N. V. Dawson, Califf, N. Desbiens, P. Layde, R. K. Oye, E. Bellamy, R. Hakim,
W. J. Fulkerson, Jr., and D. P. Wagner. The SUPPORT prognostic model. Objective estimates of survival for seriously ill hospitalized adults. Study to understand prognoses and preferences for outcomes and risks of treatments. Annals of Internal Medicine, 122(3):191-203 1995.
R. Koenker and B. Bassett, Jr. Regression quantiles. Econometrica, 46(1):33-50, 1978.
10

---

Under review as a conference paper at ICLR 2023
R. Koenker and K. F. Hallock. Quantile regression. Journal of economic perspectives, 15(4):143- 156, 2001.
H. Kvamme and Ø. Borgan. Continuous and discrete-time survival prediction with neural networks. Lifetime Data Analysis, 27:710-736, 2021.
H. Kvamme, Ø. Borgan, and I. Scheel. Time-to-event prediction with neural networks and cox regression. Journal of Machine Learning Research, 20(129):1 1-30,, 2019.
C. Lee, W. R. Zame, Yoon, and M. van der Schaar. DeepHit: deep learning approach to survival analysis with competing risks. In Proceedings of AAAI-18, pp. 2314-2321, 2018.
G. L. Yao, P. C. Albertsen, D. F. Moore, W. Shih, Y. Lin, R. S. DiPaola. M. J. Barry, A. Zietman, M. O'Leary, E. Walker-Corkery, and S.-L. Yao. Outcomes of localized prostate cancer following conservative management. Journal of the American Medical Association, 2009.
A. Mura, M.C. Galavotti, H. Hykel, and B. de Finetti. Philosophical Lectures on Probability: collected, edited, and annotated by Alberto Mura. Synthese Library. Springer Netherlands, 2008.
Neocleous, K. V. Branden, and S. Portnoy. Correction to censored regression quantiles by S. Portnoy, 98 (2003), 1001-1012. Journal of the American Statistical Association, 101(474):860- 861, 2006.
G. Parmigiani and L. Inoue. Decision Theory: Principles and Approaches. Wiley Series in Proba- bility and Statistics. Wiley, 2009.
T. Pearce, J.-H. Jeong, Y. Jia, and J. Zhu. Censored quantile regression neural networks. In Pro- ceedings of NeurIPS 2022 (to appear), 2022.
L. Peng. Quantile regression for survival data. Annual Review of Statistics and Application, 8: 413-437, 2021.
S. Portnoy, Censored regression quantiles. Journal of the American Statistical Association, 98(464): 1001-1012, 2003.
Core Team. R: A Language and Environment for Statistical Computing. Foundation for Statis- tical Computing, Vienna, Austria, 2016. hRLpptps: :/www http:www.wppp..... R-projeett org/
K. Ren, J. Qin, L. Zheng, Z. Yang, W. Zhang, L. Qiu, and Y. Yu. Deep recurrent survival analysis. In Proceedings of AAAI-19, pp. 4798-4805, 2019.
D. Rindt, R. Hu, D. Steinsaltz, and D. Sejdinovic. Survival regression with proper scoring rules and monotonic neural networks. In Proceedings of AISTATS 2022, 2022.
K. H. Schlag, Tremewan, and J. J. van der Weele. A penny for your thoughts: a survey of methods for eliciting beliefs. Experimental Economics, 18:457-490, 2015.
R. Sonabend, A. Bender, and S. Vollmer. Avoiding c-hacking when evaluating survival distribution predictions with discrimination measures. Bioinformatics, 38717):418--4184 2022.
D. E. Tjandra, Y. He, and Wiens. hierarchical approach to multi-event survival analysis. In Proceedings of AAAI 2021, pp. 591-599, 2021.
H. Uno, T. Cai, M. J. Pencina, R. B. D' D'Agotinno, and L. J. Wei. On the C-statistics for evaluating overall adequacy of risk prediction procedures with censored survival data. Statistics in Medicine, 30(10): 1117, 2011.
P. Wang, Y. Li, and C. K. Reddy. Machine learning for survival analysis: A survey. ACM Computing Surveys, 51(6): 2019.
P. Zheng, S. Yuan, and X. Wu. SAFE: A neural survival analysis model for fraud early detection. In Proceedings of AAAI-19, 1278-1285, 2019.
Qixian Zhong, Jonas Mueller, and Jane-Ling Wang. Deep extended hazard models for survival analysis. In Proceedings of NeurIPS 202/, 2021.

---

Under review as a conference paper at ICLR 2023
A PROOFS OF THEOREMS
We give proofs of the theorems, which are omitted from the main body of this paper.
A.1 PORTNOY'S 's ESTIMATOR
We show a proof of Theorem 4.5
Proof. We consider fixed e C, and prove
E
Poottny (2)
PSortnoy (F, (2,8); w), r)]
(11)
C-
tT/c=
for these four cases separately.
Case 1: c S (T), (
Case 2: max (F) <c.
Case 3: (T) <cs 8- (T).
Case 4: (T) <Cs
Note that, if Inequality C. and we can prove
holds for any C, we can marginalize the inequality with respect to
toT Sprttoy ( (F, (2, twT,erC
(2, w)))
which means that Sportnoy (f, (z, w; proper. Therefore, we prove Inequality 11 for the four cases.
Case We prove the case for S min(F-1(r), (-))) This means that Te S and (T f (1) - Te). Hence, we have
Sportnoy (2,),,) (
ift C,
( -1 (F- if
(1- (f-r)(f-11)) (r) (f-))) (r)-t)
ift $ C,
-Te(1 - (7)-))/(11--) ift > c.
By Assumption 3.1 we have Pr(l < c/ c) Pr(( () = and Pr(1 > c/C = c) = Te.
Hence, we have
B C=
[S (F, (2,0);,,77)
Pr(l s c/C = c) (7) (1 -7)
E
[1]
tuT|C=CAte"
-Pr(t> dC= = c)Te(1 7) F (r)/(11 Tc)
Te(1 (7
[t]
1 1- (=
11] + Te(1 -T)
= -11 (1-T)
B
E
C=c,te
1 tuTIC=
Since this value the same for Sportnoy (f,, (z,0); w. 7) and Sportnoy (F, (z, 0); w. 7), we have
E C=
[Sportno (F, (2,0);00 w. t-TC==
[Spornny (z, 8); w, 7)].
12

---

Under review as a conference paper at ICLR 2023
Case 2. We prove the case for
<c
If F (F) = F- (T) c, then we have
Sportwoy (F, (2. W.
(T)
if
+ wp (T), (1) ift>c
(f-))(F-(t)-)))
f
(7) (T)-
if (r) <tsc.
wT (t)-) -1-)))))(1-)((-((1
ift>6
(1 (1-))((-()--)) F
(1)- (T) (T),
(1)-))))
(1)- if/>4
(1 (1-))(F-1(1-- t) f
(F)
(F) <t,
where we used -wr(f-1 (T) t) w(1 (F 1 (T) t) when (T) < < (T) and w N 0 for the incquality.
If 8- (T) F (T) < c, then we have
F
Sportaoy (2,6);
(-(F-1 tr),
ift c,
(F) (1 (f)
ifl e,
(1 (1- (F) -t)
(f-
f-
(-) <tsc.
(F- - -1 (1-
ift>c
(1 (f)))) f
(f))()-(()- (T) <IS FF)
(T)- (F
-T(F-((1) ift>c
(1-))( +) if)
(T)-t) if <t,
where we used (F (T) - t) > w(1 - () - t) when f-1(7) < S (7) and
> 0 for the inequality.
By Assumption 3.1 we have Pr(l S F F = P = and Pr(FF-(77) <
C = 1 T. Hence, we have
E [Sportnoy (f, (2,0); w. 7) twr
2 F - (1-7)
1)
E
[
(T
-Pr(F- < HCC c)
E
I (7){t
(1- WT C=0,5-1)))
+T
E
[1].
By using similar argument, we have
E [Sportnoy (F, (2,8):u w. 7) -1 (=T/(=cttp-110 tTTCC=e,F Ftt))tt tt.
E
[t] +
E
Co
13

---

Under review as a conference paper at ICLR 2023
Note that this equation holds with equality.
Hence, we have
E C-T
Porrtnoy (F, (2,));
[S (2, w. 7)
w/
Case 3.
We prove the case for F <T5
We have
Sportnoy (F, (2,0);,,7)) = (T),
i
(F ( (F)) ift c
(1 (1-))((-((t-)))
if
wll (f) -)) -101-((()--011(-((-----0-------- (f) ift>
(1 (F-))(-1 F
2
(7) if <iSc
(T) if/>c
where used -wr (7- 1) S w(1 (T) t) when F-1(7) < 1 s c (T)
and w 2 0, and w(1 r) (F-
-T (001f-1100) ( - c) when F-)(7) > cand w 2
for the inequality. By Assumption 3.1 we have Pr(t < C c) = Pr(t S F-) =
(r) L c/c = c) = To T. Pr(l cC =c) = Te. Hence, we have
E SS F, (2,0),,,77) InT
2 F Prtss -11 (1-
E WT (r)
[1]
()) (T) +T
E
[8)
(+))tt
-Pr(t> > dC
= (11-r) t~T CT
E
[t] +T
E
| + TC.
By using similar argument, we have
(1 (T) F-
Sportnoy (F, (2,0);
(T -t)
(F) <tsc
-(( -c
ift>c
and have
E t~T
[SPortnoy (F (2) - -1
E
+r F
E
[1 +cc
Note that this equation holds with equality.
Hence, we have
E C=
[Sportnoy (F, (2)));
[S (F, (z, (), w. 7)].
w WT/C=
Case 4. We prove the case for (-5
14

---

Under review as a conference paper at ICLR 2023
We have
Sportnoy (F, (2, w);
(T),
i
up ( (1- 200) ift>
(1-))((-())-t))
i
(T)-
(T) <1sc,
- (1- itt>
(1-
if i
(1 (1-))((-2(7)--)
if (T) itt>
w(1 - (F)- -c) - (1-w)r(F-1(6) - c)
(1-))(E-
ift
-Tc(1 -) (T) -)((1 ()((1//) TT) cc
- t) when F (T) t < c and w 2 0, and -wr (T) -c) > w(1 -(1 ( (7) - c) when c > (r) and w 2 0 for the inequality, and (T Tc) Te) when To S T for the last equality. By Assumption 3.1 we have Pr(t - c| c) = P S = To and Pr(t > cC = c) = 1 Tc. Hence, we have
> where we used -wT(F- (F) - (F)
B bT/C=
Sportno (F, (2, w. m)
2
c) F (T) 7)
E
[
t~ CT
Te
-Pr(1> cC c)Te(1 7) --) Te)
1- Te
[ Tc(1 - T)
= -11-1
By using similar argument, we have
[ Ta(1 T)
E C=
PSurtnoy (F, -1
E
c.
WTIC=s,te
1 - To
Note that this equation holds with equality.
Hence, we have
E C=
Sportnoy (F, +)
E w
[S (F, (2, w. [)]]
A.2 VARIANT OF LOGARITHMIC SCORE
We show a proof of Theorem 4.6
Proof. We consider fixed C, and let t be sample obtained from T. Let be the index such
that G. E c < (i+1- Since Assumption 3.1 holds, we have Pr(ç, < t - C--1CC = c) = =Pr(ç, <
t < (++) = F(G++1) F(G;) fj for any j < i, Pr(G, < 1 cCC = F(c) F(G.). and
15

---

Under review as a conference paper at ICLR 2023
Pr(c < [ c) Pr(c F(c). Hence, we have log (f, (z, 8); w, (ci)
BENTIC=elS log
Σ Pr(( < < C5+1/C C c) log
-Pr(G: <1 cc -c) log cog
-Pr(c < 1C cc) (wlog ft + - w) log(1 F((++)))
-E fy log log
Kó
(F((c) - -F(Gi) log fi
(1 (1-F(c) (w log + w- log(1 F((++1))
Σ Sj log I3 FF(()) F(S++1));
where we used = (F((4+1) - F(c))/(1 - F(c)) for the last equality.
Hence, we have
B (F,(2) (2, w, ((i) -Sa (2, 8); (6))=n))
- -- flog f - log 15) (1 F(GG+1))(lo((1 F(G++)) - log(1 F((++1))
0,
(12)
where we used the fact that the Kullback-Leibler divergence between two probability distributions non-negative for the inequality. This means that the inequality
Σ Pa log pk) 2 0
holds for any two probability distributions and Pa and the equality holds only if ps P.. for all k. Here, we use an + + 2)-dimmnsio vector p (po, Pp..P...... ,P.....PP.1... Pi+1), and we set Pa = fa all
k < and we set p:+1 = - 1(G+1). Note that the vectors and p constructed in this way is probability distribution (i.e., E.P4 Ph: 1).
Since Inequality have
(12 holds for any C, we marginalize the inequality with respect to C, and
(F, (z, w. (c)) (<)) 2 CSCer (z, (); w. f
which means that Scen-log log (F, (z, ()) is proper. Morcover, the equality holds only if F. = F, and therefore, SCen-log [F, 8)) is strictly proper.
A.3 VARIANT OF BRIER SCORE
We show proof of Theorem 4.7
Proof. We consider fixed C, and let t be sample obtained from T. Let be the index such
that S < c S S441. Assuming that Assumption 3.1 holds, we have Pr(ç, < t S (j C = c) =
Pr(c, < G+1) = F(G+1)- - F(G;) f3 for any < Pr(G < < C = = F(c)))(((()))
16

---

Under review as a conference paper at ICLR 2023
and Pr(c< ) F(c).
Hence, we have
Σ jKi
Pr(s <t ( (1 (1- k j/k
+Pr(s: <1 <1s (1 (1-B))++> Σ h
[1-
+Pr(c< < t|C [Pr(cct(C=oc) =))
wii(1- - fl)2 (1 + Σ 83 + Σ
(1 +1 wy)
jンز
Ei (1 (1- (1-13)²+ JZi
1j)2+ + Σ Ph (F((c) - -(()))
(1-11)++ (1-
Σ cx
j
+(1 - +(-F(()) wi((1 - fi)2 + (1 w;: + Σ
13 + Σ
W (1
+ (1-
jCi
5>b
= Σ - - +1),
where we used
0 if 8 = 1 and G+1 < = =1
if 8 = 1 and
< 5 li+1
if S G
for the first equality and
(F(G++))F(())(CC) F(c) (1/(1-)7)
i = and 0 < = S Gi+1
W/ (0/(1-(())
5 = 0 and (+1 < 2 =
for the last equality.
Hence have
C=o -Seen-Brier (F, (2, E Brier (F, (z, 8))] Σ(1 - -1 - -f5 - f2))
1
= Σ -1,²
> 0.
(13)
Note that the equality holds only if f; Sj holds for all j.
Since Inequality (13) holds for any C, we have
EPT SScen Brier (F, (z, 6))] 2 C (F, (2))
which means that SCen- Brier (F, (z, (2) is strictly proper.
B ADDITIONAL EXPERIMENTS
In this section, we report the results of our additional experiments. The source codes used in our experiments are attached the supplementary material.
In our experiments, we used the fichain, prostateSurvival, and support datasets, and we split the data points into training (60%), validation (20%), and test (20%). For each dataset, we divided the
17

---

Under review as a conference paper at ICLR 2023
Table 2: Comparison between two variants of the logarithmic score for
Metric
Loss Function
fichain
prostateSurvival
support
SCon lig-simple
3Cen-log
6.4618 + 0.1204
1.3460 0.0476
1.5422 - 0.0704
SCen Mion-log-sippe
6.4176 + 0.1266
1.3447 0.0451
1.5368 = 0.0701
D-calibration
SCen log
0.0045 + 0.0004
0.0002 0.0000
0.0370 = 0.0032
SCen lig-simple
0.0127 0.0013
0.0002
0.0001
0.0349 + 0..024
KM-calibration
SCen Blgg
0.0048
0.0026
0.0048
0.0028
0.0057 + 0.0027
See Jug-siimlee
0.0614 0.0081
0.0083 + 0.0024
0.0061 + 0.0033
Table 3: Comparison between two variants of the logarithmic score for B = 16
Metric
Loss Function
flchain
prostateSurvival
support
Sceu-log sean-log-ssmppg
SCen-log
3.6774 = 0,0386
1,2880 = 0.0247
-ogn-log-stmplee
3,6676 + 0.0424
1.3447 = 0.0451
1
D-calibration
S log
0.0005 + 0.0002
0.0001 0.0000
1.0147=0.000
- lig-ll
0.0013 0.0004
0.0002
0.0000
0.0143 = 0.0021
KM-calibration
SCen -Cen-log
0.0117 0.0046
0.0142 0.0006
0.0149 = 0.0080
Cen CCen-log-sing ligple
0.0162 + 0.0049
0.0158 0.0063
0.0158 + 0.0100
Table 4: Comparison between two variants of the logarithmic score for B = 32
Metric
Loss Function
flchain
prostateSurvival
support
Sen eimmpe
SCon-log
1.5054 0.0508
1.3608 = 0.0295
1.8307 = 0.0452
SCen -ig--llee
1.5059 + 0.0513
1.3609 + 0.0301
1.8296 + 0.0446
D-calibration
SCen
0.0003 0.0001
0.0001
0.0000
0.0063 1 0.0009
Sleu-log-simpg
0.0003 + 0.0001
0.0001 + 0.0000
0.0062 = 0.0012
KM-calibration
SCen llgg
0.0206 + 0.0049
0.0312 + 0.0084
0.0299 + 0.0T15
S Seen-logsstupl
0.0213 0.0049
0.0343 = 0.0102
0.0288 = 0.0127
time interval 00, 2max + E), where 10-3, into B - equal-length intervals to get the thresholds { for distribution regression-basee survival analysis, and we divided the unit interval [o, I] into B equal-length intervals to get the quantile levels { for quantile regression-bases survival analysis.
All our experiments were conducted on virtual machine with an Intel Xeon CPU (3.30 GHz) processor without any GPU and 64 GB of memory running Red Hat Enterprise Linux Server 7.6. We used Python 3.7.4 and PyTorch 1.7.1 for the implementation.
We constructed multi-layer perceptron (MLP) for our experiments. It consists of three hidden layers containing 128 neurons, and the number of outputs was B. The type of activation function after the hidden layer the rectified linear unit (ReLU), and the activation function at the output node was softmax. We can satisfy the assumption that (t) is monotonically increasing continuous function by using it. In distribution regression based survival analysis, each output of MLP estimates
= F[G - ((G)) for i = 0, 1,., - 1. By using these outputs {f) we can calculate { and we can represent the function (() as piecewise linear function connecting the values {F(G)) Since fi > 0 holds for all i, F(t) estimated in this way is a monotonically increasing continuous function. We can estimate F for quantile regression- based survival analysis by using a similar way.
First, we investigated the differences the prediction performances between SCen- (defined in Eq. (4) and "-cen-log-simp (defined in Eq. (5)) by using Scen log- simple. D-calibration, and KM- calibration as metrics. Tables 2 4 show the results for B = 8, 16, 32, respectively, where the bold numbers were used to emphasize the difference between two scoring rules. These results showed that the prediction performance of these two scoring rules were similar for the prostateSurvival and support datasets even for B 8. However they showed different prediction performance for the fichain dataset for B = 8 and B = 16, but the performance difference were negligible for B = 32. Therefore, used B 32 in the other experiments in this paper.
18

---

Under review as a conference paper at ICLR 2023
Table 5: Prediction performances (lower is better) with various loss functions for B 32
Metric
Model
fichain
prostateSurvival
support
SCon log lig-siplle
DeepHit (a 0.1)
1.5200 + 0.0398
1.3644 0.0293
1.8481 + 0.0453
DeepHit (a 1)
1.5858 + 0.0495
1.3813 0.0318
1.9996 0.0525
DeepHit (a 10)
2.0313 0.1648
1.5688 0.0823
2.3657 0.0441
DRSA
1.6783 = 0.0393
1.4631 0.0273
S-CRPS
+ +
1,4589 + 0.0042
2.1162 + 0.1095
IPCW BS(() game
1.9265 + 0.1093
1.6413 0.0743
2.3581 + 0.1604
-Cnn-log
1.5054 0.0508
1.3608 + 0.0295
1.8307 + 0,0452
SCen-Brier
1.5137 = 0.0557
1.3680 = 0.0291
1.8467 + 0,0448
SCen-RPS RPS
1.6737 = 0.0821
1.4821 + 0.0639
2.1636 + 0,10
Pootnoy
1.6641 = 0.0518
1.4352 + 0.0420
2.0645 + 0.0455
D-calibration
DeepHit (a 0.1)
0.0005 + 0.0002
0.0001 0.0000
0.0056 1 0.0009
DeepHit (a 1)
0.0008 - 0.0003
0.0003 + 0.0001
0.0062 + 0.0010
DeepHit (a 10)
0.0138 = 0.0046
0.0064 = 0.0035
0.0179 + 0.0053
DRSA
0.0043 + 0.0011
0.0047 + 0.0004
0.0057 + 0.0006
S-CRPS
0.0032 + 0.0005
0.0018 0.0004
0.0072 + 0.0011
IPCW BS(() game
0.0022 = 0.0006
0.0083 + 0.0018
0.0060 + 0,0008
Cen-
0.0003 = 0.0001
0.0001 + 0.0000
0.0063 1 0.0009
SCen-Brier Brier
0.0004 = 0.0002
0.0001 + 0.0000
0.0071 + 0.0009
SCen- RPS
0.0005 = 0.0003
0.0010 + 0.0005
0.0045
SPortnoy
0.0071 = 0.0031
0.0055 + 0.0041
0.0237 0.0037
KM-calibration
DeepHit (a 0.1)
0.0264 = 0.0071
0.0418 + 0.0139
0.0249 0.0067
DeepHit (a 1)
0.0362 = 0.0084
0.0599 0.0341
0.0545 + 0.0110
DeepHit (a 10)
0.2077 + 0.0543
0.4937 + 0.1772
0.4273 + 0.1188
DRSA
0.1929 + 0.0135
0.1845 + 0.0050
0.2103 + 0.0162
S-CRPS
0.2759 = 0.1279
0.6414 + 0.3043
0.4090 + 0.1499
IPCW BS(() game
0,2770 = 0.0789
0.4246 + 0.0841
0.5325 + 0.1342
SCen
0.0206 = 0.0049
0.0312 = 0.0084
0.0299 0.0115
Cen Brier
0.0268 = 0.0071
40824
0.0492 + 0.0125
S Cen CCn-RRS
0.1553 = 0.0349
0.5931 + 0.3846
0.2668 + 0.1192
SPortnoy
0.0434 + 0.0067
0.1895 + 0.1413
0.0809 + 0.0381
Next, we computed the prediction performance of several loss functions used in the state-of-the-art neural network models. The loss function of DeepHit (Lee et al. 2018) consists of two terms. The first term is equal to the extension of the logarithmic score SCen log-smplees and the second term is used to improve a ranking metric between patients. The parameter is used to control the balance between these two terms, and the weight for the second term is increased by using a large a. The loss function of DRSA (Ren et al.) (019) set a = 0.25 for the parameter. S-CRPS (Avati et al., 2019) is a variant of the ranked probability score, but Rindt et al. (2022) showed that this scoring rule is not proper in terms of theory of scoring rules. We also implemented the IPCW BS(() game model, which is proposed in (Han et al.) 2021). Table shows the results. The prediction performance of DeepHit degraded by using a large O, which means that it is better to use Scen -oon-log-simple by setting a = 0. The other prediction models did not outperform Scen-log and BCin-Brier-
also be seen
variant of logarithmic score, and
Finally, we show an ablation study on the training models with and without the EM algorithm. Figure 3 shows the average survival functions for B 32, which means that the average of F(t)= = 1 Ft) for all patients in test dataset were shown. The parameter w (or {w:))__)) is included in the computation of the gradient in the neural network training of the prediction model without the EM algorithm, whereas the prediction model with the EM algorithm handles the parameter as constant. The actual survival functions were estimated by the Kaplan-Meier estimator. These results showed that the average predictions for the extension of the logarithmic score were close to the Kaplan-Meier curve regardless of the use of the EM algorithm for the three datasets. As for the other three estimators, the average predictions with the EM algorithm were closer than those without the EM algorithm to the Kaplan-Meier curves. These results mean that need the EM algorithm except for the extension of the logarithmic score.
19

---

Under review as a conference paper at ICLR 2023
fichain
prostateSurvivel
support
ligrrthmiic
0.8
0.8-
0.8-
Kap an-tteeer
0.6-
0.6
Survea 010
0.6
0.2
Ingetthmiic w/ EM
0.2
ligarthhmic EM
0.2
ogarfthmic wlo EN Kapla-Meeer
logofithmic w/o EN
Kaplan Haler
0.0
0.0
0.0
2000
1500
(a) SCan log on flchain
(b) Sen on prostateSurvival
(c) Scen llg on support
fichain
prostatesurvival
support
1.0
1.0
1.0
Brier EM
Erior wio EH
0.8
0.8-
0.5
Kaplan Weier
Sunovllatt
synwalraate
0.4
0.4
0.44
0.2
Brier wt EM
0.2
Brier EM
0.2
Brirr
Brier
Meier
Kaplon
1000
2000
5000
4200
5000
20
40
60
80
100
300
2000
1360
2000
Tima
Time
Time
(d) SCen Brler fichain
(c) Scen Brier prostateSurvival
S Scen Brier on support
fichain
prostateSurvival
support
1.0
1.0
RP5 ml EM
RP5
0.8
Meler
0.6
0.6
Survaal 220
0.6
0.4
0.4
RP5 wio
wio
ReplanNwerr
0.0
1000
2000
3000
4000
5000
0.0
20
80
0.0
500
1000
1500
2000
Time
Time
(g) Scen RPS on flchain
(h) SCen-RPs on prostateSurvival
Scen - RPS on support
Aichiin
prostateSurvival
support
1.0
1.0
Portnoy sd
Portney
0.0
0.8-
0.8
Keilen-Meier
0.6
Sursva 1010
0.4
0.4
0.2
Parincy et EN
0.2
Pbrinoy et EM
0.2
Perinay wis BM
Portncy wa BM
Saplan Noier
Kaplon Moirr
0.0
0.0
0.0
1000
2000
5000
5000
80
500
1000
1500
2000
Time
Time
(i) S Porttny on fichain
(k) Sportnoy on SurvtateSurvival
(1) Sportnay on support
Figure 3: Comparisons of average survival functions with and without EM algorithm for B = 32
20


Reviews:
[{"review_id": "nG5SwiZUvjH", "reviewer": "Reviewer_hCyF", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose four extensions of training objectives in survival analysis. Under some conditions, these metrics are strictly proper scoring rules. The main contributions are four new strictly proper objectives.", "summary_of_the_review": "I think the authors deal with an interesting and important topic: proper scoring rules in survival analysis. They also made some contribution to propose new ones. But I think more evaluations need to be done as mentioned in weakness. I am happy to raise my score if the authors can address my point both theoretically and empirically in the weakness part.", "strength_and_weaknesses": "Strength:\n1. Strictly proper scoring rules are important for both training and evaluation in survival analysis. Due to the existence of censoring, strictly proper scoring rules are hard to find. The common training objectives are usually not strictly proper. The authors tackle this difficulty and propose four new extensions of existing objectives. \n\n2. The authors also propose another evaluation metric for calibration based on the KL-divergence between Kaplan-Meier estimator and the proposed estimation. \n\nWeakness:\n1. The authors propose four strictly proper scoring rules under the condition parameters w are correct. However, unless we know the true distribution. But the true distribution is never known. So the authors need to discuss how hard we can get w in practice. And if we cannot get the w correctly, is using wrongly estimated w based \"strictly proper scoring rule\" still better than using those non-strictly proper ones?\n\n2. In experiments, the authors only compare four new objectives. The authors have not compare with training with the non-proper version of the proposed algorithm. To make the things easier to compare with, the authors can use synthetic distributions, in which they can get the true distribution/true parameters w in the objectives, and the authors test the performance using the strictly proper scoring without estimation loss from w. The authors should also include some common training objectives, for example, likelihood during the comparison.\n\n3. The authors should consider some other common evaluation metrics, for example, concordance. Though concordance is not a proper scoring rule, but it has a practical meaning in survival analysis, i.e., the patient with high risk should be estimated with high risk. This is helpful when allocating medical resources. \n\n4. Closely related to the first point. Are the proposed objectives really helpful in training deep models? Deep models may not find the global optima in optimization. Finding good local optima is more important. So are \"strictly proper scoring rules\" really an important point? Will non-proper rule be better for optimization. The authors do not need to answer this in details. This is a hard topic.\n\n\nMinor:\n1. In introduction, bullet point Brier score, \"show that neither of them are not proper in terms of the theory of scoring rules.\" I think the authors may want only one negation here. maybe remove \"not\".\n2. I think when the authors define four new objectives, they probably need to define with the expectation. For example, the expectation of the Portnoy’s estimator is strictly proper but not the estimator itself.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "UTI_d_lBfy", "reviewer": "Reviewer_rsGQ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper studies how to use proper scoring rules to solve the survival analysis problem, where we want to estimate the distribution F(t) of a random variable T using possibly censored data, which are samples of Z = min{T, C} and whether Z = T or C. While any proper scoring rule can be used as a loss function to train a model (e.g., a neural network) to estimate a distribution using uncensored data, to train on censored data the original proper scoring rule needs to be adapted. The authors adapt four proper scoring rules (Pinball, Logarithmic, Brier, and Ranked Probability Score), and theoretically prove that these adapted socring rules are proper, under the assumption that a parameter w (that depends on the distribution F(t)) is known.\n\nIn practice, however, the parameter w is unknown, so the authors use EM and CQRNN algorithms to choose w and train the model together. Experiments show that the adapted logarithmic scoring rule is the best.", "summary_of_the_review": "I currently recommend weak reject due to the lack of novelty and clarity. But if the authors can provide more contexts, like comparisons with previous works, to demonstrate the novelty of their works, I might change my opinion. ", "strength_and_weaknesses": "[Strength]\n\nSurvival analysis is definitely an important problem to study, and the use of proper scoring rules to solve this problem is also well motivated. The authors prove that the proper scoring rules adapted to the survival problem are proper under some condition. This is an interesting result.\n\n[Weakness]\n\nHowever, I can't really tell the novelty of this work. Just as the authors mention, many works like (Portnoy, 2003), (Pearce et al, 2022) already proposed to use proper scoring rules to solve the survival analysis problem. Although the authors claim to be the first to prove that Portnoy's estimator (which is adapted from Pinball loss) is proper, this conclusion is under the strong assumption that the parameter w (which depends on the unknown distribution F(t)) is known. Under this assumption, the conclusion that Portnoy's estimator (and other three adapted scores) is proper is not surprising. It seems to me that the most challenging part of using scoring rules to solve survival analysis is how to estimate the parameter w, instead of proving properness. The methods to estimate w in this paper, however, basically follow from previous works.\n\nThe relationship with previous works is unclear. Introduction says that Rindt et al (2022) show that variants of Brier score and ranked probability score are not proper, but the variants in this paper are claimed to be proper (if w is known). Are they different variants? Also, did Portnoy (2003) or Pearce et al (2022) already prove that Portnoy's estimator is proper if we know w?\n\n\n[Feedback, not weakness]\n\nIt might help to include some results in Appendix B (additional experiments) in the body, in particular, the comparison between the proper scoring rules in this paper and the custom loss functions in other state-of-the-art neural networks. One motivation for the authors to study properness of scoring rules in the survival analysis problem is because the scoring rules are used as loss functions in training neural networks, where properness is desired. So, the empirical comparison between proper scoring rules and other custom loss functions is worth including.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "BjYN2nqD_X", "reviewer": "Reviewer_PxCN", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work is after finding proper scoring rules in survival analysis. They have a parameter vector w whose true specification underpins the proofs their provide for the discussed scoring rules being proper. They approximate the parameter vector w using an EM algorithm which can in turn be plugged into their scoring rule.", "summary_of_the_review": "I have doubts about the applicability of this work. It is not obvious to me how this work can be applied to a real-world scenario where explanatory covariates exist and the population from which the data is sampled is not homogeneous. The experiments did not seem to counter my current understanding.", "strength_and_weaknesses": "It is not obvious from the paper how explanatory covariates can be integrated. The parameter vector which is central in the correctness of the deduced proofs is distribution dependent. For each covariate combination, the values for w changes. Therefore, I cannot think of applying what this paper proposes in a survival analysis application in which the sample is not thought to be drawn from a homogeneous population.\n\nThe experiments are not shedding any light on what I mentioned above. Table 1 has several metrics for which several loss functions have been tried but the author is left guessing what to take out of it. Since this paper is concerned with survival analysis, it would have been better to show that the metrics that this paper proposes to be proper, when minimized, prove to be better than those normally minimized for which such guarantees do not exist.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject