Censored Quantile Regression Neural Networks for Distribution-Free Survival Analysis
Authors: Tim Pearce
Jong-Hyeon Jeong
yichen jia
Jun Zhu
Venue: NeurIPS 2022 Accept (2022)
Keywords: deep learning
neural networks
survival analysis
quantile regression
censored data

Abstract: This paper considers doing quantile regression on censored data using neural networks (NNs). This adds to the survival analysis toolkit by allowing direct prediction of the target variable, along with a distribution-free characterisation of uncertainty, using a flexible function approximator. We begin by showing how an algorithm popular in linear models can be applied to NNs. However, the resulting procedure is inefficient, requiring sequential optimisation of an individual NN at each desired quantile. Our major contribution is a novel algorithm that simultaneously optimises a grid of quantiles output by a single NN. To offer theoretical insight into our algorithm, we show firstly that it can be interpreted as a form of expectation-maximisation, and secondly that it exhibits a desirable `self-correcting' property. Experimentally, the algorithm produces quantiles that are better calibrated than existing methods on 10 out of 12 real datasets.

Censored Quantile Regression Neural Networks for Distribution-Free Survival Analysis
Tim Pearce Jong-Hyeon Jeong', Yichen Jia Jun Zhu
Dept. of Comp. Sci. & Tech., NRist Center, Tsinghua-Bosch Joint ML Center, Tsinghua University 2 Microsoft Research, $ UUniversity of Pittsburgh
Abstract
This paper considers doing quantile regression on censored data using neural networks (NNs). This adds to the survival analysis toolkit by allowing direct prediction of the target variable, along with a distribution-free characterisation of uncertainty, using flexible function approximator. We begin by showing how an algorithm popular in linear models can be applied to NNs. However, the resulting procedure is inefficient, requiring sequential optimisation of an individual NN at each desired quantile. Our major contribution is a novel algorithm that simultancously optimises grid of quantiles output by a single NN. To offer theoretical insight into our algorithm, we show firstly that it can be interpreted as form of appectationmmaxiissate 'self-correctng'' property. Experimentally, the algorithm produces quantiles that are better calibrated than existing methods on 10 of 12 real datasets.
and secondly that it exhibits desirable
Code:
1 Introduction
Domains such as biomedical sciences and reliability engineering often produce datasets with a particular challenge- for many datapoints the target variable is not directly observed and only a lower or upper bound is recorded. For example, when modelling the time to failure of a machine, in many cases it might have been preemptively maintained before failure event observed, meaning only a lower bound on the time to failure is recorded. This known as censored data, which is studied in the field of survival analysis.
Recently, there has been interest combining ideas from survival analysis with neural networks (NNs), in the hope of leveraging the capabilities of deep learning for this important class of problem. Quantile regression has proven valuable in survival analysis, since it directly predicts the variable of interest, naturally capturing the uncertainty of the conditional distribution with assumptions made about the distribution of residual errors [eng 2021]. Despite a wealth of research the linear setting, explorations into its combination with NNs are at early stage (Jia and Jeong 2022 This paper advances this line of research, showing how an estimator from Portnoyy 2003), used for doing quantile regression in linear models left and right censored data, can be combined with NNs.
Section 2 introduces Portnoy's estimator, and outlines a sequential grid algorithm that used in its optimisation for linear models [Neocleous et al. 2006). As our first contribution, Section 3.1 shows how this algorithm can be directly adapted to work with NNs. Unfortunately, the resulting method inefficient, requiring sequential optimisation of new NN at each predicted quantile. Section 3.2 outlines our main contribution; an improved algorithm that is well suited to NNs, named 'censored quantile regression neural network' (CQRNN). Our CQRNN algorithm is significant departure from
Majority of work completed while Tim Pearce was at Tsinghua University. Correspondance to: tim, pearce micmosoft.com; deszj @etsinghua.edu.cm
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

the sequential approach, and we offer theoretical insights into it in Section + interpreting it as form of expectation-ne expectationmmxximisttton (EM), and also analysing a "selffcorrecting' property. In Section 6 the algorithm's is effectiveness is empirically demonstrated by benchmarking against alternative methods on simulated and real data. Careful ablations illuminate effects of the algorithm's 's hyperparameters.
2 Background
We first introduce censored data. Consider a dataset D = {{11 M/, x; " is the input. y/4 E R is the (possibly censored) variable to regress on, and A; represents the observed/censored indicator. We assume two data generating distributions, one for the target variable, t, ~ p:(()x;), and another for the censoring variable, q ~ Pe(c(x,). In the case of right censoring. we only observe the smaller of these two, which forms our dataset labels, yi = min (4, G), and
UN. Avt}. where
A, =
if ccnsored, <t,, Whilst this paper concentrates on the right censored case, for all estimators else
and algorithms discussed, left censoring can be handled by simply inverting the labels, i.c. using -y; Vi (Koenker 2022].
In this paper make common assumptions about pt(()x;) and pe(c(x;); the target is independent of censoring given the covariates, t -. c x (sometimes termed 'random' censoring), but both t and e depend on x in different and possibly non-linear ways. These are more general than some alternative assumptions, such as using fixed-value censoring, = constant vi (e.g. [Powell [986))
or requiring the censoring distribution be independent of (e.g. (Dia and Jeong 2022).
2.1 Quantile Regression without Censoring
Ignoring censoring for a moment, the conditional quantile function is given by,
(())x)) = (y:. : p(u S |r,+ = =}
(1)
where, T E (0, To learn such function for a single target quantile, quantile regression minimises 'checkmark' loss, pr (-.-) -) IR + R. KKenkee and Bassett 1978..
1 N N
Cebock (0, D,T) =
Σ
Pr (Uis Di,r):
(2)
t=1
Pr (yt, Di.r) - Dr,r) (T - 1l0irr > wil),
(3)
for model parameters, and a prediction, তt, = + (xi, 0), made by some model 1h : :RD - R,
with T[-] as the indicator function. linear model refers to when, Úi.r = 0īx; x,, with, 0, E R.D.
2.2 Portnoy's Censored Quantile Regression Estimator
Directly optimising the loss in Eq. 2 can produce undesirable models when a dataset contains censored observations. Naively ignoring censoring indicators excluding all censored data will, in the general case, induce unfavourable bias [Zhong et al. [222T] The algorithms proposed in this paper re-weighting scheme introduced in Portnoy (0033 that does account for censoring. Define two disjoint sets of indices, one for censored and one for observed datapoints, Scensored and Sobserved. letting, No [Senssoed and No == [Sobsorved. Portnoy's estimator minimises.
C (0, D.
w."" (O..T...y')
Σ Pr (r(, Dt.r) +
Σ
w;pt(yy. (1 (1,)))(('(' 05,r). (4)
While the loss for observed datapoints is unchanged, censored datapoints have been split into two "pseudo" datapoints one at the censored value, and one at some large value, y" >> max; y4 (Section 6.4 discusse y further). Weight is apportioned between each pair of pscudo datapoints by,
9j
Wi = 1 Qi
(5)
where lj is the quantile at which the datapoint was censored, with respect to the target value distribution, i.e. pe(4; > cj x))x)) We use E R. and q R. to denote the vector of all weights and quantiles respectively, indexed as wy and gj- Given these weights. Portnoy's 's estimator has been shown to be analogous to the Kaplan-Meict (KM) estimator [Portnoy] 2003

---

Algorithm Sequential grid algorithm for NNs.
Require: Dataset D, M parametric models er v)(() each with randomly initialised parameters 0,, ordered quantiles to be estimated grid,, learning rate a, pseudo y value y"
Seensored < {i {0, .......NH A, = 07, Sobserved + f {0, 10. N} A, = 13, Keross <- 0 for 2 to M do
grid, [i]
if then
Initialise quantile estimates to zero
q r zeros(N.)
else
& Find indices which have been crossed and update their estimated quantiles K + -j Scensored : (i-(d S Mj n Wgrid, (x;) > M3
q ] grid,
K cross f- K Krrsss U K
q [
Sets w 0 for uncrossed datapoints
w Q) Q)
0 t arg mins, (f., (D,
> Fully optimise Eg.
2.3 Sequential Grid Algorithm
challenge with Portnoy's estimator is that creates circularity. If the true quantiles q and hence for all censored datapoints were known, Eq. couul be optimised straightforwardly. but the very reason to perform this optimisation is to estimate such quantiles! Prior work has tackled this issuc in various ways (see Section 5). Here discuss the most widely-used algorithm. It was originally presented in Portnoy [2003] and slightly modified in Neocleous et al. 2006]. we refer to it as the "sequential grid algorithm'.
Described in Algorithm I it sequentially steps through a grid of M desired quantiles, arranged in strictly increasing order, and typically evenly spaced, e.g., grid, {0.1, 0.2.... 0.8, 0.9} for M = 9. A new model is fitted at each quantile. The algorithm terminates either when all quantiles in the grid have been iterated through, or if only censored datapoints lie above the current quantile. (Portnoy (003)) suggest handling the first quantile specially, but we have simplified this see later.) Note introduce å & w to explicitly designate model estimates of q & w.
CQRNN (ours)
Excl. censor
DeepQuantReg
LogNorm MLE
Norm linear Norm uniform
Geoond
The pardua
canda
Exponential
LogNorm
Figure ID synthetic datasets of varying functions (rows), fitted by various methods (columns). Estimated quantiles (blue through pink) compared to ground truth quantiles (dashed black lines).
CQRNN recovers quantiles closest the ground truth on most datasets.
3

---

3 Censored Quantile Regression and Neural Networks
This section considers applying the sequential grid algorithm to NNs, explaining why this is inefficient. It then describes the CQRNN algorithm, our main contribution, which avoids these pitfalls.
3.1 The Sequential Grid Algorithm and Neural Networks
To the best of our knowledge the sequential grid algorithm has been proposed and implemented only on linear models. Our first contribution showing that, with three adjustments, it can be used in NNs.
Adj. 1) The M models producing estimates, M5.r, are chosen be NNs, of any architecture with a single output. Adj. 2) While the linear version of the algorithm minimises Eq. Al analytically with parametric programming approach, for NNs this is instead minimised with stochastic gradient descent. Adj. 3) Portnoy 2003 proposed using a lengthy process for the first quantile. They set w = 1 and optimise Eq. After optimisation, if any censored datapoints lie below the first quantile, these are removed from the dataset, and the procedure repeats until there are none. This repeated optimisation could go on for many iterations for large datasets and is costly for NNs. Instead, we simply set q = 0, and optimise the first quantile once only. We show this is roughly equivalent in Appendix
Unfortunately, this adaption is still inefficient. Inspecting Algorithm | one notes that M NNs must be trained sequentially, meaning time complexity (both at training and at test) and memory complexity is (MM). This is not the case when doing quantile regression with NNs without censoring a single NN typically uses multiple outputs, one per quantile, and their losses can be minimised simultaneously [Taylor 2000]. For modest numbers of quantiles, this results in little overhead relative to a single NN, since the extra quantile outputs require only linear heads, while the bulk of the parameter count and computation time arise from the shared trunk of the NN.
Note that these inefficiencies are less problematic for linear models optimisation of Eq. 4 can be performed quite rapidly, and memory complexity cannot obviously be improved beyond O(M). As such, it is the of NNs that motivates the search for an alternative algorithm.
3.2 CQRNN Algorithm
Our proposed algorithm uses single NN outputting grid of quantile estimates that can be optimised simultaneously. This results in more efficient algorithm in terms of training time, test time, and memory. Concretely, if the NN is a single hidden-layer NN with neurons, then the model is defined, 1 (x;, 0) OI, <00 xJxx), with On E RD shared for all estimates and 01.- E RR for the specific output head, and non- non-linearity, ø.
The key insight behind our algorithm is that estimates of the censored weights, w, be bootstrapped from the model while it learns, without requiring sequential optimisation. Intuitively, as the model trains, the estimates of the censored quantiles. q, and hence the censored weights, w, improve, and these feedback to create a more accurate loss function, allowing further improvement of the model.
We summarise the method in Algorithm 2 The quantile of a censored datapoint is estimated as whichever NN output is closest under the current model parameters. Whilst the algorithm written in two steps, getting up-to-date estimates of censored quantiles, 9 requires only forward pass, which is done at the point of loss optimisation anyway, so actually comes 'for free". In our implementations, the algorithm is extended to leverage mini-batches and modern optimisers (e.g. Adam).
4 Analysis of CQRNN
The previous section introduced the CQRNN algorithm, motivating it intuitively, while this section considers its correctness. Providing precise analytical guarantees is very challenging, both due the involvement of non-convex NNs, and the bootstrapping nature of the algorithm. We therefore consider more holistic ways of understanding and providing confidence in the method. We firstly show that it can be interpreted as a flavour of EM ('generalised hard' EM) connecting to this well-studied class of algorithms provides some justification of CQRNN's s design. We secondly validate the way the model bootstraps its own censored quantile estimates by describing cerrecoore property.

---

Algorithm 2 CQRNN algorithm.
Require: Dataset D, one parametric model (-()) with randomly initialised parameters 0 that can output M quantile predictions, quantiles to be estimated grid,, learning rate pseudo value y".
Sensored + {i (o.1........ .N. A, = 0}, Sobserved + E {0, 10.1....... N} A; =
while not convereged do
1. Hard expectation step
for j E Scemoored do
dj f- arg min, 10 U/1
Estimate quantile of censored data under current model
új (T 4,) 4;)
2. Purtial maximisation step
abL(0,, T, w. y00 Vr E grid,
Gradient step minimise Eq.
4.1 Interpretation as Expeetation-Maxxiisttinn
EM is two-stage iterative optimisation technique for finding maximum likelihood solutions. It can be useful when one has both observed and latent variables, and where knowing the latent variables would allow straightforward optimisation of model's parameters and vice versa.
In problem, interpret the weights of the censored datapoints, w, latent variables, observed variables as x & Mr and the NN parameters 0 are to be optimised. Under our notation, the vanilla EM procedure BBshhop 2006 be written follows (using randomly initialising (old),
1. Expectation step. Evaluate, p(w x. U, gold), to set up the expectation,
2(0, gold) = -J
p(wpp((pp))) gold log p(y, w/x, 0)dw = w (logply w/x, 0)]
(6)
2. Maximisation step. Perform optimisation of the expected likelihood.
gnew arg max, 2(0, gold)
3. Set gold +- gnew and repeat until convergence.
We now show how CQRNN method in Algorithm 2 can be interpreted as flavour of EM, specifi- cally generalised hard EM. The key steps in this interpretation are: 1) The likelihood for datapoints at each quantile are chosen to follow an asymmetric Laplace distribution. 2) The expectation step is treated as hard assignment under the current NN parameters. 3) The likelihood is only partially maximised each iteration.
Likelihood Form
A likelihood function is required for both the E & M steps. Theorem I shows how Eq. interpreted negative log likelihood.
can be
Theorem D
Let the likelihood for each datapoint at each quantile be an asymmetric Laplace distribution with scale, = V and asymmetry. l == T/VT 72, The negative log likelihood
is,
log p w,))
Σ rEgrid,
Cpen (0. w. constant.
(8)
Proof sketch. The asymmetric Laplace distribution has been used in Bayesian treatments of quantile regression without censoring (Yu and Moyeed 2001). We extend this Portnoy's s loss with weighted likelihood form using the censored weights. Appendix A provides the full proof.
Expectation as a Hard Assignment of Latent Variables
In some models, such as a Gaussian mixture model, the expectation step can be computed through analytical evaluation of the latent posterior distribution [Bishop| 2006). In our case, p( w xx, y. 6old).
5

---

is intractable, though Eq.
depends linearly on each element w so we need only consider the The algorithm can then be interpreted as making an assignment under
expectation, Bu,, x.y. w the current model parameters,
Σ wyp( (en( ,,, 60ld) = T -- 93
Emg by puolee (uyy) =
(9)
1 "
7-9, if wy = 1 where 13 arg min, 1ū Vj 0 else
p(w,|2,, Xz,, Bj. Bold)
(10)
This is a 'hard' assignment, where latents are assigned to the most likely values under the current model (e.g. datapoints are attributed to the nearest clusters in the K-means algorithm [Bishop] 2006)).
and gives rise to a class of algorithms termed hard EM (Samdani et al. 2012.
Note that the in Eq. 6 we can rewrite, log p(u, w|x, 0) log ( 0, w). Since we choose only a single setting for in the outer expectation, this dependence disappears, and we require maximisation of, IE 0.y [logp(y))s 0, w)]-
Partial Maximisation
second departure from the standard EM algorithm, is that Algorithm 2 performs only a partial maximisation of the likelihood, taking single gradient step. This has been shown to provide similar guarantees, and is termed "generalised' EM (Neal and Hinton 1998].
One could consider an alternative version of the algorithm that is closer to the standard EM procedure, where the estimated quantiles of censored data points are fixed while the maximisation step of NN parameters is run to convergence over multiple training epochs, before the estimates are updated in the expectation step, and repeating. In our case this is less effective - the maximisation requires running forward pass through the NN, and obtaining estimated quantiles under current NN parameters once this is done is trivial, so we can access up-to-dat to-d estimates essentially for free. Appendix Figure 5 empirically demonstrates that CQRNN's partial maximisation approach produces the fastest convergence.
4.2 Self-Correcting Property
The CQRNN bootstraps weight estimates from the current model as it trains. It's not immediately obvious why this bootstrapping approach should converge to something sensible what if bad initial weight estimates lead to worse ones? As second insight into the CQRNN algorithm, Theorem 2 shows that when a censored weight, ú.j is estimated incorrectly, the algorithm acts in a way to adjust the estimated quantiles in a favourable way - we refer to this as 'self seolf-correcting.
Theorem D If q is underestimated, one iteration of the algorithm acts to increase the quantile predictions, Ui.r, by the same amount, or even higher, than the weight had been correct. If â; is overestimated, Di.r. is decreased by the same amount, or even lower, than with the correct weight.
Proof sketch. Denote qj the true quantile that censored datapoint j is censored in, and wy the corresponding true weight. We derive the expression for the gradient wrt the predicted quantiles, D5.7. finding that if új is underestimated it holds that, &C D.r.w.y))
and hence gradient descent applies an adjustment in the desired direction of equal or greater magnitude than if the true weight had been used. The reverse holds for overestimated vō.. Appendix proviides a full proof.
(9,
89j.
005.-
Related Work
Survival analysis and NNs. Following the widespread success of deep NNs over the past decade, there has been a wave of research applying NNs to survival analysis for instance by modifying the CoxPH model [atzzman et al. 2018). or framing the task as ordinal classification [Lee et al. 2018). Closer to our work are methods that use NNs to output parameters of distribution such as the Weibull [Martinsson] 2016), seek robust training objectives for these models [Avati al.
2019) or help with their optimisation (Tng al. 2022
Our work stands out as offering way to

---

directly estimate the target variable at pre-specified quantiles, without enforcing any distributional assumption. A limitation is that the distribution may only be predicted these quantiles (unless additional assumptions are made to allow interpolation between these).
Quantile regression and NNs. Quantile regression has proven an attractive option to enable NNs to move beyond point predictions. This allows quantification of a NN's alcatoric uncertainty (Tagasovska and Lopez-Paz Paz 20191. It is attractive due to its straightforward implementation, and avoidance of any distributional assumption. For example, it has found use in reinforcement learning to capture the distribution of rewards, rather than just the mean [Dabney et al., 2017]. Since NNs are flexible function approximators, particular attention has been paid to the crossing quantile problem [Bondell] ct al.) 2010 Zhou al.) 2020 Brando et al.) 2022]. CQRNN borrows ideas from this line of work, and further tackles the challenge of learning quantiles under censored data.
Censored quantile linear regression. There is much work censored quantile regression methods for linear models. [Pwwll] [1986] developed an estimator under fixed-valur censoring which can be implemented with an algorithm from Fitzenberger [1997]. Portnoy] [2003] developed an estimator under random censoring based on the KM estimator, while Peng and Huang [2008] developed an alternative based on the Nelson-Aalen (NA) estimator. KM and NA are closely related, and Portnoy and Peng's ' methods have been reported to offer similar empirical performance [Koenker] 2008
Koenker [2022] provides all above methods in the popular 'quantreg' R package. Other notable methods include Yang et al. [2018). based on the data augmentation algorithm, and Wang and Wang 2009), whose estimator is similar to Portnoy's 's but utilises local estimates of the KM, computed with kernel method. See [Pegg] 2021 for review of the broader area. Our work allows modelling of flexible non-linear quantile functions, leveraging the powerful representation learning abilities of NNs. Although, this sacrifices the interpretability of coefficients of linear models.
Censored quantile regression and NNs. Only a small amount of work has been done in this area. The 'qrnn' R package [annon 2019 offers the ability to train NNs under fixed-value left censoring, adopting an idea from the linear setting [Friederichs and Hense 2007, Cannon 201T]. Huttel et al.) [2022] explore our objective but assume censoring times, Cs are available for both censored and uncensored data points (i.e. the censoring distribution is known). We do not require this assumption. DeepQuantReg (Jia and Jeong, 2022] tackles the same objective as this paper. They showed that improvements in quantile estimation can be obtained relative to naive methods. Their work differs from ours significantly- they base their method around an estimator for the median from Huang et al. 2007 requiring an assumption that the censoring distribution is independent of covariates.
6 Experiments
This section empirically investigates several questions. Q1) How does the proposed CQRNN method compare with existing methods? This is done qualitatively on synthetic 1D functions in Section 6. and quantitatively synthetic and real datasets in Section 6.2 Q2) How does the sequential grid algorithu compare to the CQRNN algorithm, both in terms of predictive accuracy and efficiency?
Explored in Section 6.3 Q3) How is the CQRNN algorithm affected by its hyperparameters? We
investigate the impact of grid fidelity and y in Section 6.4
All experiments use dully-connected NNs with two hidden layers of 100 neurons, except for SurvM- NIST, when three convolutional layers are used. Grid size M set to 5, 9 or depending on dataset size. Appendix B contains further details on hyperparameter settings, metrics, and datasets. Appendix presents further results.
Datasets. We use three types of dataset. Type 1) Synthetic target data with synthetic censoring. Type 2) Real target data with synthetic censoring. Type 3) Real target data with real censoring. Whilst type 3 captures the kind of datasets we care about most, evaluation of quantiles is challenging since in real-world survival data the target conditional quantiles are not obtainable even at test time (Li and Peng] 2017). In contrast, type 1 offers access to the ground truth quantiles for clean evaluation, but properties of these datasets may be less realistic. We introduce type 2 as a middle ground, which takes a real-world dataset without censoring, and synthetically overlays a censoring distribution to create training data. At test time, we have the option of not applying the censoring, providing samples from the clean target distribution, allowing clearer evaluation. We summarise all datasets in Table 2 Full details are in Appendix [B.4] Our experiments exceed the number and variety of
datasets used in popular recent works Goldstein et al. 2020 Zhong et al. 2021
7

---

Table 1: Performance of our proposed CQRNN algorithm (Algorithm compared to the sequential grid method for NNs (Algorithm D. over 200 random seeds.
Number quantiles
Training time
Test time speed up
TOMSE difference
CQRNN sig. better in TQMSE?
Seq, statistical
speed up
saving
Seq. grid CORNN
sig. betler in TOMSE?
significntt difference
95% conf. interval
Norm non-lin
9
12.4%
9.6x
8.3x
-0.003 0.001
Exponential
9
12.4%
8.0x
8.3x
-0.093 0.176
Weibull
-0.014 0.016
Loggoom
12.9
-0.039 0.028
Norm uniferm
9
12.6%
8.1%
8.3X
0.175 = 0.035
Norm heavy
31,0%
18.5%
16.2x
0.108 0.211
med.
17.9%
-0.44 0.014
Norm light
19
31.7%
19.2%
16.2x
-0.035 + 0.003
Norm same
34.3%
21.2%
16.2X
-0.390 0.049
LogNorm heavy
19.9%
LogNorm med.
19
31.3%
13.1%
16.2%
0.019 + 0,002
LogNorm light
31.3%
20.2%
16.2%
40.033 0.004
LogNorm
18.4%
Total:
8/14 3/14 3/14
Metries. Our objective is to measure how closely model's predicted quantiles match those of the ground truth target distribution. We favour different metrics for each dataset type.
For type datasets, since targets are generated synthetically it is possible to analytically compute the ground truth target quantile for input x,, which we denote Vi.r. We compute the mean squared error (MSE) between the predictions and the ground truths across three quantiles, 7 E [0.1, 0.5, 0.9). (we ensure these always present in grid,). This is our first-choice metric when available. True quantile MSE (TQMSE) == $/ Exejo.10.5.00)) ( - -..r)
In type 2 datsets samples from the uncensored target distribution available but not the syn- thetic generating function. Our preferred metric is the checkmark loss across the three quantiles. Uncensored quantile loss (UQL) = N EE Σ Pr (y Dt,r).
For datasets of type 3, we use two metrics. The concordance index (C-index) is computed using the median (r 0.5). But this may not reveal anything about systematic bias of different models. nor about other quantiles т * 0.5. We secondly use censored D-Calibration (CensDCal) (Haider et al., 2020) which measures whether the empirical proportion of datapoints falling between pairs of consecutive quantiles, matches the target proportion, Tj+1 Ty. MSE of the deviation is then computed. There is also an uncensored version (UnDCal)), which we can compute for type datasets. Appendix B. gives further details.
Baselines. We compare against three methods that can be used predict the quantiles of target distribution using a NN. Excl. Censor a method that naively excludes censored datapoints from the training data, optimising the loss Eq. B DeepQuantReg the only existing method in the literature proposing explicit output of quantiles from NN on censored data (Tia and Jeong 2022). LogNorm MLE- NN outputting parameters of lognormal distribution. that's trained via maximum likelihood estimation (MLE) (details in Appendix [B3).. and is a standard baseline in related work (e.g. (Avti al. 2019 Goldstein et al., 2020) since this distribution often suits
properties of real-world time-to-event survival data KKleinbum and Klein 2012
6.1 Qualitative 1D Analysis
Figure I visualisee the quantiles predicted by CQRNN and baseline methods for ID datasets (Table 2 describes dataset functions, Figure visualises further ID datasets). Each dataset contains 500
datapoints drawn from, x ~ (0. 2) and grid, E {0.1, 0.3, 0.5, 0.7, 0.9). CQRNN usually learns quantiles that are closer the ground truth than baseline methods, particularly at higher quantiles.
Excl. censor systemically makes underpredictions, and this worsens at higher quantiles since larger values of y are more likely to be censored and excluded. DeepQuantReg avoids this systemic underprediction, but appears to introduce bias of its own. LogNorm MLE provides variable results on Norm uniform it fails with excessive variance (an issue also observed by Avati et al. (209)), but on LogNorm, since the target distribution matches the distribution output by the NN, it performs well.

---

6.2 Benchmarking
Type datasets
Our main experiment benchmarks CQRNN
TONSE better)
against all baselines across wide variety of datasets, covering various domains, sizes, di- mensionalities and censoring proportions (see Table 2. Some hyperparameter tuning was per- formed for each method (Appendix B. Figure plott the preferred metric for each dataset type, though patterns were consistent across metrics
CQRNN (curs)
Eud.
1gp 10-4 10
DespQuanRRce Loghorm MLE
Exponercal
LogYor
TOMSE (bwer betted
Table provides a full breakdown.
tanvy
med
light
N come
Light he leg'l med Ligh light ogll come
For type I (synthetic) datasets, the quantitative results follow our qualitative observations, with
Type 2 datasets
CQRNN producing lowest TQMSE all but the datasets generated from Log Normal distri- bution, when sometimes placed second behind LogNorm MLE.
UCL
Hossing
CQRNN also produces the lowest UQL on all datasets type 2. LogNorm MLE performs
Type datasets
poorly these, presumably since they are not typical time-to-event datasets which are well captured by the Log Normal distribution.
6.8 0.0
Cinde (eaher
For type 3 datasets, CQRNN and LogNorm MLE usually perform best in terms of C-index, with error bars tending to overlap. In CensDCal, CQRNN consistently perform best, matched only by LogNorm MLE on two datasets, when error bars overlap. Excl. censor is the weakest method, with DeepQuantReg midway between it and CQRNN.
CruDCCll (loser better
METABRIC
WHAS
SUPPORT
GBSC
TMB
BesiMSK
LOCCBM
Figure 2: Main benchmarking results. Mean + one standard error over ten runs.
6.3 Comparison of Sequential Grid and CQRNN
We ran head-to-head comparison of CQRNN and the sequential grid algorithm all type synthetic datasets, and all type 3 real datasets. Table T compares on type datasets, along with statistical tests of significance for TQMSE differences (200 random seeds). Appendix Table provides results for the type datsets, reporting C-index and CensDCal (50 random seeds). Significance tests are described in Appendix B.
CQRNN delivers benefits speed up both at training and test time, running an order of magnitude faster than the sequential grid algorithm. CQRNN also dramatically reduces model size. The magnitude of these benefits is largely determined by the size of the quantile grid, M, which is explicit in the sequential algorithm's 's time and space complexity. O(M), but largely avoided in CQRNN.
Differences in the quality of quantiles of the two algorithms usually slight (after-all both leverage the same estimator), though CQRNN shows statistically significant gains on 8/14 type 1 datasets in terms of TQMSE, 4/7 type 3 datsets in terms C-Index, and 7/7 type datasets in terms CensDCal. We hypothesise that having single NN output all quantiles provides a helpful inductive bias, encouraging similarity between adjacent quantiles, that's 's not present in independently trained NNs.
6.4 Hyperparameter Investigation
The CQRNN algorithm includes two hyperparameters - the grid of quantiles, grid,, and the pseudo y value, y". We ran ablations to empirically investigate the effect of these, as well as remedies for the "crossing "crossing-quantile problem, and a comparison of partial vs. full optimisation. Here we summarise findings, Appendix C provides full details.
We tested grid sizes, M E (9, 19, 39), on several type datasets. general, finer grid (larger M) is slightly beneficial, though there is variance between datasets and gains sometimes only seen

---

for larger datasets (>5,000 datapoints). Selection of y" requires some care for CQRNN. We defined it in terms of the maximum y value in the training set, y cy- max, yı. for a hyperpuremeter. Cy- >> 1. Performance can be improved by tuning Cy- for each dataset, but using Cy- = 1.2 provided consistently reasonable results this was the value used in our benchmarking experiments. We trialled two methods for combating the crossing-quantil problem. 1) Adding crossing loss penalty. 2) Constraining the NN architecture to oulput quantiles adding to the previous prediction. Neither of these methods significantly affected performance.
7 Discussion & Conclusion
This paper has taken a popular idea from survival analysis, Portnoy's 's censored quantile regression estimator, and shown how it can be efficiently combined with NÑs in a new algorithm, CQRNN, We provided theoretical insight by interpreting it a flavour of EM. Empirically the method outperformed existing approaches, consistently producing more accurate quantile estimates across a range of synthetic and real datasets. For example, across datasets of type 2 and 3, CQRNN was best calibrated in 10 out of 12 instances (see CensDCal in Table 4.
Limitations. Firstly, our theoretical results have not said that solutions will converge on global optimum. Secondly, we have only tested CQRNN on a modest number of drawn from the biomedical domain. It's possible that some datasets may cause CQRNN problems, as we found on BreastMSK (Figure 2 In particular, cases where higher quantiles are undefined should be handled with care.
datasets, many
Conclusion. To summarise, our work contributes toward unlocking the benefit that modern machine learning could bring to important domains such as healthcare and machinery prognostics. By outputting quantiles, CQRNN naturally communicates a measure of uncertainty in its predictions, which makes it particularly suitable to these high-stakes applications, and valuable addition to the toolkit combining deep NNs with survival analysis.
Table 2: Summary of all datasets used.
Ditrect
Feats
Train data
Test data
Prop. censored
Target sampling distibaatio
Consering sampling
distributionn
Type 1 Synthetic datasets with syathetlc censoring
Norm lincar
0.20
2/ (a+ 1)")
(dr - (0.8x - 0.4)")
Norm non lincar
500
1000
0.24
Aija sin) + 10, (0.5x + 0.5)²)
(22r + 10,
Esponential
500
1000
0.30
Exp(2z + 4)
Exp 33x + 15)
Weibull
500
1000
0.22
Weibull 2( 1) 10,
Wei 20,
LogNorm
Lognorta(( 1).
2/(0, 10)
Norm uniform
500
1000
0.62
Af e0s(2x) 13, (22 0.51%)
14(0. 18)
Norm heavy
4
2000
1000
0.80
/Bkka xf 23 2sin((zgtt)) G2 - (.j)")
24(0. 12)
Norm med.
4
2000
0.49
Mi0, 21)
Norm light
2000
Norm same
4
2000
1000
0.50
Equal to target dist.
Logmorm heavy
4100
1000
0.75
Legnormn 11/10
14(0, 0.4) 2400) 1.0)
LogNNorm mod.
4000
1000
0.52
I Norm light
8
4100
1000
0:23
24 3.5)
L LogNorm same
8
4000
1000
0.50
Equal to target dist.
Type
2
Real datusets with synthetic censoring
Housing
Real
24(0,a;)
Protein
3658-
9116
0.11
Real
2400, ea)
Wine
11
5197
1300
0.69
Real
24(0.ca)
PHE
36734
9184
0.52
Real
2410,
Sur-MNIST
28
12000
Gammna dist. per MNIST
Type 3
Real datasets with real censoring
METABRIC
9
1523
381
0.42
Real
Real
WHAS
6
1310
328
0:57
Real
Real
SUPPORT
7098
Real
Real
GBSG
7
1785
447
0.42
Real
Real
TMBImmuno
3
1928
332
0.49
Real
Real
BreastMSK
5
1467
367
0.77
Real
Real
LGGGBM
5
510
128
0.60
Real
Real
Acknowledgement
This work was supported by the National Key Research and Development Program of China 2020AAA0106002), NSF of China Projects (Nos. 62061 136011, U19B2034, U1811161,, U19A2081] 1); Beijing NSF Project (No. JQ19016); grant from Tsinghua Institute for Guo Qiang: and the High Performance Computing Center, Tsinghua University. J.Z. was also supported by the XPlorer Prize.
10

---

References
Anand Avati, Tony Duan, Sharon Zhou, Kenneth Jung, Nigam H. Shah, and Andrew Y. Ng. Count- down regression: Sharp and calibrated survival predictions. 35th Conference on Uncertainty in Artificial Intelligence, UAI 2019, 2019.
Christopher M Bishop. Pattern Recognition and Machine Learning. Springer, 2006. ISBN 1788578110796. doi: 00007//0097811041133333
Howard D. Bondell, Brian J. Reich, and Huixia Wang. Noncrossing quantile regression curve estimation. Biometrika, 97:825-838, 2010. ISSN 00063444. doi: 10. 109//iomme//ss4448...
Axel Brando, Joan Gimeno, José Rodríguez-Serrano, and Jordi Vitri. Deep non-crossing quantiles through the partial derivative. AISTATS, 2022.
Alex J. Cannon. Quantile regression neural networks: Implementation in and application to
precipitation downscaling. Computers and Geosciences, 2011. ISSN 00983004. doi: 10.1016/j. cageo.2010.07.005
Alex Cannon. Quantile regression neural network 'qrnn'. Package, 2019, doi: 10.1016/j.cagen 2010.07.005>and.
Will Dabney, Mark Rowland, Marc G. Bellemare, and Rémi Munos. Distributional reinforcement learning with quantile regression. AAAI, 10 2017. URLp:tt///arxii http://aaxiiv..g/aarr/s//ssa wwwpg/abs/771..... 10044
Bernd Fitzenberger. A guide to censored quantile regressions. Handbook of Statistics, 15:405-44 437, 1997. ISSN 01697161. doi: 10.0016/001699-716(9911111)0
P. Friederichs and A. Hense. Statistical downscaling of extreme precipitation events using censored quantile regression. Monthly Weather Review, 135:2365-2378 2007. ISSN 0027-0644. doi: 10. 1175/MWR3403.1. URL htttp:t////ourraaa ametsoc ptrg//do/abb////e 10. 1175//MR55/33 1
Mark Goldstein, Xintian Han, Aahlad Puli, Adler J. Perotte, and Rajesh Ranganath. X-cal: Explicit calibration for survival analysis. Advances in Neural Information Processing Systems, 2020. ISSN 10495258.
Humza Haider, Bret Hoehn, Sarah Davis, and Russell Greiner. Effective ways to build and evaluate individual survival distributions. Journal of Machine Learning Research, 21:1-63, 2020. ISSN 15337928.
Jian Huang, Shuangge Ma, and Huiliang Xie. Least absolute deviations estimation for the accelerated failure time model. Statistica, Sinica, pages 1-23, 2007.
Frederik Boe Huttel, Inon Peled, Filipe Rodrigues, and Francisco C. Pereira. Modeling censored mobility demand through censored quantile regression neural networks. IEEE Transactions on Intelligent Transportation Systems, 11 2022. ISSN 15580016. doi: 10.1109/TITS.2022 20T.:T.S..22..1111999 3190194.
Yichen Jia and Jong Hyeon Jeong. Deep learning for quantile regression under right censoring: Deepquantreg. Computational Statistics and Data Analysis, 165, 2022. ISSN 01679473. doi:
URL battps://doft htt.///ioooorr///// www.oorgo/o.0. 1016/j csda. 2021. 107323
Jared I.. Katzman, Uri Shaham, Alexander Cloninger, Jonathan Bates, Tingting Jiang, and Yuval Kluger. Deepsurv: Personalized treatment recommender system using a cox proportional hazards deep neural network. BMC Medical Research Methodology, 18:1-15, 2018. ISSN 14712288. doi: 10. 10.1188//228744-18844448
David Kleinbaum and Mitchel Klein. Survival Analysis: A Self-Learning Text. Springer, third edition, 2012.
Roger Koenker. Censored quantile regression redux. Journal of Statistical Software, 27:1-25, 2008. ISSN 15487660. doi: 10. 86637/js.v0227.1 v027.i06.
Roger Koenker. Quantile regression, 'quantreg', R Package, 2022. doi: 10. 10.1017/CB899985511550033
Roger Koenker and Gilbert Bassett. Regression quantiles. Econometrica, 46:33-50, 1978.

---

Changhee Lee, William R Zame. Jinsung Yoon, and Mihaela Van Der Schaar.
Deephit: A URL http
deep learning approach survival analysis with competing risks. AAAI, 2018.
//medianetlab.. ee. ucla. edu wwwpppdppppppaaaaaaaa AAAAI AAAI 2018 DeepHit
Ruosha Li and Limin Peng. Assessing quantile prediction with censored quantile regression models. Biometrics., 2017. doi: 10.1111/biom. 126.m...22......m www.1.22sesssiiiii Assessing.
Egil Martinsson. Wtte-rnn: : Weibull time to event recurrent neural network. Master's Thesis, 2016.
Radford M. Neal and Geoffrey E. Hinton. A view of the em algorithm that justifies incremental, sparse, and other variants. Learning in Graphical Models, 1998.
Tereza Neocleous, Karlien Vanden Branden, and Stephen Portnoy, Correction to censored regression Journal of the American Statistical Association, doi: 10. 1e./199//1221409 4500000000000000000880
quantiles by portnoy. 98 (2003), 1001-1012. 101:860-861, 2006.
ISSN 1537274X.
Limin Peng. Quantile regression for survival data. Annual Review of Statistics and Its Application, 413 437, 2021. ISSN 2326831X. IX. doi: 10.1 1144//nuree-sstatiiiia stttsssiiiiss-022200------ 042720-020233..
Limin Peng and Yijian Huang. Survival analysis with quantile regression models. Journal of the American Statistical Association, (03:637-649, 2008. ISSN 01621459. doi: 10.1198/ 0162 01621450800000335.
Stephen Portnoy. Censored regression quantiles. Journal of the American Statistical Association, 98: 1001-1012, 2003. URL http::/www. jswor..www...................... http:///...../b/0000300/t/////a. org/terms
James I.. Powell. Censored regression quantiles. Journal of Econometrics, 32:143-155, 1986. ISSN 03044076. doi: 10.101660000044447666600001666
Rajhans Samdani, Ming Wei Chang, and Dan Roth. Unified expectation maximization. NAACL HLT 2012 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Proceedings of the Conference, 2012.
Natasa Tagasovska and David Lopez-Paz. Single-model uncertainties for deep learning. NeurlPS, 2019. /ttp:////// www.rrg//aaaa/aaa aaaaaaaa///111 http://wwwww.rrr/aaa///11.11..00 00908
Weijing Tang, Jiaqi Ma, Qiaozhu Mei, and Ji Zhu. Soden: A scalable continuous-time survival surviva model through ordinary differential equation networks. JMLR, 8 2022. URL http:://rraxxi.or//abbb 8008.0863
James W Taylor. A quantile regression neural network approach to estimating the conditional density of multiperiod returns. Journal of Forecasting, 19:299-311 2000.
Huixia Judy Wang and Lan Wang. Locally weighted censored quantile regression. Journal of the American Statistical Association, 114:1117-1128.0 2009. ISSN 01621459. doi: 10. 1988/aaaaaa tm08230. httphttss: htt:://ii.orgr//// 10. 1198/ jasa. 2009. tm08230
Xiaorong Yang, Naveen Naidu Narisetty, and Xuming He. A new approach to censored quantile regression estimation. Journal of Computational and Graphical Statistics, 27:417-425, 2018. ISSN 15372715. doi:
Keming Yu and Rana A Moyeed. Bayesian quantile regression. Statistics Probability Letters, 54: 437-447, 2001.
Qixian Zhong, Jonas Mueller, and Jane-Ling Wang. Deep extended hazard models for survival analysis. NeurIPS, 2021.
Fan Zhou, Jianing Wang, and Xingdong Feng. Non-crossing quantile regression for deep reinforce- ment learning. NeurIPS, 2020.
12


Reviews:
[{"review_id": "ywvkrYAnsf", "reviewer": "Reviewer_doRh", "summary": "This paper proposes a neural network based approach for doing quantile regression on possibly censored data, i.e. when the target variable is sometimes not directly observed and instead a lower or upper bound is known, as in the case of survival data.\nThe authors first build on the linear approach on Portnoy 2003 and extend it straightforwardly to neural networks. However this approach requires the sequential optimization of a new NN for each quantile level to be predicted, which is computationally heavy.\nThen a new method is proposed for simultaneous quantile regression , which is interpreted as a form of expectation maximization, allowing to estimate all the quantiles simultaneously.\n", "questions": "I think it would be interesting to discuss and relate this work to the recent literature on simultaneous quantile regression with NN that alleviates or avoids the crossing quantile problem. In particular, it would be interesting to discuss how these methods could be adapted to censored data using your construction.\n\nA non-exhaustive list:\n- *Tagasovska, N., & Lopez-Paz, D. (2019). Single-model uncertainties for deep learning. Advances in Neural Information Processing Systems, 32.*\n- *Zhou, F., Wang, J., & Feng, X. (2020). Non-crossing quantile regression for distributional reinforcement learning. Advances in Neural Information Processing Systems, 33, 15909-15919.*\n- *Brando, A., Center, B. S., Rodriguez-Serrano, J., & Vitria, J. (2022, May). Deep Non-Crossing Quantiles through the Partial Derivative. In International Conference on Artificial Intelligence and Statistics (pp. 7902-7914). PMLR.*\n\nMinor comments:\nAlgorithm 1: since $\\tau \\leftarrow grid_\\tau[i-1]$,  $\\hat{\\boldsymbol{q}}[K]$ and $\\hat{\\boldsymbol{q}}[\\neg K_{cross}]$ are assigned the same value, is this correct?\n\nAlgorithm X, Equation X, Appendix X,… should start with a capital letter\n\nTo avoid confusion, I think it is better to say quantile level for $q_j$\n\nLines 187-191: $\\bar{w}_j$ is the true weight, $\\hat{w}_j$ is the estimate, what is $w_j$ then?\n\nLine 324: which were the methods use to combat the crossing-quantile problem?\n\n", "limitations": "Yes", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The paper is clearly written and the background on censored data and survival analysis is properly explained.\n- The analysis gives interesting insights and useful guarantees on the algorithm\n- The experiments are interesting and appropriate.\n\nWeakness:\n- Lacks a discussion on the recent literature on simultaneous quantile regression with NN\n\n\n \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "hTX64kd6KXh", "reviewer": "Reviewer_mH2a", "summary": "The authors proposed a neural network approach for quantile regression on censored data, based on Portnoys's estimator. The authors also proposed a new algorithm to mitigate the computational challenge when combining neural networks and the existing sequential grid algorithm for Portnoys's estimator. In addition, the authors made the connection between the proposed CQRNN algorithm and the EM algorithm for further understanding and compared CQRNN with existing methods on both synthetic and real-world datasets.", "questions": "Please see the weaknesses above.", "limitations": "Yes, the authors discussed the limitations.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* The proposed method is not just replacing the linear model with a neural network in the existing algorithm for Portnoy's estimator. They did identify new computational challenges when optimizing a neural network through the existing sequential grid algorithm, which was less problematic for linear models, and proposed a new CQRNN algorithm that achieves faster speeds and saves memory (while the improvement mainly depends on the size of the quantile grid).\n* The connection to the EM algorithm under certain distribution assumptions and the self-correcting property are very interesting. I have a related question below.\n* The authors provided solid numerical evaluations and comparisons with existing approaches both qualitatively and quantitatively, and in terms of both prediction accuracy and computation efficiency. \n\n\nWeaknesses:\n* It would be worthwhile to add more discussion in \"related work\" so as to explicitly highlight the improvement or limitation in comparison to other existing methods.\n* For partial maximization, the authors mentioned the alternative version that is closer to the standard EM algorithm. While in each iteration the proposed algorithm saves computational costs by taking a single gradient step, the number of iterations required for convergence might also change or even increase. Do the authors have any comments on this or have the authors investigated the impact of taking a single gradient instead of multiple gradient descents on the overall computing time/convergence?\n* From Table 2, the censoring distribution in LogNorm (and Norm uniform, etc.) in synthetic datasets seems to be independent of covariates. In this case, the assumption in DeepQuantReg also holds while CQRNN still performs better than DeepQuantReg. Can the authors explain this? \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "3VrjEXkhPja", "reviewer": "Reviewer_qGAg", "summary": "The authors use neural networks to replace the linear models in censored quantile regression. And the authors further improve the sequential grid algorithm with bootstrap weights. The authors explain the soundness of their method from an analogy of EM. The authors compare their method with three baselines on synthetic and real dataset.", "questions": "1. Can we get a more concrete guarantee on the convergence of the propose algorithm? Some guarantee when the model is linear should also be good.\n2. Quantile regression seems to give us quantiles for certain $\\tau$'s . But for a valid survival model, we need the whole distribution. How do we get the whole distribution? Or why do we not need to whole distribution in survival modeling? In Portnoy (2003), $\\tau$ seems to be part of the function so we do not infinite amount of $\\tau$'s. But in the authors' setting, for a different $\\tau$, we need a different model. This requires the infinite amount of $\\tau$'s to get the whole distribution.\n3. Why do authors choose lognormal model as the baseline for survival modeling?", "limitations": "One extra limitation seems to be that CQRNN could not get the full survival distribution.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The method is novel. Introducing nn into quantile regression and the new bootstrap algorithm are both contributions to the society.\n2. The authors make some effort to explain why and how the bootstrap algorithm CQRNN works.\n3. The writing is clear. It's easy for me to follow the paper.\n\nWeakness:\n1. I appreciate that the authors explain how CQRNN work from an EM analogy and self-correcting property. But they still sound vague to me. Can we have a simpler example to understand it in a more rigorous way? For example, if the model is linear, how the convergence property is like using the CQRNN bootstrap algorithm? \n2. The empirical comparison does not seem to be thorough if the authors want to prove that their model is a good \"survival model\".\n- On synthetic datasets, the authors only compare the performance only on several $\\tau$'s (0.1, 0.5, 0.9). But for a survival model, why do we only care about certain $\\tau$'s? How do we get values at other $\\tau$'s that the quantile regression has not modeled? For a survival model, quantile at certain $\\tau$'s is not the only thing we care about. We may need to look at the whole survival distribution. And then we compare other scores, for example, Brier score.\n- The authors only consider lognormal model as their baseline and the reason is that they believe \"this distribution often suits properties of real-world time-to-event survival data.\" Why do you believe so? As far as I know, lognormal models have strict distributional assumptions and may not have good performance. It is also hard to train due to the exp transformation from normal. Other models like DeepHit, SODEN may have better performance on concordance or calibration.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "-TJhni2iFfj", "reviewer": "Reviewer_57zr", "summary": "The paper considers the task of quantile regression in a survival analysis context, i.e. one where some examples are right-censored. As in almost all recent works, the encoder backbone is a neural network. The authors start by adapting the sequential grid (SG) algorithm to work with neural networks, as opposed to strictly linear models. Noting the inneficiency of the adaptation, they propose CQRNN, a novel algorithm which is more efficient. CQRNN departs from SG in that the assignment weights are updated during training.\nIn addition to introducing their proposed algorithm and presenting experimental results jusitying it's adoption, the authors provide a qualitative discussion of their algorithm. In particular, they frame it as an Expectation-Maximization algorithm, and provide a proof sketch as to why their proposed solution should result in \"sensible\" weight allocations.", "questions": "- Could the authors re-do the comparison in table 1? I would like to confirm whether the purported advantage for their method is statistically significant.\n- Would it make sense to adapt classical survival models (the authors cite a few in their work) to this context by simply changing the target to be the quantile center? I am curious what would be the relative performance of these approaches: quite a few are very strong in their original field.\n\nEdit: The authors have responded to my questions. Taking this into account, I am raising my score.", "limitations": "I don't see any potential negative societal impact for this work. If anything, survival analysis is a field that has the potential to *positively* impact medicine.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I. Strenghts\n- The paper is well-written, and reasonably clear. The discussion is easy to follow, and the relevant prior works necessary to understand the novel approach are made explicit.\n- The algorithm is clearly presented (good explanations + an explicit algorithm).\n- Experiments are well detailed: hyperparameter optimization of the competing methods was done (to a certain extent).\n\nII. Weaknesses.\n- As the authors note, the performance gains over vanilla sequential grid are minor (I am referring to table 1). In particular, it is hard to tell whether they are statistically significant without error bars or similar information. As the difference between the two is a major point (otherwise the contribution is incremental), the question I ask in the next section has the goal to clarify this point.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept