SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Given an equivalent kernel with degenerate form, the evaluation of the predictive variance,  $\sigma = h(\pmb{y},\pmb{y}) - h(\pmb{y})^{\top}(\pmb{Z} + \pmb{H})^{-1}\pmb{h}(\pmb{y})$ , needs the computation of  $\mathcal{O}(M^3 +NM^2)$ :  $N\times N$  matrix  $\pmb{H}$  can be decomposed into a product of  $N\times M$  matrix  $\pmb{R}$  and its transpose as  $\pmb{H} = \pmb{R}\pmb{R}^{\top}$ . The matrix inversion is transformed as  $(\pmb{Z} + \pmb{R}\pmb{R}^{\top})^{-1} = \pmb{Z}^{-1} - \pmb{Z}^{-1}\pmb {R}(\pmb{I}_M + \pmb{R}^\top \pmb{Z}^{-1}\pmb {R})^{-1}\pmb{R}^\top \pmb{Z}^{-1}$  through the Woodbury matrix identity, which costs  $\mathcal{O}(M^3 +NM^2)$ . Note that  $\pmb{Z}^{-1}\pmb{R}$  and  $\pmb{R}^{\top}\pmb{Z}^{-1}$  cost  $\mathcal{O}(NM)$  because  $\pmb{Z}$  is a diagonal matrix.
In computing the marginal likelihood (19), the complexities of the first, the second, and the third terms are  $\mathcal{O}(N)$ ,  $\mathcal{O}(M^3 + NM^2)$ , and  $\mathcal{O}(M^3 + JM^2)$ , respectively, where the matrix determinant lemma is used in computing the second term.
In total, the computational complexity of SurvPP is  $\mathcal{O}(NMQ + (N + J)(d + M)M + M^3)$ , where  $Q$  is the number of gradient descent iterations. This feasible computation, which scales linearly with the number of observed events,  $N$ , the data size,  $J$ , and the covariate dimensionality,  $d$ , is achieved by exploiting the representer theorem. It should be emphasized here that data size  $J$  is not part of the gradient descent iteration term. This is a clear advantage over conventional survival models which naively incur  $\mathcal{O}((dN + dJ + JN)Q)$  computation: Typically,  $J$  is larger than  $N$  in a time-varying covariate scenario (see the counting process format in Section 2.2), and the discrepancy between  $J$  and  $N$  becomes more substantial when the measurements of covariates are made more frequently or/and over a longer period of time.
Survival Models for Time-invariant Covariates: The most popular survival model is the Cox proportional hazards model [5] (CoxPH), which is a semi-parametric model that forms the hazard function as
$$
\lambda (t) = h (t) \exp \left(\beta_ {1} y _ {1} + \beta_ {2} y _ {2} + \dots\right), \tag {22}
$$
where the base hazard function,  $h(t)$ , is obtained by the non-parametric Breslow/Fleming-Harrington estimator [3, 13], and the log-linear regression coefficient  $(\beta_{1},\ldots ,)$  is estimated by maximizing the partial likelihood function. CoxPH is very efficient to compute and scales linearly with the number of events [39], but cannot address the nonlinear dependence of survival times on covariates. To overcome this limitation, a vast number of survival models have been proposed that replace the log-linear parametric function (22) with a non-linear one, such as generalized boosted models [38], random survival forests [17], Gaussian process models [11, 25], and deep neural network models [10, 22, 49]. For a comprehensive review, see Wang et al. [46].
Survival Models for Time-varying Covariates: The counting process format of input [1] plays a central role in extending static survival models into those suitable for time-varying covariates. For each individual, the counting process format splits her/his observation period into multiple short intervals, assigns a constant value of covariate to each interval, and marks the intervals as being censored when no event is present, resulting in a set of pseudo-individual right/left-censored observations with a constant covariate (see Section 2.2). CoxPH was extended to accommodate the counting process format, and the extended CoxPH can estimate the hazard function based on survival data with time-varying covariates [1]. To alleviate the simplest assumption in the extended CoxPH, generalized boosted models [16] and random survival forests [47, 48], both of which are ensemble approaches, have been also extended to accommodate the counting process format, where the extended CoxPH is employed as a weak learner or a tree. While they can address the nonlinear dependence of survival times on time-varying covariates, the large number of learners/trees needed for accurate predictions could make the algorithms too slow and ineffective for real-time applications. Also, Cygu et al. reported that random survival forests for time-varying covariates required substantial amounts of computer memory for large datasets [7].
Exogeneity and Endogeneity of Covariates: When considering survival analysis in the presence of time-varying covariates, we need to distinguish between exogenous and endogenous covariates. Kalbfleisch and Prentice [21] define an endogenous (internal) covariate as the output of a stochastic
Figure 1: Performance on the dataset of log-linear hazard function  $\lambda_{lin}(t)$ . (A) Box plot of TLL and AUC as functions of evaluation point: the higher, the better. (B) The CPU times demanded for estimating a hazard function. The error bars represent the standard deviations. For GBM and SurvPP, the average cpu times over 9-points grid search of the hyperparameter are displayed.
process that is generated by the individual under study. In contrast, an exogenous (external) covariate is not influenced by the individual under study. Diggle et al. [9] suggest using similar but slightly different definitions. The primary purpose of survival analysis with exogenous covariates is to estimate a hazard function as an explicit function of covariates, and to make predictions of failure times under various possible future covariate functions (i.e., what-if analysis). In contrast, the primary purpose of survival analysis with endogenous covariates is to make predictions of failure times by using the past observations of covariates, where joint modeling approaches with recurrent neural networks, which jointly model the stochastic process of covariates and failure times, have been developed intensively [29, 33]. In this paper, we consider exogenous covariates, and survival models for exogenous covariates cannot be compared directly with those for endogenous covariates because the tasks are different. Note that a joint modeling approach for exogenous time-varying covariates has been proposed very recently[32], which was not included in the benchmark models because it was published a month after this paper's submission.
Permanent Process: The permanent process is a variant of Gaussian Cox process that assumes the square root of hazard function to be generated from a Gaussian process [31]. Its computational advantages have recently been highlighted in machine learning research [12, 18, 24, 25, 30, 45]. In particular, the representer theorem in the permanent process has been exploited through the RKHS theory [12], the Mercer's theorem [45], and the path integral formulation [23, 24]. Although the key derivation of SurvPP is based on the path integral methodology used for augmented permanent process (APP) [24], SurvPP is a non-trivial extension of APP: (i) SurvPP can accommodate multiple trials of event sequence data, while APP assumes one trial of event sequence; (ii) in SurvPP, the end time of observation is a stochastic variable that depends on the time of event occurrence, while APP assumes that the end time of observation is given. We discovered, for the first time, that the representer theorem holds for such a complicated point process or a survival model.
We examined the validity of our proposed model by comparing it with conventional survival models on synthetic and real-world data. As benchmark models, we adopted Cox proportional hazards model (CoxPH), generalized boosted model (GBM), and random forest-based model (RFM): We implemented CoxPH and GBM with the established algorithms provided in the packages survival.coxph [41] and gbm3.gbmt [16], respectively; We implemented RFM by using class 1trcrrf in the open R code provided by Yao et al. [48]. The benchmark models can accommodate time-varying covariates via counting process format of input (Surv(t0,t1, event) in survival). For our proposal, we implemented SurvPP by using TensorFlow-2.10<sup>1</sup>. A MacBook Pro with 12-core CPU (Apple M2 Max) was used, where GPU was set as off (tf_device('/cpu:0')) for a
Figure 2: Performance on the dataset of non-linear hazard function  $\lambda_{non}(t)$ . (A) Box plot of TLL and AUC as functions of evaluation point: the higher, the better. (B) The CPU times demanded for estimating a hazard function. The error bars represent the standard deviations. For GBM and SurvPP, the average cpu times over 9-point grid search of the hyperparameter are displayed.
fair benchmark comparisons. For GBM and SurvPP, the hyper-parameters were optimized through 9-point grid search: the number of trees and the shrinkage for GBM, and the kernel parameters for SurvPP. For details of the model configurations, see Appendix C.
We created two survival datasets with time-varying 2-D covariate. One was generated from a hazard function with a log-linear dependence on covariates, and the other with a nonlinear dependence on covariates as follows:
$$
\lambda_ {l i n} (t) = h (t) \exp \left[ 1. 5 y _ {1} (t) + 3. 5 y _ {2} (t) \right], \quad \lambda_ {n o n} (t) = h (t) \exp \left[ 2 - 5 \left(y _ {1} ^ {2} (t) + y _ {2} ^ {2} (t)\right) \right], \tag {23}
$$
where  $h(t)$  is the base hazard function defined by a Weibull hazard function,  $h(t) = 2 \cdot t^{3/2}$ . For each of the datasets, we considered  $U$  individuals, each of which had a 2-D covariate function of time:
$$
\mathbf {y} _ {u} (t) = \left(y _ {1} ^ {u} (t), y _ {2} ^ {u} (t)\right) = \left(\alpha_ {u} \cos \left(2 \pi \omega_ {u} t + \pi \gamma_ {u}\right), \alpha_ {u} ^ {\prime} \cos \left(2 \pi \omega_ {u} ^ {\prime} t + \pi \gamma_ {u} ^ {\prime}\right)\right), \quad u = 1, \dots U, \tag {24}
$$
where  $\alpha_{u}(\alpha_{u}^{\prime})$  and  $\gamma_u(\gamma_u^{\prime})$  were sampled uniformly over [0, 1],  $\omega_{u}$  was sampled uniformly over [5, 10], and  $\omega_{u}^{\prime}$  was sampled uniformly over [20, 30]. We set the censoring time as 1 for all individuals, and an event time generated from (23), denoted by  $t_*$ , was censored ( $\eta_u = 0$ ) when  $t_*$  was over 1:  $T_{u} = \min(t_{*}, 1)$ .
The predictive performances were evaluated based on the dynamic area under the ROC curve (AUC) [34] and the test log-likelihood (TLL),
$$
\operatorname {A U C} (t) = \frac {\sum_ {u = 1} ^ {U} \sum_ {u ^ {\prime} = 1} ^ {U} \mathrm {I} \left(T _ {u} > t\right) \mathrm {I} \left(T _ {u ^ {\prime}} \leq t\right) w _ {u} \mathrm {I} \left(S \left(t \mid \boldsymbol {y} _ {u} (\cdot)\right) \geq S \left(t \mid \boldsymbol {y} _ {u ^ {\prime}} (\cdot)\right)\right)}{\left(\sum_ {u = 1} ^ {U} \mathrm {I} \left(T _ {u} > t\right)\right) \left(\sum_ {u = 1} ^ {U} \mathrm {I} \left(T _ {u} \leq t\right) w _ {u}\right)}, \tag {25}
$$
$$
\operatorname {T L L} (t) = \frac {1}{U} \sum_ {u = 1} ^ {U} \left[ \mathrm {I} \left(t \geq T _ {u}\right) \cdot \eta_ {u} \log \rho \left(T _ {u} \mid \boldsymbol {y} _ {u} (\cdot)\right) + \log S \left(\min  \left(t, T _ {u}\right) \mid \boldsymbol {y} _ {u} (\cdot)\right) \right], \tag {26}
$$
where  $w_{u}$  is the inverse probability of censoring weight, and  $\rho(t|\mathbf{y}_u(\cdot))$  and  $S(t|\mathbf{y}_u(\cdot))$  are the estimated hazard and survival functions given covariate map  $\mathbf{y}_u(\cdot)$ , respectively. The dynamic AUC essentially estimates the C-index at each time, and is commonly used in the literature of survival analysis with time-varying covariates.
The evaluation point,  $0 \leq t_{e} \leq 1$ , was set as  $t_{e} \in \{0.3, 0.5, 0.7, 0.9\}$ . For each dataset, we randomly split the  $U$  individuals into 10 subgroups, assigned one to test and the others to training data, and
Figure 3: Performances on real-world datasets. TLL (the higher, the better), AUC (the higher, the better), and CPU times (the lower, the better) on PBC dataset (A) and SANAD dataset (B).
conducted 10-fold cross evaluation of the predictive performances. We reformatted the training data into counting process format by equally splitting individual  $u$ 's observation periods into  $J_{u}$  sub-region and assuming the covariates to be constant within a sub-region; the survival models were applied to the training data. In this experiment, we considered  $U = 10^{3}$  and  $J_{u} \in \{1,5,10,20,50\}$ : the number of observed events  $N$  was 809 for  $\lambda_{lin}$  and 818 for  $\lambda_{non}$ , respectively.
It should be noted here that when time-varying covariates are considered, the time duration from the entry,  $t$ , is itself a time-varying covariate. SurvPP considered a 3-D covariate map,  $\bar{y}_u(t) \equiv (y_1^u(t), y_2^u(t), t)$ , and performed hazard function estimation, while the other models assumed the hazard function to be  $h(t)f(\pmb{y}(t))$ , and  $h(t)$  and  $f(\pmb{y})$  were estimated separately.
Figure 1 displays the predictive performance on the dataset of  $\lambda_{lin}(t)$ . Figure 1A shows that, except for RFM, there were no significant differences in the performances between the compared models, but CoxPH slightly outperformed the other models in TLL at some evaluation periods. The result is plausible because the underlying generative process was perfectly consistent with the log-linear and proportional assumptions of CoxPH. The comparison in TLL between  $J_u = 5$  and  $J_u = 20$  suggests that more frequently measured covariates would yield better estimations/predictions in scenarios of time-varying covariates. Therefore, the feasible computational complexity of SurvPP regarding  $J_u$  is a clear advantage over the reference models, which is confirmed by Figure 1B.
Figure 2 displays the predictive performance on the dataset of  $\lambda_{non}(t)$ , showing that the non-parametric models, SurvPP and GBM, significantly outperformed CoxPH, while the performance gaps between SurvPP and GBM were marginal and of no significance. However, Figure 2B shows that SurvPP could estimate the hazard function hundreds of times faster than GBM when covariates were measured frequently ( $J_u = 50$ ), and was even faster than the simplest CoxPH. The preferable scalability of SurvPP's algorithm regarding  $J_u$  comes from the fact that  $J_u$  (or  $J = \sum_u J_u$ ) is not part of the iterative optimization (Section 2.6), which is a fruit of SurvPP's representative theorem. RFM was comparable with SurvPP and GBM in AUC, but was worse than either of them in TLL. RFM might need more careful tuning of the hyper-parameters, but this was not fully investigated this time.
We conducted additional experiments on independent validation datasets (see Appendix D.2), and on larger synthetic datasets  $(U\leq 10^{5})$  to examine the model's computation scalability regarding the event number  $N$  (see Appendix D.3).
We examined the validity of SurvPP against the benchmark models on two real-world survival data sets, Mayo Clinic Primary Biliary Cholangitis data (PBC) and Standard And New Antiepileptic
Drugs study data (SANAD), provided by R packages survival (LGPL-3) [41] and joineR (GPL-3) [35], respectively. In PBC, 312 patients with primary biliary cirrhosis were enrolled in a randomized medical trial at the Mayo Clinic between 1974 and 1984 [8], where events were the time of death; one static and eleven time-varying covariates were measured at entry and at yearly intervals, which include age at entry, alkaline phosphotase, logarithm of serum albumin, presence of ascites, aspartate aminotransferase, logarithm of serum bilirubin, serum cholesterol, condition of edema, presence of hepatomegaly or enlarged liver, platelet count, logarithm of prothrombin time, and presence or absence of spiders. SANAD was an unblind randomized trial that recruited patients with epilepsy for whom carbamazepine (CBZ) was considered to be standard treatment and they were randomized to CBZ or the newer drug lamotrigine (LTG), where 605 patients were included and event was the time to treatment failure; we adopted calibrated dose as a time-varying covariate, and three static covariates including age of patient at randomization, gender, and randomized treatment (CBZ or LTG); calibrated dose was measured at 166-day intervals on average, and we used a linear interpolation to obtain the values of calibrated dose at which the measurement intervals were regularly trisected. We randomly split the individuals into 10 subgroups, assigned one to test and the others to training data, and conducted 10-fold cross evaluation of the predictive performances.
Figure 3A displays the predictive performance on PBC. It shows that  $\mathrm{CoxPH}$  achieved very high AUCs ( $>0.9$ ), suggesting that the underlying generative process could be consistent with  $\mathrm{CoxPH}$ 's simple assumption of log-linearity and proportionality. Thus the semi-parametric model,  $\mathrm{CoxPH}$ , is likely to achieve equal or slightly better performances than the nonparametric models, which is consistent with the result. Figure 3B plots the predictive performance on SANAD, where the not so high AUCs of  $\mathrm{CoxPH}$  suggests that the underlying dependence of intensity on covariates is nonlinear. The figure shows that  $\mathrm{SurvPP}$  achieved better performance than  $\mathrm{CoxPH}$ , and achieved comparable performance while being substantially faster than GBM.
We have proposed a non-parametric Bayesian survival model to address survival data with time-varying covariates. We tailored a permanent process such that the latent hazard function is defined on covariate space and right-censored observations in a counting process format can be handled, which we call the survival permanent process (SurvPP). Through the path integral formulation of Gaussian process, we showed that SurvPP encompasses a representative theorem, and derived a feasible estimation algorithm that scales linearly with the number of observed events. We evaluated SurvPP on synthetic data, confirming that it achieved comparable predictive accuracy while being tens to hundreds of times faster than state-of-the-art methods.
Limitations & future work: We examined SurvPP for relatively low-dimensional covariates, and its potential suitability for high-dimensional covariates remains to be clarified. Because SurvPP is based on a normal Gaussian process, the (equivalent) kernel function might be too simple to discover meaningful representations in high-dimensional covariate data. A promising direction is to apply deep kernel learning to SurvPP, where high-dimensional covariates are transformed by nonlinear mapping with a deep architecture. A technical issue is that we could not search an appropriate set of (a lot of) parameters of neural networks, because our proposed scheme performs the hyperparameter optimization by grid search, not by gradient descent. Also, Gaussian kernels, which were used in the paper, naively require a scale parameter for each dimension of data, resulting in a high dimensional kernel parameter for high-dimensional covariate scenarios. The grid search becomes prohibitively costly with high dimensional parameters, but we addressed the problem by a well-known approach that normalizes (e.g., centering and scaling) each dimension of the data and puts a common scale parameter across all dimensions of normalized data. This approach empirically works robustly, but a more sophisticated approach could improve the performance of SurvPP. Variational Bayesian approximations with inducing points might address the technical issues of hyper-parameter optimization, and thus is an important next step in our study.
As in ordinary permanental processes, the nodal line problem could arise in SurvPP: the posterior distribution of the latent variable  $x(\cdot)$  has many local modes since  $\pm x(\cdot)$  can lead to similar hazard functions  $\lambda = x^2(\cdot)$ , and artificial zero crossings of  $x(\cdot)$  could happen, especially on locations where the hazard function is low. John and Hensman [18] have proposed to extend the quadratic link function to include an offset parameter  $\beta$ , so that  $\lambda(\pmb{y}) = (x(\pmb{y}) + \beta)^2$ , which is valid for SurvPP.
[38] Greg Ridgeway. Generalized Boosted Models: A guide to the gbm package. Update, 1(1): 2007, 2007.
[39] Noah Simon, Jerome Friedman, Trevor Hastie, and Rob Tibshirani. Regularization paths for Cox's proportional hazards model via coordinate descent. Journal of Statistical Software, 39 (5):1, 2011.
[40] Terry Therneau, Cindy Crowson, and Elizabeth Atkinson. Using time dependent covariates and time dependent coefficients in the Cox model. Survival Vignettes, 2:3, 2017.
[41] Terry M. Therneau. A Package for Survival Analysis in R, 2023. URL https://CRAN.R-project.org/package=survival.R package version 3.5-7.
[42] Andrea Vedaldi and Andrew Zisserman. Efficient additive kernels via explicit feature maps. IEEE Transactions on Pattern Analysis and Machine Intelligence, 34(3):480-492, 2012.
[43] Sreekanth Vempati, Andrea Vedaldi, Andrew Zisserman, and C. V. Jawahar. Generalized RBF feature maps for efficient detection. In BMVC, pages 1-11, 2010.
[44] Grace Wahba. Spline Models for Observational Data, volume 59. SIAM, 1990.
[45] Christian J. Walder and Adrian N. Bishop. Fast Bayesian intensity estimation for the permanent process. In International Conference on Machine Learning, volume 70, pages 3579-3588. PMLR, 2017.
[46] Ping Wang, Yan Li, and Chandan K. Reddy. Machine learning for survival analysis: A survey. ACM Computing Surveys (CSUR), 51(6):1-36, 2019.
[47] Shannon Wongvibulsin, Katherine C. Wu, and Scott L. Zeger. Clinical risk prediction with random forests for survival, longitudinal, and multivariate (RF-SLAM) data analysis. BMC Medical Research Methodology, 20(1):1-14, 2020.
[48] Weichi Yao, Halina Frydman, Denis Larocque, and Jeffrey S. Simonoff. Ensemble methods for survival function estimation with time-varying covariates. Statistical Methods in Medical Research, 31(11):2217-2236, 2022.
[49] Qixian Zhong, Jonas W. Mueller, and Jane-Ling Wang. Deep extended hazard models for survival analysis. In Advances in Neural Information Processing Systems 34, 2021.
A Derivation of MAP Estimator
We detail the derivation of the MAP estimator (12). The functional derivative of  $S(x(\pmb{y}), \underline{x}(\pmb{y}))$  should be zero on MAP estimator  $\hat{x}(\pmb{y})$ :
$$
\begin{array}{l} \delta S \left(\hat {x} (\boldsymbol {y}), \underline {{\hat {x}}} (\boldsymbol {y})\right) = \int_ {\mathcal {Y}} \left[ \frac {\delta S}{\delta \hat {x} (\boldsymbol {y})} \delta x (\boldsymbol {y}) + \frac {\delta S}{\delta \underline {{\hat {x}}} (\boldsymbol {y})} \delta \underline {{x}} (\boldsymbol {y}) \right] d \boldsymbol {y} + O \left((\delta x) ^ {2}\right) \\ \simeq \int_ {\mathcal {Y}} \left[ 2 \sum_ {j = 1} ^ {J} \Delta_ {j} \hat {x} (\boldsymbol {y}) \delta (\boldsymbol {y} - \boldsymbol {y} _ {j}) - \sum_ {n = 1} ^ {N} \frac {2}{\hat {x} (\boldsymbol {y})} \delta (\boldsymbol {y} - \tilde {\boldsymbol {y}} _ {n}) + \frac {1}{2} \underline {{\hat {x}}} (\boldsymbol {y}) \right] \delta x d \boldsymbol {y} \\ + \int_ {\mathcal {Y}} \frac {1}{2} \hat {x} (\boldsymbol {y}) \delta \underline {{\boldsymbol {x}}} d \boldsymbol {y} \\ = \int_ {\mathcal {Y}} \left[ 2 \sum_ {j = 1} ^ {J} \Delta_ {j} \hat {x} (\boldsymbol {y}) \delta (\boldsymbol {y} - \boldsymbol {y} _ {j}) - \sum_ {n = 1} ^ {N} \frac {2}{\hat {x} (\boldsymbol {y})} \delta (\boldsymbol {y} - \tilde {\boldsymbol {y}} _ {n}) + \underline {{\hat {x}}} (\boldsymbol {y}) \right] \delta x d \boldsymbol {y} = 0, \\ \end{array}
$$
where the following relation was used,