SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \int_ {\mathcal {Y}} \hat {x} (\boldsymbol {y}) \delta \underline {{x}} d \boldsymbol {y} = \int_ {\mathcal {Y}} \hat {x} (\boldsymbol {y}) \int_ {\mathcal {Y}} k ^ {*} (\boldsymbol {y}, \boldsymbol {y} ^ {\prime}) \delta x (\boldsymbol {y} ^ {\prime}) d \boldsymbol {y} ^ {\prime} d \boldsymbol {y} \\ = \int_ {\mathcal {Y}} d \boldsymbol {y} ^ {\prime} \delta x (\boldsymbol {y} ^ {\prime}) \int_ {\mathcal {Y}} k ^ {*} (\boldsymbol {y}, \boldsymbol {y} ^ {\prime}) \hat {x} (\boldsymbol {y}) d \boldsymbol {y} \\ = \int_ {\mathcal {Y}} \underline {{\hat {x}}} (\boldsymbol {y} ^ {\prime}) \delta x d \boldsymbol {y} ^ {\prime}. \quad \because) k ^ {*} (\boldsymbol {y}, \boldsymbol {y} ^ {\prime}) = k ^ {*} (\boldsymbol {y} ^ {\prime}, \boldsymbol {y}) \\ \end{array}
$$
Thus the following equation is derived,
$$
\underline {{\hat {x}}} (\boldsymbol {y}) + 2 \sum_ {j = 1} ^ {J} \Delta_ {j} \hat {x} (\boldsymbol {y} _ {j}) \delta (\boldsymbol {y} - \boldsymbol {y} _ {j}) = \sum_ {n = 1} ^ {N} \frac {2}{\hat {x} \left(\tilde {\boldsymbol {y}} _ {n}\right)} \delta (\boldsymbol {y} - \tilde {\boldsymbol {y}} _ {n}), \quad \boldsymbol {y} \in \mathcal {Y}. \tag {A1}
$$
By applying operator  $\mathcal{K}$  to (A1), we obtain a linear integral equation that derives the MAP estimator  $\hat{x}(\pmb{y})$  as follows,
$$
\hat {x} (\boldsymbol {y}) + 2 \sum_ {j = 1} ^ {J} \Delta_ {j} \hat {x} (\boldsymbol {y} _ {j}) k (\boldsymbol {y}, \boldsymbol {y} _ {j}) = 2 \sum_ {n = 1} ^ {N} k (\boldsymbol {y}, \tilde {\boldsymbol {y}} _ {n}) \hat {x} \left(\tilde {\boldsymbol {y}} _ {n}\right) ^ {- 1}, \quad \boldsymbol {y} \in \mathcal {Y}. \tag {A2}
$$
The linearity of the integral equation permits a representation of the form
$$
\hat {x} (\boldsymbol {y}) = 2 \sum_ {n = 1} ^ {N} h (\boldsymbol {y}, \tilde {\boldsymbol {y}} _ {n}) \hat {x} (\tilde {\boldsymbol {y}} _ {n}) ^ {- 1},
$$
where  $h(\pmb{y}, \pmb{y}')$  is a positive semi-definite kernel that solves integral equation (13). Note that the derivations follow Kim [23].
B Derivation of Predictive Distribution and Marginal Likelihood
We now know the mode of the posterior,  $\hat{x} (\pmb {y})$ , and consider a Taylor expansion of functional action potential  $S(x(\pmb {y}),\underline{x} (\pmb {y}))$  centered on the mode such that
$$
S \left(x (\boldsymbol {y}), \underline {{x}} (\boldsymbol {y})\right) \simeq S \left(\hat {x} (\boldsymbol {y}), \underline {{\hat {x}}} (\boldsymbol {y})\right) + \frac {1}{2} \iint_ {\mathcal {Y} \times \mathcal {Y}} \sigma^ {*} (\boldsymbol {y}, \boldsymbol {y} ^ {\prime}) (x (\boldsymbol {y}) - \hat {x} (\boldsymbol {y})) (x (\boldsymbol {y} ^ {\prime}) - \hat {x} (\boldsymbol {y} ^ {\prime})) d \boldsymbol {y} d \boldsymbol {y} ^ {\prime}, \tag {B1}
$$
where  $\sigma^{*}(\pmb{y},\pmb{y}^{\prime}) = \frac{\delta^{2}S(x,\underline{x})}{\delta x(\pmb{y})\delta x(\pmb{y}^{\prime})}\big|_{x = \hat{x}}$  is the second derivative of  $S$ . The first term in the Taylor expansion vanishes due to the stationary condition. The quadratic approximation of the action integral corresponds to the approximation of the posterior process by a GP, and the predictive covariance or the kernel function for the posterior GP, denoted by  $\sigma (\pmb {y},\pmb{y}^{\prime})$ , can be obtained by the functional inversion of  $\sigma^{*}(\pmb {y},\pmb{y}^{\prime})$ , which results in
$$
\sigma (\boldsymbol {y}, \boldsymbol {y} ^ {\prime}) = h (\boldsymbol {y}, \boldsymbol {y} ^ {\prime}) - h (\boldsymbol {y}) ^ {\top} (\boldsymbol {Z} + \boldsymbol {H}) ^ {- 1} h (\boldsymbol {y} ^ {\prime}), \tag {B2}
$$
where the definitions of  $Z$  and  $H$  are taken from (20). The full derivation of (B2) is given in [24]. When the latent function  $x(\pmb{y})$  follows a posterior GP with mean of  $\hat{x}(\pmb{y})$  and kernel  $\sigma(\pmb{y}, \pmb{y}')$ , it is easily verified that the value of the squared function,  $\lambda = x^2(\pmb{y})$ , on each point of the covariate domain  $\pmb{y} \in \mathcal{V}$  follows a Gamma distribution defined by (21).
Furthermore, under Laplace approximation (B1), we can obtain the marginal likelihood,  $p(\mathcal{D})$ , in (10) by performing the path integral as,
$$
\log p (\mathcal {D}) = \log \int \exp \left[ - S (x (\boldsymbol {y}), \underline {{x}} (\boldsymbol {y})) - \frac {1}{2} \log | \mathcal {K} | \right] \mathcal {D} x \simeq - S (\hat {x} (\boldsymbol {y}), \underline {{\hat {x}}} (\boldsymbol {y})) + \frac {1}{2} \log \frac {| \Sigma |}{| \mathcal {K} |}, \tag {B3}
$$
where  $|\Sigma|$  is the functional determinant of integral operator  $\Sigma = \int_{\mathcal{Y}} \cdot \sigma(\boldsymbol{y}, \boldsymbol{y}') d\boldsymbol{y}'$ . We can rewrite the result in a more tractable form by substituting (11, B2) into (B3):
$$
\log p (\mathcal {D}) = \log | \boldsymbol {Z} | - \frac {1}{2} \log | \boldsymbol {I} _ {N} + \boldsymbol {Z} ^ {- 1} \boldsymbol {H} | - \frac {1}{2} \log | \boldsymbol {I} _ {M} + 2 \boldsymbol {A} | ^ {- 1} + (\log 2 - 1) N.
$$
Full derivations are provided in [24].
C Model Configuration
Survival Permanental Process (SurvPP)
We set the number of features for Random feature map [37] ( $M$ ), learning parameter ( $lr$ ), and stop condition ( $G$ ) for Adam [26] as follows:
$$
M = 5 0 0, \quad l r = 0. 0 5, \quad G <   1 0 ^ {- 5}.
$$
We applied to SurvPP a multiplicative Gaussian kernel
$$
k \left(\boldsymbol {y}, \boldsymbol {y} ^ {\prime}\right) = \prod_ {d = 1} ^ {3} e ^ {- \left(\theta \left(y _ {d} - y _ {d} ^ {\prime}\right)\right) ^ {2}}, \quad \boldsymbol {y} = \left(t, y _ {1}, y _ {2}\right),
$$
where hyper-parameter  $\theta$  was optimized for each data by maximizing the marginal likelihood through grid search. In the experiments on synthetic data, we selected a set of nine values for  $\theta$  as the grid points,
$$
\theta \in \{0. 1, 0. 2, 0. 5, 0. 7, 1. 0, 2. 0, 5. 0, 7. 0, 1 0. \}
$$
We implemented SurvPP by using TensorFlow-2.10. A MacBook Pro with 12-core CPU (Apple M2 Max) was used, with the GPU inactivated (tf_device('/cpu:0')) for a fair comparison with the benchmarks.
Cox Proportional Hazards Model (CoxPH)
We implemented CoxPH through package survival.coxph (LGPL-3) [41]. The calls in coxph to fit a model and compute a base hazard function were
$$
\begin{array}{l} > \operatorname {c f i t} = \operatorname {c o x p h} (\operatorname {S u r v} (\operatorname {S t a r t}, \operatorname {S t o p}, \operatorname {E v e n t}) \sim \operatorname {c o v 1} + \operatorname {c o v 2}, \operatorname {d f}) \\ > \text {s f i t} = \text {s u r v f i t} (\text {c f i t}, \text {l i s t} (\operatorname {c o v 1} = 0, \operatorname {c o v 2} = 0)), \\ \end{array}
$$
where df was the survival data in counting process format.
Generalized Boosted Model (GBM)
We implemented GBM through package gbm3. gbtm [16] (GPL). The call in gbtm to fit a model was
$$
\begin{array}{l} > \operatorname {g f i t} = \operatorname {g b m t} (\operatorname {S u r v} (\operatorname {S t a r t}, \operatorname {S t o p}, \operatorname {E v e n t}) \sim \operatorname {c o v 1} + \operatorname {c o v 2}, \operatorname {d a t a} = \operatorname {d f}, \\ d i s t r i b u t i o n = \text {g b m} _ {\cdot} \text {d i s t} \left(" C o x P H ^ {\prime}\right), \\ c v \text {f o l d s} = 1 0, \text {t r a i n} \quad \text {p a r a m s} = \text {p a r a m s}, \\ \operatorname {p a r \_ d e t a i l s} = \operatorname {g b m P a r a l l e l} (\operatorname {n u m \_ t h r e a d s} = 1 2)), \\ \end{array}
$$
where params represents the hyperparameter. We selected a set of nine hyperparameters for the grid search,
$$
\text {n u m} _ {\text {t r e e s}} \in \{5 0 0, 1 0 0 0, 2 0 0 0 \} \times \text {s h r i n k a g e} \in \{0. 0 0 1, 0. 0 0 5, 0. 0 1 \},
$$
and found the one that minimized the cross validation error (gfit $\text{valid失误}$ ), where num_treess and shrinkage represent the number of trees and the shrinkage/learning rate, respectively.
Random Forest-based Model (RFM)
We implemented RFM through package LTRCforests (GPL) [48]. The call to fit a model was
$$
\begin{array}{l} > r f i t = l t r c r r f (S u r v (S t a r t, S t o p, E v e n t) \sim c o v 1 + c o v 2, d a t a = d f, \\ \operatorname {i d} = \operatorname {I D}, \operatorname {m t r y} = \operatorname {c e i l i n g} (1 0), \operatorname {n t r e e} = 1 0 0). \\ \end{array}
$$
C.2 Real-world Data: PBC
$$
M = 5 0 0, \quad l r = 5 0, \quad G <   1 0 ^ {- 5}.
$$
$$
k (\boldsymbol {y}, \boldsymbol {y} ^ {\prime}) = \prod_ {d = 1} ^ {1 3} e ^ {- \left(\theta \left(y _ {d} - y _ {d} ^ {\prime}\right)\right) ^ {2}}, \quad \boldsymbol {y} = (t, y _ {1}, \dots , y _ {1 2}),
$$
where the hyper-parameter  $\theta$  was optimized for each data by maximizing the marginal likelihood through grid search. In the experiments on synthetic data, we selected a set of nine values for  $\theta$  as the grid points,
$$
\theta \in \{0. 0 1, 0. 0 2, 0. 0 3, 0. 0 4, 0. 0 5, 0. 0 6, 0. 0 7, 0. 0 8, 0. 0 9 \}.
$$
Here, we normalized the 13 covariates so that  $y_{d} \rightarrow 0.1(y_{d} - \mathrm{mean}[y_{d}]) / \mathrm{std}[y_{d}]$ .
As in the experiments on synthetic data, GPU was set off (tf_device('/cpu:0')) for a fair comparison with the benchmarks.
The calls in coxph to fit a model and compute a base hazard function were
> cfit = coxph(Surv(Start, Stop, Event) ~ age + edema + alk.phos + chol + ast + platelet + spiders + hepato + ascites + albumin + bili + protime, df)  
> sfit = survfit(cfit, list(age = 0, edema = 0, alk.phos = 0, chol = 0, ast = 0, platelet = 0, spiders = 0, hepato = 0, ascites = 0, albumin = 0, bili = 0, protime = 0)),
The call in gbt to fit a model was
> gfit = gbmT(Surv(Start, Stop, Event) ~ age + edema + alk.phos + chol + ast + platelet + spiders + hepato + ascites + albumin + bili + proftime, data = df, distribution = glm_dist("CoxPH"), cv_folds = 10, train_parameters = params, par_details = glmParallel(num Threads = 12)),
The call to fit a model was
$$
\begin{array}{l} > r f i t = l t r c r r f (S u r v (S t a r t, S t o p, E v e n t) \sim a g e + e d e m a + a l k. p h o s + c h o l + a s t \\ + p l a t e l e t + s p i d e r s + h e p a t o + a s c i t e s + a l b u m i n + b i l i \\ + p r o t i m e, d a t a = d f, i d = I D, s t e p F a c t o r = 1. 5). \\ \end{array}
$$
C.3 Real-world Data: SANAD
$$
M = 5 0 0, \quad l r = 1 0, \quad G <   1 0 ^ {- 5},
$$
$$
k (\boldsymbol {y}, \boldsymbol {y} ^ {\prime}) = 0. 1 \prod_ {d = 1} ^ {5} e ^ {- \left(\theta \left(y _ {d} - y _ {d} ^ {\prime}\right)\right) ^ {2}}, \quad \boldsymbol {y} = (t, y _ {1}, \dots , y _ {4}),
$$
$$
\theta \in \{0. 6, 0. 7, 0. 8, 0. 9, 1. 0, 1. 1, 1. 2, 1. 3, 1. 4 \}.
$$
Here, we normalized the 5 covariates so that  $y_{d} \rightarrow 0.1(y_{d} - \mathrm{mean}[y_{d}]) / \mathrm{std}[y_{d}]$ .
$$
\begin{array}{l} > \operatorname {c f i t} = \operatorname {c o x p h} (\operatorname {S u r v} (\operatorname {S t a r t}, \operatorname {S t o p}, \operatorname {E v e n t}) \sim \operatorname {a g e} + \operatorname {g e n d e r} + \operatorname {t r e a t} + \operatorname {d o s e}, \operatorname {d f}) \\ > \text {s f i t} = \text {s u r v f i t} (\text {c f i t}, \text {l i s t} (\text {a g e} = 0, \text {g e n d e r} = 0, \text {t r e a t} = 0, \text {d o s e} = 0)), \\ \end{array}
$$
$$
\begin{array}{l} > \text {g f i t} = \text {g b m t} (\text {S u r v} (\text {S t a r t}, \text {S t o p}, \text {E v e n t}) \sim \text {a g e} + \text {g e n d e r} + \text {t r e a t} + \text {d o s e}, \\ \mathrm {d a t a} = \mathrm {d f}, \text {d i s t r i b u t i o n} = \mathrm {g b m} _ {\text {一}} \text {d i s t} \left(" \mathrm {C o x P H}"\right), \\ c v \text {f o l d s} = 1 0, \text {t r a i n} \text {p a r a m s} = \text {p a r a m s}, \\ \operatorname {p a r \_ d e t a i l s} = \operatorname {g b m P a r a l l e l} (\operatorname {n u m \_ t h r e a d s} = 1 2)), \\ \end{array}
$$
Figure D1: Performances on the dataset of non-linear hazard function  $\lambda_{\text{non}}(t)$  for different numerical integration rules. SurvPP_Rie# and SurvPP_Sim# represent SurvPPs with Riemann sum and Simpson's rule approximations of integration, respectively. # is the number of representative time points for covariate,  $J_u$ .
$$
\begin{array}{l} > r f i t = l t r c r r f (S u r v (S t a r t, S t o p, E v e n t) \sim a g e + g e n d e r + t r e a t + d o s e, \\ \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \text {d a t a} = d f, i d = I D, s t e p F a c t o r = 1. 5). \end{array}
$$
D.1 More Accurate Evaluation of Equivalent Kernel
The functional form of covariate map,  $\pmb{y}(t)$ , involves the sum of outer products of feature maps,  $\pmb{A}$ , defined by Equation (16), and  $\pmb{A}$  can be considered as a Riemann sum approximation of the time integral as follows:
$$
\boldsymbol {A} = \sum_ {u = 1} ^ {U} \int_ {0} ^ {T _ {u}} \phi (\boldsymbol {y} _ {u} (t)) \phi (\boldsymbol {y} _ {u} (t)) ^ {\top} d t \sim \sum_ {j = 1} ^ {J} \Delta_ {j} \phi (\boldsymbol {y} _ {j}) \phi (\boldsymbol {y} _ {j}) ^ {\top}. \tag {D1}
$$
Thus, when the covariate map,  $\pmb{y}_u(t)$ , and the feature map,  $\phi(\pmb{y})$ , are both smooth, the Riemann sum approximation can be replaced by a more accurate approximation,
$$
\boldsymbol {A} = \sum_ {u = 1} ^ {U} \int_ {0} ^ {T _ {u}} \phi (\boldsymbol {y} _ {u} (t)) \phi (\boldsymbol {y} _ {u} (t)) ^ {\top} d t \sim \sum_ {j = 1} ^ {J} w _ {j} \phi (\boldsymbol {y} _ {j}) \phi (\boldsymbol {y} _ {j}) ^ {\top}, \tag {D2}
$$
where the weights,  $w_{j}$ , depend on the approximation rule. This time, we adopt Simpson's 1/3 rule, and add an experiment on the synthetic nonlinear data,  $\lambda_{\text{non}}(t)$ , to check for an improvement in accuracy. Note that Simpson's 1/3 rule corresponds to quadratic interpolation. Figure D1 shows that adopting Simpson's rule achieved better predictive performance on TLL with a smaller discretization number  $(J_{u})$ .
D.2 Experiment on Independent Validation Data Sets
In Section 4, we evaluated the performances on synthetic data with the cross-validation approach because it is common in the machine learning literature. But synthetic data offers the unique advantage of being able to generate as much data as needed, and we here conducted the performance assessment with the independent validation approach. Figure D2 displays the result on independent validation sets of synthetic data ( $U = 900$  for each of 10 training data, and  $U = 100$  for each of 10 test data), showing the result similar to the cross-validation approach (Figure 1-2).
D.3 Experiment on Larger Data Sets
To examine the computation scalability of the compared models versus the number of observed events  $N$ , we created data sets with user size  $U \in \{10^3, 10^4, 5 \cdot 10^4, 10^5\}$  according to the nonlinear
Figure D2: Performance on the independent validation synthetic datasets. (A) Log-linear hazard function  $\lambda_{lin}(t)$ . (B) Non-linear hazard function  $\lambda_{non}(t)$ . The results are consistent with Figure 1-2
Figure D3: The CPU times demanded for estimating a hazard function versus the number of observed events. The error bars represent the standard deviations across 10 trials. The dashed line represents a line of  $\mathrm{CPU} \propto N$  as reference. For GBM and SurvPP, the average cpu times over 9-point grid search of the hyperparameter are displayed. The CPU times of GBM and RFM exceeded  $10^{3}$  seconds with  $N > 10^{4}$ , and the estimations were given up.
scenario (see Section 4.1),
$$
\lambda_ {n o n} (t) = h (t) \exp \left[ 2 - 5 \left(y _ {1} ^ {2} (t) + y _ {2} ^ {2} (t)\right) \right], \quad h (t) = 2 \cdot t ^ {3 / 2},
$$
which resulted in the data sets with  $N \in \{818, 8082, 40660, 81066\}$ . Here, we set  $J_{u}$  to 10 for the counting process format of data. For each dataset, we randomly split the  $U$  individuals into 10 subgroups, repeated assigning 9 subgroups to training data, and conducted 10 trials of evaluations of the CPU times demanded for estimating a hazard function. Figure D3 displays the CPU times as function of  $N$  of training data. It shows that the CoxPH computation clearly scaled linearly with  $N$ , while that of SurvPP seems to be a little more than linear. This is because that each iteration of gradient descent algorithm scaled linearly with  $N$ , but the number of iterations to meet stop condition  $G < 10^{-5}$  increased moderately with  $N$ . Among the non-parametric approaches (SurvPP, GBM, and RFM), SurvPP achieved the fastest computation at hundreds of times faster than the others, regardless of the number of observed events  $N$ .
D.4 Sensitivity to Hyper-Parameters
Kernel methods/GP models are generally sensitive to kernel parameter values, and a effective way of optimizing the kernel parameter is essential. Figure D4 plots AUC performance on PBC dataset for SurvPPs with different search ranges of kernel parameters: the range of  $\mathrm{SurvPP_0}$  is  $\theta \in \{0.1, 0.2, 0.5, 0.7, 1.0, 2.0, 5.0, 7.0, 10.0\}$ , which was used in the supplement; the range of  $\mathrm{SurvPP_1}$  is  $\theta \in \{0.01, 0.02, 0.03, \dots, 0.09\}$ . Figure D4 shows that  $\mathrm{SurvPP_1}$  achieved substantially
Figure D4: Performances on PBC dataset for SurvPPs with different search ranges of kernel parameters. (A) The search range of  $\mathrm{SurvPP_0}$  is  $\theta \in \{0.1, 0.2, 0.5, 0.7, 1.0, 2.0, 5.0, 7.0, 10.0\}$ . (B) The search range of  $\mathrm{SurvPP_1}$  is  $\theta \in \{0.01, 0.02, 0.03, \ldots, 0.09\}$ .
Figure D5: Performances on MIMIC III dataset. (A) Box plot of TLL and AUC as functions of evaluation point: the higher, the better. (B) The CPU times demanded for estimating a hazard function. The error bars represent the standard deviations. The average cpu times over 9-point grid search of the hyperparameter are displayed.
better AUC performance than  $\mathrm{SurvPP_0}$ , where  $\mathrm{SurvPP_1}$  is displayed in Figure 3. A more sophisticated algorithm for hyperparameter optimization could enhance the performance of  $\mathrm{SurvPP}$ , which is the next step in our study.
D.5 Experiment on ICU Data Set
We examined the validity of SurvPP against GBM on MIMIC-III Clinical Database (MIMIC III), the large publicly available dataset of over 50,000 ICU admissions from the Beth Israel Deaconess Medical Center [19], where events were deaths. We extracted admissions and measurements from CareVue (ITEMID) such that admissions shared 14 measured covariates with each other, which resulted in 133 admissions. We adopted sex as a static covariate and the 14 measurements as time-varying covariates. We randomly split the individuals into 3 subgroups, assigned one to test and the others to training data, and conducted 3-fold cross evaluation of the predictive performances. The model configuration follows the experiment on PBC data set (see Appendix C).
Figure D5 displays the predictive performance on MIMIC III. It shows that SurvPP achieved better TLL performance than GBM, and SurvPP achieved comparative AUC performance with GBM. The smaller variance of SurvPP's AUC implies that SurvPP works more robustly than GBM, but further experiments are needed to investigate it.