SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Condition 4.5.  $G_{\theta}(x)$  is viewed as a bivariate function  $G:\Theta \times \mathcal{B}\mapsto \mathbb{R}$ , where  $\mathcal{B}$  is a compact set on  $\mathbb{R}$ . The functions  $G_{\theta}(x),\frac{\partial}{\partial\theta} G_{\theta}(x),\frac{\partial}{\partial x} G_{\theta}(x),\log g_{\theta}(x),\frac{\partial}{\partial\theta}\log g_{\theta}(x),\frac{\partial}{\partial x}\log g_{\theta}(x)$  are bounded on  $\Theta \times \mathcal{B}$ .
We define two metrics that measures convergence of parameter estimates: For the PF scheme, let  $\phi_0 = (h_0,m_0,\theta_0)$  be the true parameters and  $\widehat{\phi}_n = (\widehat{h}_n,\widehat{m}_n,\widehat{\theta}_n)$  be the estimates. We abbreviate  $\mathbb{P}_{\phi_0,Z = z}$  as the conditional probability distribution of  $(T,\delta)$  given  $Z = z$  under the true parameter, and  $\mathbb{P}_{\widehat{\phi}_n,Z = z}$  as the conditional probability distribution of  $(T,\delta)$  given  $Z = z$  under the estimates. Define the following metric
$$
d _ {\mathbb {P F}} \left(\widehat {\phi} _ {n}, \phi_ {0}\right) = \sqrt {\mathbb {E} _ {z \sim \mathbb {P} _ {Z}} \left[ H ^ {2} \left(\mathbb {P} _ {\widehat {\phi} _ {n} , Z = z} \| \mathbb {P} _ {\phi_ {0} , Z = z}\right) \right]}, \tag {9}
$$
where  $H^{2}(\mathbb{P}\parallel \mathbb{Q}) = \int \left(\sqrt{d\mathbb{P}} -\sqrt{d\mathbb{Q}}\right)^{2}$  is the squared Hellinger distance between probability distributions  $\mathbb{P}$  and  $\mathbb{Q}$ . The case for the FN scheme is similar: Let  $\psi_0 = (\nu_0,\theta_0)$  be the parameters and  $\widehat{\nu}_n = (\widehat{\nu}_n,\widehat{\theta}_n)$  be the estimates. Analogous to the definitions above, we define  $\mathbb{P}_{\psi_0,Z = z}$  as the true conditional distribution given  $Z = z$ , and  $\mathbb{P}_{\widehat{\psi}_n,Z = z}$  be the estimated conditional distribution, we will use the following metric in the FN scheme:
$$
d _ {\mathsf {F N}} \left(\widehat {\psi} _ {n}, \psi_ {0}\right) = \sqrt {\mathbb {E} _ {z \sim \mathbb {P} _ {Z}} \left[ H ^ {2} \left(\mathbb {P} _ {\widehat {\psi} _ {n} , Z = z} \| \mathbb {P} _ {\psi_ {0} , Z = z}\right) \right]}. \tag {10}
$$
Now we state our main theorems. We denote  $\mathbb{P}$  as the data generating distribution and use  $\widetilde{O}$  to hide poly-logarithmic factors in the big-O notation.
Theorem 4.6 (Rate of convergence, PF scheme). In the PF scheme, under condition 4.1, 4.3, 4.5, we have that  $d_{PF}\left(\widehat{\phi}_n,\phi_0\right) = \widetilde{O}_{\mathbb{P}}\left(n^{-\frac{\beta}{2\beta + 2d}}\right)$ .
Theorem 4.7 (Rate of convergence, FN scheme). In the FN scheme, under condition 4.2, 4.4, 4.5, we have that  $d_{FN}\left(\widehat{\psi}_n,\psi_0\right) = \widetilde{O}_{\mathbb{P}}\left(n^{-\frac{\beta}{2\beta + 2d + 2}}\right)$ .
Remark 4.8. The idea of using Hellinger distance to measure the convergence rate of sieve MLEs was proposed in [70]. Obtaining rates under a stronger topology such as  $L_{2}$  is possible if the likelihood function satisfies certain conditions such as the curvature condition [26]. However, such kind of conditions is in general too stringent for likelihood-based objectives, instead, we use Hellinger distance that has minimal requirements. Consequently, our proof strategy is applicable to many other survival models that rely on neural function approximation such as [56], with some modification to the regularity conditions. For proper choices of metrics in sieve theory, see also the discussion in [11, Chapter 2].
Figure 1: Visualizations of synthetic data results under the NFM framework. The plots in the first row compare the empirical estimates of the nonparametric component  $\nu(t,Z)$  against its true value evaluated on 100 hold-out points, under the PF scheme. The plots in the second row are obtained using the FN scheme, with analogous semantics to the first row.
In this section, we report the empirical performance of NFM, we will focus on the following two research questions:
RQ1(Verification of statistical correctness): The results listed in section characterized the convergence results in theory, providing a crude guide on the number of samples required for an accurate estimate. Nonetheless, theoretical rates are often pessimistic, thus we want to investigate whether a moderate number of sample size suffices for good approximation.
RQ2(Assessment of empirical efficacy): While NFM is theoretically sound in terms of estimation accuracy, the theory we have developed does not necessarily guarantee its empirical efficacy as a method of doing prognosis. It is therefore valuable to inspect how useful NFM is regarding real-world predictive tasks in survival analysis.
5.1 Synthetic experiments
To answer RQ1, we conduct synthetic experiments to check the empirical convergence. Specifically, we investigate the empirical recovery of underlying ground truth parameters under various level of sample size.
Ground truth We set the true underlying model to be a nonlinear gamma-frailty model with a 5-dimensional feature. We generate three training datasets of different scales, with  $n \in \{1000, 5000, 10000\}$ . The assessment will be made on a fixed test sample of 100 hold-out points that are independently drawn from the generating scheme of the event time. A censoring mechanism is applied such that the censoring ratio is around  $40\%$  for each dataset. The precise form of the frailty model as well as the generating distribution of the feature vectors are detailed in appendix C.2.
Empirical recovery results We report the empirical recovery of the nonlinear component  $\nu(t,Z)$  regarding the hold-out test set in in figure 1. We observe from graphical illustrations that under a moderate sample size  $n = 1000$ , NFM already exhibits satisfactory recovery for a (relatively low-dimensional) feature space, which is the prevailing case in most public benchmark datasets. We also present additional assessments about: (i) The recovery of  $m(Z)$  when using PF scheme in appendix D.1, (ii) The recovery of survival functions under both PF and FN scheme in appendix D.2, and (iii) The numerical recovery results of survival function in appendix D.3.
Table 1: Survival prediction results measured in IBS and INBLL metric (%) on four small-scale survival datasets. In each column, the boldfaced score denotes the best result and the underlined score represents the second-best result (both in mean).
Model	METABRIC	RotGBSG	FLCHAIN	SUPPORT
IBS	INBLL	IBS	INBLL	IBS	INBLL	IBS	INBLL
CoxPH	16.46±0.90	49.57±2.66	18.25±0.44	53.76±1.11	10.05±0.38	33.18±1.16	20.54±0.38	59.58±0.86
GBM	16.61±0.82	49.87±2.44	17.83±0.44	52.78±1.11	9.98±0.37	32.88±1.05	19.18±0.39	56.46±0.10
RSF	16.62±0.64	49.61±1.54	17.89±0.42	52.77±1.01	9.96±0.37	32.92±1.05	19.11±0.40	56.28±1.00
DeepSurv	16.55±0.93	49.85±3.02	17.80±0.49	52.62±1.25	10.09±0.38	33.28±1.15	19.20±0.41	56.48±1.08
CoxTime	16.54±0.83	49.67±2.67	17.80±0.58	52.56±1.47	10.28±0.45	34.18±1.53	19.17±0.40	56.45±1.10
DeepHit	17.50±0.83	52.10±2.16	19.61±0.38	56.67±1.10	11.83±0.39	37.72±1.02	20.66±0.32	60.06±0.72
DeepEH	16.56±0.65	49.42±1.53	17.62±0.52	52.08±1.27	10.11±0.37	33.30±1.10	19.30±0.39	56.67±0.94
SuMo-net	16.49±0.83	49.74±2.21	17.77±0.47	52.62±1.11	10.07±0.40	33.20±1.10	19.40±0.38	56.87±0.96
SODEN	16.52±0.63	49.39±1.97	17.05±0.63	50.45±1.97	10.13±0.24	33.37±0.57	19.07±0.50	56.15±1.35
SurvNode	16.67±1.32	49.73±3.89	17.42±0.53	51.70±1.16	10.40±0.29	34.37±1.03	19.58±0.34	57.49±0.84
DCM	16.58±0.87	49.48±2.23	17.66±0.54	52.26±1.23	10.13±0.50	33.40±1.38	19.29±0.42	56.68±1.09
DeSurv	16.71±0.75	49.61±2.15	17.98±0.46	53.23±1.15	10.06±0.62	33.18±1.93	19.50±0.40	57.28±0.89
NFM-PF	16.33±0.75	49.07±1.96	17.60±0.55	52.12±1.34	9.96±0.39	32.84±1.15	19.14±0.39	56.35±1.00
NFM-FN	16.11±0.81	48.21±2.04	17.66±0.52	52.41±1.22	10.05±0.39	33.11±1.10	18.97±0.60	55.87±1.50
5.2 Real-world data experiments
To answer RQ2, we conduct extensive empirical assessments over 6 benchmark datasets, comprising five survival datasets and one non-survival dataset. The survival datasets include the Molecular Taxonomy of Breast Cancer International Consortium (METABRIC) [16], the Rotterdam tumor bank and German Breast Cancer Study Group (RotGBSG)[43], the Assay Of Serum Free Light Chain (FLCHAIN) [20], the Study to Understand Prognoses Preferences Outcomes and Risks of Treatment (SUPPORT) [43], and the Medical Information Mart for Intensive Care (MIMIC-III) [39]. For all the survival datasets, the event of interest is defined as the mortality after admission. In our experiments, we view METABRIC, RotGBSG, FLCHAIN, and SUPPORT as small-scale datasets and MIMIC-III as a moderate-scale dataset. We additionally use the KKBOX dataset [46] as a large-scale evaluation. In this dataset, an event time is observed if a customer churns from the KKBOX platform. We summarize the basic statistics of all the datasets in table 3.
Baselines We compare NFM with 12 baselines. The first one is linear CoxPH model [14]. Gradient Boosting Machine (GBM) [27, 10] and Random Survival Forests (RSF) [37] are two tree-based nonparametric survival regression methods. DeepSurv [42] and CoxTime [46] are two models that adopt neural variants of partial likelihood as objectives. SuMo-net [56] is a neural variant of NHR. We additionally chose six latest state-of-the-art neural survival models: DeepHit [47], SurvNode [32], DeepEH [75], DCM [51], DeSurv [19] and SODEN [64]. Among the chosen baselines, DeepSurv and SuMo-net are viewed as implementations of neural CoxPH and neural NHR and are therefore of particular interest for the empirical verification of the efficacy of frailty.
Evaluation strategy We use two standard metrics in survival predictions for evaluating model performance: integrated Brier score (IBS) and integrated negative binomial log-likelihood (INBLL). Both metrics are derived from the following:
$$
\mathcal {S} (\ell , t _ {1}, t _ {2}) = \int_ {t _ {2}} ^ {t _ {1}} \frac {1}{n} \sum_ {i = 1} ^ {n} \left[ \frac {\ell (0 , \widehat {S} (t | Z _ {i})) I (T _ {i} \leq t , \delta_ {i} = 1)}{\widehat {S} _ {C} (T _ {i})} + \frac {\ell (1 , \widehat {S} (t | Z _ {i})) I (T _ {i} > t)}{\widehat {S} _ {C} (t)} \right] d t. \tag {11}
$$
Where  $\widehat{S}_C(t)$  is an estimate of the survival function  $S_{C}(t)$  of the censoring variable, obtained by the Kaplan-Meier estimate [41] of the censored observations on the test data.  $\ell :\{0,1\} \times [0,1]\mapsto \mathbb{R}^{+}$  is some proper loss function for binary classification [28]. The IBS metric corresponds to  $\ell$  being the square loss, and the INBLL metric corresponds to  $\ell$  being the negative binomial (Bernoulli) log-likelihood [30]. Both IBS and INBLL are proper scoring rules if the censoring times and survival times are independent. We additionally report the result of another widely used metric, the concordance index (C-index), in appendix D. Since all the survival datasets do not have standard
5. Otherwise, one may pose a covariate-dependent model on the censoring time and use  $\widehat{S}_C(t|Z)$  instead of  $\widehat{S}_C(t)$ . We adopt the Kaplan-Meier approach since it's still the prevailing practice in evaluations of survival predictions.
Table 2: Survival prediction results measured in IBS and INBLL metric (\%) on two larger datasets. In each column, the boldfaced score denotes the best result and the underlined score represents the second-best result (both in mean). Two models are not reported, namely SODEN and DeepEH, as we found empirically that their computational/memory cost is significantly worse than the rest, and we fail to obtain reasonable performances over the two datasets for these two models.
Model	MIMIC-III	KKBOX
IBS	INBLL	IBS	INBLL
CoxPH	20.40±0.00	60.02±0.00	12.60±0.00	39.40±0.00
GBM	17.70±0.00	52.30±0.00	11.81±0.00	38.15±0.00
RSF	17.79±0.19	53.34±0.41	14.46±0.00	44.39±0.00
DeepSurv	18.58±0.92	55.98±2.43	11.31±0.05	35.28±0.15
CoxTime	17.68±1.36	52.08±3.06	10.70±0.06	33.10±0.21
DeepHit	19.80±1.31	59.03±4.20	16.00±0.34	48.64±1.04
SuMo-net	18.62±1.23	54.51±2.97	11.58±0.11	36.61±0.28
DCM	18.02±0.49	52.83±0.94	10.71±0.11	33.24±0.06
DeSurv	18.19±0.65	54.69±2.83	10.77±0.21	33.22±0.10
NFM-PF	16.28±0.36	49.18±0.92	11.02±0.11	35.10±0.22
NFM-FN	17.47±0.45	51.48±1.23	10.63±0.08	32.81±0.14
train/test splits, we follow previous practice [75] that uses 5-fold cross-validation (CV): 1 fold is for testing, and  $20\%$  of the rest is held out for validation. In our experiments, we observed that a single random split into 5 folds does not produce stable results for most survival datasets. Therefore we perform 10 different CV runs for each survival dataset and report average metrics as well as their standard deviations. For the KKBOX dataset, we use the standard train/valid/test splits that are available via the pycox package [46] and report results based on 10 trial runs.
Experimental setup We follow standard preprocessing strategies [42, 46, 75] that standardize continuous features into zero mean and unit variance, and do one-hot encodings for all categorical features. We adopt MLP with ReLU activation for all function approximators, including  $\hat{h}$ ,  $\hat{m}$  in PF scheme, and  $\hat{\nu}$  in FN scheme, across all datasets, with the number of layers (depth) and the number of hidden units (width) within each layer being tunable. We tune the frailty transform over several standard choices: gamma frailty, Box-Cox transformation frailty and  $\mathrm{IGG}(\alpha)$  frailty, with their precise forms detailed in appendix C.3. A more detailed description of the tuning procedure, as well as training configurations for baseline models, are reported in appendix C.3.
Results We report experimental results of small-scale datasets in table 1, and results of two larger datasets in table 2. The proposed NFM framework achieves competitive performance which is comparable to the other state-of-the-art models. In particular, NFM attains best performance in mean on 5 of the 6 datasets, and is statistically significantly better over all the baselines at 0.05 empirical level on the MIMIC-III dataset.
Ablation on the benefits of frailty To better understand the additional benefits of introducing the frailty formulation, we compute the (relative) performance gain of NFM-PF and NFM-FN, against their non-frailty counterparts, namely DeepSurv [42] and SuMo-net [56]. The evaluation is conducted for all three metrics mentioned in this paper. The results are shown in table 6. The results suggest a solid improvement in incorporating frailty, as the relative increase in performance could be over  $10\%$  for both NFM models. A more detailed discussion is presented in section D.5.
6 Discussion and conclusion
We have introduced NFM as a flexible and powerful neural modeling framework for survival analysis, which is shown to be both statistically correct in theory, and empirically effective in predictive tasks. While our proposed framework provides a theoretically-principled tool of neural survival modeling, a few limitations and challenges need to be addressed in future works including predictive guarantees and better evaluation protocols, which we elaborate in appendix D.6.
We would like to thank professor Zhiliang Ying and professor Guanhua Fang for helpful discussions. Wen Yu's research is supported by the National Natural Science Foundation of China Grants (12071088). Ming Zheng's research is supported by the National Natural Science Foundation of China Grants (12271106).
[66] A. van der Vaart, A. van der Vaart, A. van der Vaart, and J. Wellner. Weak Convergence and Empirical Processes: With Applications to Statistics. Springer Series in Statistics. Springer, 1996.
[67] A. W. Van der Vaart. Asymptotic statistics, volume 3. Cambridge university press, 2000.
[68] A. Wehenkel and G. Louppe. Unconstrained monotonic neural networks. Advances in neural information processing systems, 32, 2019.
[69] A. Wienke. Frailty models in survival analysis. Chapman and Hall/CRC, 2010.
[70] W. H. Wong and X. Shen. Probability inequalities for likelihood ratios and convergence rates of sieve mles. The Annals of Statistics, pages 339-362, 1995.
[71] W. C.-H. Wu, M.-Y. Yeh, and M.-S. Chen. Predicting winning price in real time bidding with censored data. In Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 1305-1314, 2015.
[72] D. Yarotsky. Error bounds for approximations with deep relu networks. Neural Networks, 94:103-114, 2017.
[73] Z. Ying. A large sample study of rank estimation for censored regression data. The Annals of Statistics, pages 76-99, 1993.
[74] D. Zeng and D. Lin. Efficient estimation of semiparametric transformation models for counting processes. Biometrika, 93(3):627-640, 2006.
[75] Q. Zhong, J. W. Mueller, and J.-L. Wang. Deep extended hazard models for survival analysis. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 15111-15124. Curran Associates, Inc., 2021.
[76] Q. Zhong, J. W. Mueller, and J.-L. Wang. Deep learning for the partially linear cox model. The Annals of Statistics, 2021.
A Examples of frailty specifications
We list several commonly used frailty models, and specify their corresponding characteristics via their frailty transform  $G_{\theta}$ :
Gamma frailty: Arguably the gamma frailty is the most widely used frailty model [48, 49, 53, 69, 21], with
$$
G _ {\theta} (x) = \frac {1}{\theta} \log (1 + \theta x), \theta \geq 0. \tag {12}
$$
When  $\theta = 0$ ,  $G_0(x) = \lim_{\theta \to 0} G_\theta(x)$  is defined as the (pointwise) limit. A notable fact of the gamma frailty specification is that when the proportional frailty (PF) assumption (2) is met, if  $\theta = 0$ , the model degenerates to CoxPH. Otherwise if  $\theta = 1$ , the model corresponds to the proportional odds (PO) model [3].
Box-Cox transformation frailty: Under this specification, we have
$$
G _ {\theta} (x) = \frac {(1 + x) ^ {\theta} - 1}{\theta}, \theta \geq 0. \tag {13}
$$
The case of  $\theta = 0$  is defined analogously to that of gamma frailty, which corresponds to the PO model under the PF assumption. When  $\theta = 1$ , the model reduces to CoxPH under the PF assumption.  $\mathbf{IGG}(\alpha)$  frailty: This is an extension of gamma frailty [45] and includes other types of frailty specifications like the inverse gaussian frailty [35], with
$$
G _ {\theta} (x) = \frac {1 - \alpha}{\alpha \theta} \left[ \left(1 + \frac {\theta x}{1 - \alpha}\right) ^ {\alpha} - 1 \right], \theta \geq 0, \alpha \in [ 0, 1). \tag {14}
$$
In the one-dimensional parameter paradigm, the parameter  $\alpha$  is assumed known instead of being learnable. When  $\alpha = 1/2$ , we obtain the gamma frailty model. When  $\alpha \to 0$ , the limit corresponds to the inverse Gaussian frailty.
Statistability of regularity condition 4.5 In [45, Proposition 1], the authors verified the regularity condition of gamma and  $\mathrm{IGG}(\alpha)$  frailties. Using a similar argument, it is straightforward to verify the regularity of Box-Cox transformation frailty.
B Proofs of theorems
B.1 Preliminary
Additional definitions The theory of empirical processes [66] will be involved heavily in the proof. Therefore we briefly introduce some common notations: For a function class  $\mathcal{F}$ , define  $N(\epsilon, \mathcal{F}, \|\cdot\|)$  to be the covering number of  $\mathcal{F}$  with respect to norm  $\|\cdot\|$  under radius  $\epsilon$ , and define  $N_{\perp}(\epsilon, \mathcal{F}, \|\cdot\|)$  to be the bracketing number of  $\mathcal{F}$  with respect to norm  $\|\cdot\|$  under radius  $\epsilon$ . We use VC  $(\mathcal{F})$  to denote the VC-dimension of  $\mathcal{F}$ . Moreover, we use the notation  $a \lesssim b$  to denote  $a \leq Cb$  for some positive constant  $C$ .
Before proving theorem 4.6 and 4.7, we introduce some additional notations that will be useful throughout the proof process.
In the PF scheme, define
$$
\begin{array}{l} l (T, \delta , Z; h, m, \theta) = \delta \log g _ {\theta} \left(e ^ {m (Z)} \int_ {0} ^ {T} e ^ {h (s)} d s\right) + \delta h (T) + \delta m (Z) \\ - G _ {\theta} \left(e ^ {m (Z)} \int_ {0} ^ {T} e ^ {h (s)} d s\right), \\ \end{array}
$$
where we denote  $g_{\theta} = G'(\theta)$ . Under the definition of the sieve space stated in condition 4.3, we restate the parameter estimates as
$$
\left(\widehat {h} _ {n}, \widehat {m} _ {n}, \widehat {\theta} _ {n}\right) = \underset {\widehat {h} \in \mathcal {H} _ {n}, \widehat {m} \in \mathcal {M} _ {n}, \theta \in \Theta} {\operatorname {a r g m a x}} \frac {1}{n} \sum_ {i \in [ n ]} l (T _ {i}, \delta_ {i}, Z _ {i}; \widehat {h}, \widehat {m}, \theta).
$$
Similarly, in the FN scheme, we define
$$
l (T, \delta , Z; \nu , \theta) = \delta \log g _ {\theta} \left(\int_ {0} ^ {T} e ^ {\nu (s, Z)} d s\right) + \delta \nu (T, Z) - G _ {\theta} \left(\int_ {0} ^ {T} e ^ {\nu (s, Z)} d s\right)
$$
Under the definition of the sieve space stated in condition 4.4, we restate the parameter estimates as
$$
\left(\widehat {\nu} _ {n} (t, z), \widehat {\theta} _ {n}\right) = \underset {\widehat {\nu} \in \mathcal {V} _ {n}, \theta \in \Theta} {\operatorname {a r g m a x}} \frac {1}{n} \sum_ {i \in [ n ]} l (T _ {i}, \delta_ {i}, Z _ {i}; \widehat {\nu}, \theta).
$$
We denote the conditional density function and survival function of the event time  $\tilde{T}$  given  $Z$  by  $f_{\tilde{T} |Z}(t)$  and  $S_{\tilde{T} |Z}(t)$ , respectively. Similarly, we denote the conditional density function and survival function of the censoring time  $C$  given  $Z$  by  $f_{C|Z}(t)$  and  $S_{C|Z}(t)$ . Under the assumption that  $\tilde{T} \perp C \mid Z$ , the joint conditional density of the observed time  $T$  and the censoring indicator  $\delta$  given  $Z$  can be expressed as the following:
$$
\begin{array}{l} {p (T, \delta \mid Z)} = {f _ {\tilde {T} \mid Z} (T) ^ {\delta} S _ {\tilde {T} \mid Z} (T) ^ {1 - \delta} f _ {C \mid Z} (T) ^ {1 - \delta} S _ {C \mid Z} (T) ^ {\delta}} \\ { = } { \lambda _ { \tilde { T } | Z } ( T ) ^ { \delta } S _ { \tilde { T } | Z } ( T ) f _ { C | Z } ( T ) ^ { 1 - \delta } S _ { C | Z } ( T ) ^ { \delta } , } \\ \end{array}
$$
where  $\lambda_{\tilde{T} |Z}(T)$  is the conditional hazard function of the survival time  $\tilde{T}$  given  $Z$ .
Under the model assumption of PF scheme,  $p(T,\delta \mid Z)$  can be expressed by
$$
p (T, \delta \mid Z; h, m, \theta) = \exp \left(l (T, \delta , Z; h, m, \theta)\right) f _ {C | Z} (T) ^ {1 - \delta} S _ {C | Z} (T) ^ {\delta}.
$$
For  $\phi_0 = (h_0, m_0, \theta_0)$  and an estimator  $\widehat{\phi} = (\widehat{h}, \widehat{m}, \widehat{\theta})$ , the defined distance  $d_{\mathsf{PF}}(\widehat{\phi}, \phi_0)$  can be explicitly expressed by
$$
d _ {\mathsf {F N}} \left(\widehat {\psi}, \psi_ {0}\right) = \sqrt {\mathbb {E} _ {Z} \left[ \int \left| \sqrt {p (T , \delta \mid Z ; \widehat {h} , \widehat {m} , \widehat {\theta})} - \sqrt {p (T , \delta \mid Z ; h _ {0} , m _ {0} , \theta_ {0})} \right| ^ {2} \mu (d T \times d \delta) \right]}.
$$
Here the dominating measure  $\mu$  is defined such that for any (measurable) function  $r(T,\delta)$
$$
\int r (T, \delta) \mu (d T \times d \delta) = \int_ {0} ^ {\tau} r (T, \delta = 1) d T + \int_ {0} ^ {\tau} r (T, \delta = 0) d T
$$
Under the model assumption of FN scheme,  $p(T,\delta \mid Z)$  can be expressed by
$$
p (T, \delta \mid Z; \nu , \theta) = \exp \left(l (T, \delta , Z; \nu , \theta)\right) f _ {C | Z} (T) ^ {1 - \delta} S _ {C | Z} (T) ^ {\delta}.
$$
For  $\psi_0 = (\nu_0,\theta_0)$  and an estimator  $\widehat{\psi} = (\widehat{\nu},\widehat{\theta})$ , the defined distance  $d_{\mathsf{FN}}\left(\widehat{\psi},\psi_0\right)$  can be explicitly expressed by
$$
d _ {\mathsf {F N}} \left(\widehat {\psi}, \psi_ {0}\right) = \sqrt {\mathbb {E} _ {Z} \left[ \int \left| \sqrt {p (T , \delta \mid Z ; \widehat {\nu} , \widehat {\theta})} - \sqrt {p (T , \delta \mid Z ; \nu_ {0} , \theta_ {0})} \right| ^ {2} \mu (d T \times d \delta) \right]}.
$$
B.2 Technical lemmas