SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Larry Han
Department of Health Sciences
lar.han@northeastern.edu
Zhu Shen
zhushen@g.harvard.edu
Jose R. Zubizarreta
Departments of Health Care Policy, Biostatistics, and Statistics
zubizarreta@hcp.med.harvard.edu
Federated or multi-site studies have distinct advantages over single-site studies, including increased generalizability, the ability to study underrepresented populations, and the opportunity to study rare exposures and outcomes. However, these studies are complicated by the need to preserve the privacy of each individual's data, heterogeneity in their covariate distributions, and different data structures between sites. We propose a novel federated approach to derive valid causal inferences for a target population using multi-site data. We adjust for covariate shift and accommodate covariate mismatch between sites by developing a multiply-robust and privacy-preserving nuisance function estimation approach. Our methodology incorporates transfer learning to estimate ensemble weights to combine information from source sites. We show that these learned weights are efficient and optimal under different scenarios. We showcase the finite sample advantages of our approach in terms of efficiency and robustness compared to existing state-of-the-art approaches. We apply our approach to study the treatment effect of percutaneous coronary intervention (PCI) on the duration of hospitalization for patients experiencing acute myocardial infarction (AMI) with data from the Centers for Medicare & Medicaid Services (CMS).
Compared to single-site studies, federated or multi-site studies confer distinct advantages, such as the potential for increased generalizability of findings, the opportunity to learn about underrepresented populations, and the ability to study rare exposures and outcomes. However, deriving valid causal inferences using multi-site data is difficult due to numerous real-world challenges, including heterogeneity of site populations, different data structures, and privacy-preserving constraints stemming from policies such as the General Data Protection Regulation (GDPR) and Health Insurance Portability and Accountability Act (HIPAA) that prohibit direct data pooling.
Recent methodological developments have focused on privacy-preserving estimation strategies. These strategies typically involve sharing summary-level information from multiple data sources [28, 29, 13, 14, 20]. However, they often require restrictive assumptions such as homogeneous data
structures and model specifications (e.g., a common set of observed covariates measured using a common data model), which are not realistic in practice.
To address these methodological gaps, we propose a multiply robust and privacy-preserving estimator that leverages multi-site information to estimate causal effects in a target population of interest. Compared to existing approaches, our method allows investigators from different sites to incorporate site-specific covariate information and domain knowledge and provides increased protection against model misspecification. Our method allows for flexible identification under different settings, including systematically missing covariates and different site-specific covariates (termed covariate mismatch). Our proposed method adopts an adaptive ensembling approach that optimally combines estimates from source sites and serves as a data-driven metric for the transportability of source sites. Moreover, the proposed method relaxes the assumption of homogeneous model specifications by adopting a class of multiply robust estimators for estimating the nuisance functions.
1.1 Related Work and Contributions
The current literature on multi-site causal inference typically assumes that a common set of confounders is observed in all sites [8, 7, 13, 14]. However, this assumption is rarely met due to variations in local practices, e.g., differing data collection standards and coding practices. In particular, the target site often lacks data on certain covariates available in the source sites, and ignoring them can result in biased and inefficient inference [32]. Recently, [30] proposed a method to address covariate mismatch by integrating source samples with unmeasured confounders and a target sample containing information about these confounders. However, they assumed that the target and source samples are obtained from the same population. [12] extended the method of [30] to the setting where the average treatment effect (ATE) is not identifiable in some sites by constructing control variates. However, their approach is focused on addressing selection biases where preferential selection of units solely depends on the binary outcome, and it is not obvious how one could extend the method to other settings. [32] extended the framework by [8, 7] to handle covariate mismatch by regressing predicted conditional outcomes on effect modifiers and then taking the means of these regression models evaluated on target site samples. Our work leverages ideas from [32] to a more general multi-site federated data setting by utilizing an adaptive weighting approach that optimally combines estimates from source sites.
Most existing approaches in the generalizability and transportability literature deal with heterogeneous covariate distributions by modeling the site selection processes [6, 2, 27, 8, 7]. These approaches, which incorporate inverse probability of selection weights [6, 2], stratification [27], and augmentation [8, 7], involve pooling individual-level information across sites. Our work differs from those in that we preserve individual data privacy, sharing only summary-level information from the target site. Specifically, we adopt density ratio models [24, 9] that only share covariate moments of the target samples. Under certain specifications, these density ratio models are equivalent to logistic regression-based selection models for adjusting heterogeneity between target and source populations. Our approach shares similarities with calibration weighting methods but leverages semi-parametric efficiency theory to enable a closed-form approximation of the variance.
Further, when data sources are heterogeneous, it would be beneficial for investigators at different sites to incorporate site-specific knowledge when specifying candidate models. However, to the best of our knowledge, existing methods require common models to be specified across sites, which may not be realistic or flexible enough [28, 29, 13, 14, 20]. In contrast, this paper takes a different stance; we accommodate variations in outcome and treatment models across sites using a multiply robust estimator, instead of imposing a uniform model. Our work builds on [22], which established an equivalence between doubly robust and multiply robust estimators using mixing weights determined by predictive risks of candidate models [17, 15, 16, 3, 4, 5]. We take advantage of this equivalent form to obtain closed-form expressions for the variance of our federated global estimator. Compared to [32], where they consider a single source site and a single target site, we are able to consider multiple source sites and multiple target sites. To prevent the negative transfer, we combine the estimates from multiple sites via a data-adaptive ensembling approach first proposed in [13].
We consider data from  $K$  sites, where each site has access to its individual-level data but is prohibited from sharing this data with any other site. The set of sites will be denoted by  $\mathcal{K} = \{1,2,\dots,K\}$ . We index the target site with  $T$  and the source sites as the remaining sites, i.e.,  $S = \mathcal{K}\setminus T$ .
For each individual  $i$ , let  $Y_{i}$  denote an observed outcome, which can be continuous or discrete.  $X_{i} \in \mathbb{R}^{p}$  represents the  $p$ -dimensional baseline covariates in source site  $k \in S$ .  $V_{i} \in \mathbb{R}^{q}$  represents the (partial) baseline covariates in the target site  $T$  such that  $V_{i} \subseteq X_{i}$ . To simplify the presentation, we assume an identical set of covariates across all source sites, although our method can accommodate scenarios where distinct covariate sets are present among the source sites. Let  $A_{i}$  represent a binary treatment indicator, with  $A_{i} = 1$  denoting treatment and  $A_{i} = 0$  denoting control.  $R_{i}$  is a site indicator with  $R_{i} = k$  if patient  $i$  is from the site  $k$ . We observe  $n_{T}$  target observations,  $D_{T} = \{Y_{i}, V_{i}, A_{i}, R_{i} = T, 1 \leq i \leq n_{T}\}$  and  $n_{k}$  source observations,  $D_{k} = \{Y_{i}, X_{i}, A_{i}, R_{i} = k, 1 \leq i \leq n_{k}\}$  for each  $k \in S$ . The total sample size is  $N = \sum_{k \in \mathcal{K}} n_{k}$ . Under the potential outcomes framework [23, 25], we denote the counterfactual outcomes under treatment and control as  $\{Y_{i}(1), Y_{i}(0)\}$ , and only one of them is observed:  $Y_{i} = A_{i} Y_{i}(1) + (1 - A_{i}) Y_{i}(0)$  [26]. The data structure is illustrated in Figure 1.
Figure 1: Schematic of the data structure in the multi-site setting.
Our goal is to estimate the target average treatment effect (TATE),
$$
\Delta_ {T} = \mu_ {1, T} - \mu_ {0, T} \quad \text {w h e r e} \quad \mu_ {a, T} = E \left\{Y _ {i} (a) \mid R _ {i} = T \right\} \text {f o r} a \in \{0, 1 \}, \tag {1}
$$
where  $\mu_{a,T}$  is the mean potential outcome under treatment  $a$  in the target population. To identify this quantity, we consider the following assumptions:
(A1) (Consistency): For every individual  $i$ , if  $A_{i} = a$ , then  $Y_{i} = Y_{i}(a)$ .
(A2) (Mean exchangeability over treatment assignment in the target population):  $E\{Y_{i}(a)\mid V_{i} = v,A_{i},R_{i} = T\} = E\{Y_{i}(a)\mid V_{i} = v,R_{i} = T\}$ .
(A3) (Positivity of treatment assignment in the target population):  $0 < P(A_{i} = 1 \mid V_{i} = v, R_{i} = T) < 1$  for any  $v$  s.t.  $P(V_{i} = v \mid R_{i} = T) > 0$ .
(A4) (Mean exchangeability over treatment assignment in the source populations):  $E\{Y_{i}(a)\mid X_{i} = x,A_{i},R_{i} = k\} = E\{Y_{i}(a)\mid X_{i} = x,R_{i} = k\} ,k\in \mathcal{S}.$
(A5) (Positivity of treatment assignment in the source populations):  $0 < P(A_{i} = 1 \mid X_{i} = x, R_{i} = k) < 1$  for any  $x$  s.t.  $P(X_{i} = x \mid R_{i} = k) > 0, k \in S$ .
(A6) (Mean exchangeability over site selection):  $E\{Y_{i}(a)\mid V_{i} = v,R_{i} = k\} = E\{Y_{i}(a)\mid V_{i} = v\} ,k\in \mathcal{K}.$
(A7) (Positivity of site selection):  $0 < P(R_{i} = k \mid V_{i} = v) < 1$  for  $k \in S$  and any  $v$  s.t.  $P(V_{i} = v) > 0$ .
Assumption (A1) is the stable unit treatment value assumption (SUTVA), requiring no interference between individuals. Assumption (A2) (Assumption (A4)) states that the mean counterfactual outcome under treatment  $a$  is independent of treatment assignment, conditional on baseline covariates in the target (source) populations. For Assumption (A2) and (A4) to hold, we require all effect modifiers to be measured in  $V$ . Assumption (A3) (Assumption (A5)) states that each individual in the
target (source) populations has a positive probability of receiving each treatment. Assumption (A6) states that the mean counterfactual outcome is independent of site selection, conditional on covariates in the target population. For Assumption (A6) to hold, we require all covariates that are distributed differently between target and source populations (shifted covariates) to be measured in  $V$ . Thus, if these effect modifiers are measured in  $V$ , Assumption (A2), (A4) and (A6) automatically hold. Assumption (A7) requires that in each stratum defined by  $V$ , the probability of being in a source population for each individual is positive. Theorem 1 shows that under Assumption (A1), (A4) - (A7), the mean counterfactual outcome for the target can be identified in the sources.
Theorem 1. If Assumptions (A1) - (A3) hold, the mean counterfactual outcomes in the target population can be identified using the target sample.
$$
\mu_ {a, T} = E \left\{Y _ {i} (a) \mid R _ {i} = T \right\} = E \left\{E \left\{Y _ {i} \mid V _ {i} = v, A _ {i} = a, R _ {i} = T \right\} \mid R _ {i} = T \right\}. \tag {2}
$$
If Assumptions (A1), (A4) - (A7) hold, the mean counterfactual outcomes in the target population can be identified using the source samples.
$$
\begin{array}{l} \mu_ {a, T} = E \left\{Y _ {i} (a) \mid R _ {i} = T \right\} \\ = E \left\{E \left\{E \left\{Y _ {i} \mid X _ {i} = x, A _ {i} = a, R _ {i} = k \right\} \mid V _ {i} = v, R _ {i} = k \right\} \mid R _ {i} = T \right\}. \tag {3} \\ \end{array}
$$
3 Site-specific Estimation
For the target site  $k = \{T\}$ , a standard augmented inverse propensity weighted (AIPW) estimator is used for  $\mu_{a,T}$  as follows
$$
\widehat {\mu} _ {a, T} = \frac {1}{n _ {T}} \sum_ {i = 1} ^ {n} \left[ \frac {I \left(A _ {i} = a , R _ {i} = T\right)}{\widehat {\pi} _ {a , T} (V _ {i})} \left\{Y _ {i} - \widehat {m} _ {a, T} (V _ {i}) \right\} + \widehat {m} _ {a, T} (V _ {i}) \right], \tag {4}
$$
where  $\widehat{m}_{a,T}(V_i)$  is an estimator for  $E\{Y_i \mid V_i = v, A_i = a, R_i = T\}$ , the outcome model in the target population, and  $\widehat{\pi}_{a,T}(V_i)$  is an estimator for  $P(A_i = 1 \mid V_i = v, R_i = T)$ , the probability of receiving treatment  $a$  in the target population.
For each source site  $k \in S$ , we propose an estimator for  $\mu_{a,T}$  as follows
$$
\begin{array}{l} \widehat {\mu} _ {a, k} = \frac {1}{n _ {k}} \sum_ {i = 1} ^ {n} \left[ \frac {I (A _ {i} = a , R _ {i} = k)}{\widehat {\pi} _ {a , k} (X _ {i})} \widehat {\zeta} _ {k} (V _ {i}) \Big \{Y _ {i} - \widehat {m} _ {a, k} (X _ {i}) \Big \} \right] \\ + \frac {1}{n _ {k}} \sum_ {i = 1} ^ {n} \left[ I (R _ {i} = k) \widehat {\zeta} _ {k} (V _ {i}) \left\{\widehat {m} _ {a, k} \left(X _ {i}\right) - \widehat {\tau} _ {a, k} (V _ {i}) \right\} \right] + \frac {1}{n _ {T}} \sum_ {i = 1} ^ {n} I (R _ {i} = T) \widehat {\tau} _ {a, k} (V _ {i}), \tag {5} \\ \end{array}
$$
where  $\widehat{\tau}_{a,k}(V_i)$  is an estimator for  $E\{m_{a,k}(x) \mid V_i = v, R_i = k\}$  and  $\widehat{m}_{a,k}(X_i)$  is an estimator for  $E\{Y_i \mid X_i = x, A_i = a, R_i = k\}$ .  $\widehat{\zeta}_k(V_i)$  estimates  $f(V_i \mid R_i = T) / f(V_i \mid R_i = k)$ , the density ratios of covariate distributions in the target population  $T$  and source population  $k \in S$ .  $\widehat{\pi}_{a,k}(X_i)$  estimates  $P(A_i = 1 \mid X_i = x, R_i = k)$ , the probability of receiving treatment  $a$  in source  $k \in S$ . The estimators (4) and (5) are derived leveraging semi-nonparametric theory. They can be seen as a functional Taylor expansion of the target functional  $\mu_{a,T}$ . This method is commonly used in causal inference to correct for the bias of the plug-in estimators. These resulting estimators exhibit favorable statistical properties, even when slow-converging non-parametric techniques are used for nuisance estimation. For a more comprehensive discussion, interested readers can refer to [21] and [18].
Compared to the transportation estimators in [8, 7], we introduce two additional nuisance functions,  $\zeta_k(V_i)$  and  $\tau_{a,k}(V_i)$ . Specifically,  $\zeta_k(V_i)$  accounts for covariate shift across sites while preserving individual's data privacy, while  $\tau_{a,k}(V_i)$  is introduced to address covariate mismatch across sites. We provide estimation procedures for these nuisance functions in the following subsections, and the theoretical guarantees of the estimator are presented in Section 5.
3.1 Density Ratio Weighting
Most existing methods adjust for covariate shift across sites by relying on inverse probability of selection weighting, which requires pooling target and source samples. However, such pooling is
often not possible due to data privacy regulations. We consider a density ratio weighting approach, which offers equivalent estimation without the need for direct data pooling (see Appendix A).
Formally, as in [13], we model the density ratios of covariate distributions in the target  $T$  and source  $k \in S$  by specifying an exponential tilt model [24, 9];  $\zeta_k(V_i; \theta_k) = f(V_i \mid R_i = T) / f(V_i \mid R_i = k) = \exp\left\{-\theta_k^\top \psi(V_i)\right\}$  where  $f(V_i \mid R_i = T)$  and  $f(V_i \mid R_i = k)$  are density functions of covariates  $V_i$  in the target  $T$  and source  $k \in S$ , respectively, and  $\psi(V_i)$  is some  $d$ -dimensional basis with 1 as its first element. With this formulation,  $\zeta_k(V_i; \theta_k) = 1$  for  $\theta_k = 0$  and  $\int \zeta_k(V_i; \theta_k) f(V_i \mid R_i = k) dx = 1$ . If we choose  $\psi(V_i) = V_i$ , we can recover the entire class of natural exponential family distributions. If we include higher-order terms, the exponential tilt model has greater flexibility in characterizing the heterogeneity between two populations [10]. We solve for  $\widehat{\theta}_k$  with the following estimating equation
$$
\frac {1}{n _ {T}} \sum_ {i = 1} ^ {N} I (R _ {i} = T) \psi (V _ {i}) = \frac {1}{n _ {k}} \sum_ {i = 1} ^ {N} I (R _ {i} = k) \psi (V _ {i}) \exp \left\{- \theta_ {k} ^ {\top} \psi (V _ {i}) \right\}. \tag {6}
$$
This procedure preserves individual privacy; choosing  $\psi(V_i) = V_i$ , the target site only needs to share its covariate means with the source sites; each source site then solves (6) with its own data to obtain the density ratios.
3.2 Multiply Robust Estimation
We relax the assumption of homogeneous model specifications across sites and allow each site to propose multiple models for nuisance functions. Our proposal follows the construction of multiply robust estimators for nuisance functions via a model-mixing approach [22].
Formally, for each site  $k \in \mathcal{K}$ , we consider a set of  $J$  candidate treatment models for the propensity scores  $\{\pi_{a,k}^j(x): j \in \mathcal{J} = \{1, \dots, J\}\}$ . Let  $\widehat{\pi}_{a,k}^j(x)$  be the estimator of  $\pi_{a,k}^j(x)$  obtained by fitting the corresponding candidate models on the data, which can be parametric, semiparametric, or nonparametric machine learning models.  $\widehat{\pi}_{a,k}(X_i) = \sum_{j=1}^{J} \widehat{\Lambda}_j \widehat{\pi}_{a,k}^j(X_i)$  denotes the weighted predictions of propensity scores, with weights  $\widehat{\Lambda}_j$  assigned to predictions by each candidate model  $j$ . To calculate the weights  $\widehat{\Lambda}_j$ , we adapt a model-mixing algorithm developed in [31] and [22] based on the cumulative predictive risks of candidate models.
First, we randomly partition the data within each site into a training set  $D_{k}^{\mathrm{train}}$  of units indexed by  $\{1,\dots,n_k^{\mathrm{train}}\}$  and a validation set  $D_{k}^{\mathrm{val}}$  of units indexed by  $\{n_k^{\mathrm{train}} + 1,\dots,n_k\}$ . Then, each candidate treatment model is fit on  $D_{k}^{\mathrm{train}}$  to obtain  $\widehat{\pi}_{a,n_k^{\mathrm{train}}}^j$  for  $j\in \mathcal{J}$ . The model-mixing weights are determined by the models' predictive risks assessed on  $D_{k}^{\mathrm{val}}$  according to the Bernoulli likelihood. Specifically,
$$
\widehat {\Lambda} _ {j} = \left(n _ {k} - n _ {k} ^ {\text {t r a i n}}\right) ^ {- 1} \sum_ {i = n _ {k} ^ {\text {t r a i n}} + 1} ^ {n _ {k}} \widehat {\Lambda} _ {j, i} \quad \text {a n d}
$$
$$
\widehat {\Lambda} _ {j, i} = \frac {\prod_ {q = n _ {k} ^ {\text {t r a i n}} + 1} ^ {i - 1} \widehat {\pi} _ {a , n _ {k} ^ {\text {t r a i n}}} ^ {j} \left(X _ {q}\right) ^ {A _ {q}} \left\{1 - \widehat {\pi} _ {a , n _ {k} ^ {\text {t r a i n}}} ^ {j} \left(X _ {q}\right) \right\} ^ {1 - A _ {q}}}{\sum_ {j ^ {\prime} = 1} ^ {J} \prod_ {q = n _ {k} ^ {\text {t r a i n}} + 1} ^ {i - 1} \widehat {\pi} _ {a , n _ {k} ^ {\text {t r a i n}}} ^ {j ^ {\prime}} \left(X _ {q}\right) ^ {A _ {q}} \left\{1 - \widehat {\pi} _ {a , n _ {k} ^ {\text {t r a i n}}} ^ {j ^ {\prime}} \left(X _ {q}\right) \right\} ^ {1 - A _ {q}}} \quad \text {f o r} \quad n _ {k} ^ {\text {t r a i n}} + 2 \leq i \leq n _ {k}, \tag {7}
$$
where  $\widehat{\Lambda}_{j,n_{k}^{\mathrm{train}} + 1} = 1 / J$ . The model-mixing estimators are consistent if one of the  $j \in \mathcal{J}$  candidate models is correctly specified [22]. A similar strategy can be used for conditional outcomes  $m_{a,k}(X_i)$  by combining a set of  $L$  candidate outcome models  $\{m_{a,k}^{l}(x): l \in \mathcal{L} = \{1, \dots, L\}\}$ . We obtain  $\widehat{m}_{a,k}(X_i) = \sum_{l=1}^{L} \widehat{\Omega}_l \widehat{m}_{a,k}^{l}(X_i)$  as the predicted outcomes with weights  $\widehat{\Omega}_l$  of candidate outcomes models under treatment  $a$  in site  $k$ . Further details are provided in Appendix B.
3.3 Covariate Mismatch
To account for covariate mismatch, we adapt the approach in [32], introducing the nuisance function  $\tau_{a,k}(V_i) = E\{m_{a,k}(x) \mid V_i = v, R_i = k\}$ , where  $m_{a,k}(x)$  is the outcome regression for treatment  $a$
in site  $k$ . First, we estimate  $m_{a,k}(X_i)$  by regressing the outcome  $Y_i$  on covariates  $X_i$  among units receiving treatment  $a$  in site  $k$ . We then regress  $\widehat{m}_{a,k}(X_i)$ , the estimates from the previous step, on  $V_i$  in the source site  $k$  to obtain  $\widehat{\tau}_{a,k}(x)$ . By doing so, we project all site-specific estimates of conditional outcomes to a common hyperplane defined by  $V_i$ . If all effect modifiers that are distributed differently between target and source populations are measured in  $V_i$ , then the information contained in the projected site-specific estimates can be transported to the target site. Finally, we take the mean of  $\widehat{\tau}_{a,k}(x)$  over the target sample, which gives us the transported estimate  $\widehat{\tau}_{a,k}(V_i)$  for the mean counterfactual outcomes under treatment  $a$  in the target population.
4 Federated Estimation
Let  $\widehat{\mu}_{a,T}$  denote the estimate of  $\mu_{a,T}$  based on target data only and  $\widehat{\mu}_{a,k}$  be the estimates of  $\mu_{a,T}$  using source data  $k\in S$ . We propose a general form of the federated global estimator as follows
$$
\widehat {\mu} _ {a, G} = \widehat {\mu} _ {a, T} + \sum_ {k \in \mathcal {K}} \widehat {\eta} _ {k} \left\{\widehat {\mu} _ {a, k} - \widehat {\mu} _ {a, T} \right\}, \tag {8}
$$
where  $\widehat{\eta}_k \geq 0$  is a non-negative weight assigned to site-specific estimates and  $\sum_{k \in \mathcal{K}} \widehat{\eta}_k = 1$ . The role of  $\eta_k$  is to determine the ensemble weight given to the site-specific estimates. We can employ diverse weighting methods by selecting appropriate values of  $\eta_k$ . For example, if  $\eta_k = 0$ , the global estimator is simply the estimator based on target data only; if  $\eta_k = n_k / N$ , the global estimator combines site-specific estimates by their sample sizes; if  $\eta_k = (1 / \sigma_k^2) / \sum_{j \in \mathcal{K}} (1 / \sigma_j^2)$  where  $\sigma_k^2 = \mathrm{Var}(\widehat{\mu}_{a,k})$ , the global estimator is the inverse variance weighting estimator, which is known to be appropriate when working models are homogeneous across sites [29].
In transportability studies, preventing negative transfer is critical when there are multiple, potentially biased source sites. If source sites are biased, both the sample size-weighted estimator and the variance-weighted estimator would inherit this bias. By examining the MSE of the data-adaptive estimator to the limiting estimand of the target estimator, the MSE can be decomposed into a variance term that can be minimized by regression of influence functions obtained from the asymptotic linear expansion of the target and source estimates, and an asymptotic bias term. This allows us to rewrite the problem of solving for ensemble weights as an adaptive LASSO problem [13, 14] as follows:
$$
\widehat {\eta} _ {k, L _ {1}} = \arg \min  _ {\eta_ {k} \geq 0} \sum_ {i = 1} ^ {N} \left[ \widehat {\xi} _ {T, i} (a) - \sum_ {k \in \mathcal {K}} \eta_ {k} \left(\widehat {\xi} _ {T, i} (a) - \widehat {\xi} _ {k, i} (a) - \widehat {\delta} _ {k}\right) \right] ^ {2} + \lambda \sum_ {k \in \mathcal {K}} | \eta_ {k} | \widehat {\delta} _ {k} ^ {2}, \tag {9}
$$