SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Theorem 1. For all  $h \in [H]$ , let  $G^h$  and  $p^h(x)$  denote the underlying DAG structure and pdf of environment  $\mathcal{E}_h$ , respectively, and let  $q(x)$  be the pdf of the mixture distribution of the  $H$  environments such that  $q(x) = \sum_{h=1}^{H} w_h p^h(x)$ . Also, let  $s(x) = \nabla \log q(x)$  be the associated score function. Then, under Assumptions  $A$  and  $B$ , we have:
(i) If  $j$  is a leaf in all DAGs  $G^h$ , then  $j$  is a shifted node if and only if  $\operatorname{Var}_X\left[\frac{\partial s_j(X)}{\partial x_j}\right] > 0$ .
(ii) If  $j$  is not a leaf in at least one DAG  $G^{h}$ , then  $\operatorname{Var}_X\left[\frac{\partial s_j(X)}{\partial x_j}\right] > 0$ .
Theorem 1 along with Proposition 1 suggests a way to identify shifted nodes. Namely, to use Proposition 1 to identify a common leaf, and then use Theorem 1 to test if such a leaf is a shifted node or not. We then proceed to remove the leaf and repeat the process. See Algorithm 1. Note that due to the fact that each environment is a result of an intervention (Definition 2) on an underlying ANM  $\mathcal{M}^*$ , it follows that the leaves in  $G^{*}$  will remain leaves in each DAG  $G^{th}$ .
Algorithm 1 iSCAN—Identifying Shifts in Causal Additive Noise models.
Input: Datasets  $X^1, \ldots, X^H$ .  
Output: Shifted variables set  $\widehat{S}$ , and topological sort  $\hat{\pi}$ .  
1: Initialize  $\widehat{S} = \{\}$ ,  $\hat{\pi} = (\cdot), \mathcal{N} = \{1, \ldots, d\}$   
2: Set  $\mathbf{X} = [(\mathbf{X}^1)^\top | \cdots | (\mathbf{X}^H)^\top]^\top \in \mathbb{R}^{m \times d}$ .  
3: while  $\mathcal{N} \neq \emptyset$  do  
4:  $\forall h \in [H]$ ,  $\operatorname{Var}^h \gets \operatorname{Var}_{\mathbf{X}^h}\left[\operatorname{diag}(\nabla^2 \log p^h(x))\right]$ .  
5:  $\operatorname{Var} \gets \operatorname{Var}_X\left[\operatorname{diag}(\nabla^2 \log q(x))\right]$   
6:  $L \gets \bigcap_{h \in [H]} \left\{j \mid \operatorname{Var}_j^h = 0, j \in [d]\right\}$ .  
7:  $\widehat{S} \gets \widehat{S} \cup \{j \mid \operatorname{Var}_j \neq 0, j \in L\}$   
8:  $\mathcal{N} \gets \mathcal{N} - \{L\}$   
9:  $\forall l \in L$ , remove the  $l$ -th column of  $\mathbf{X}^h$ ,  $\forall h \in [H]$ , and  $\mathbf{X}$ .  
10:  $\hat{\pi} \gets (L, \hat{\pi})$ .
Remark 2. See Appendix A for a practical implementation of Alg. 1. Finally, note that Alg. 1 also estimates a valid topological sort for the different environments by leveraging Proposition 1.
3.1 Score's Jacobian estimation
Since the procedure to estimate  $\mathrm{Var}_q[\frac{\partial s_j(x)}{\partial x_j}]$  is similar for estimating  $\mathrm{Var}_{p^h}[\frac{\partial s_j^h(x)}{\partial x_j}]$  for each  $h\in [H]$ , in this section we discuss the estimation for  $\mathrm{Var}_q[\frac{\partial s_j(x)}{\partial x_j}]$ , which involves computing the diagonal of the Hessian of  $\log q(x)$ . To estimate this quantity, we adopt a similar approach to the method in [45, 65]. First, we estimate the first-order derivative of  $\log q(x)$  by Stein's identity [73]:
$$
\mathbb {E} _ {q} \left[ \boldsymbol {h} (x) \nabla \log q (x) ^ {\top} + \nabla \boldsymbol {h} (x) \right] = 0, \tag {4}
$$
where  $\pmb{h}:\mathbb{R}^d\to \mathbb{R}^{d'}$  is any test function such that  $\lim_{x\to \infty}\pmb {h}(x)q(x) = 0$ . Once we have estimated  $\nabla \log q(x)$ , we can proceed to estimate the Hessian's diagonal by using second-order Stein's identity:
$$
\mathbb {E} _ {q} [ \boldsymbol {h} (x) \operatorname {d i a g} (\nabla^ {2} \log q (x)) ^ {\top} ] = \mathbb {E} _ {q} \left[ \nabla_ {\operatorname {d i a g}} ^ {2} \boldsymbol {h} (x) - \boldsymbol {h} (x) \operatorname {d i a g} (\nabla \log q (x) \nabla \log q (x) ^ {\top}) \right] \tag {5}
$$
Using eq.(4) and eq.(5), we can estimate the Hessian's diagonal at each data point. Thus allowing us to obtain an estimate of  $\mathrm{Var}_q\left[\frac{\partial s_j(x)}{\partial x_j}\right]$ . See Appendix A.1 for additional details.
Remark 3 (Consistency of Algorithm 1). The estimators in eq.(6) and eq.(7), given in Appendix A.1, correspond to Monte Carlo estimators using eq.(4) and (5), respectively, then the error of the estimators tend to zero as the number of samples goes to infinity. See for instance the discussion in Section 3.1 in [45]. We empirically explore the consistency of Algorithm 1 in Figure 2.
Remark 4 (Computational Complexity). Since we adopt the kernel-based estimator, SCORE, from [65]. The computational complexity for the estimation of the score's Jacobian in a single environment is  $\mathcal{O}(dm_h^3)$ . In Algorithm 1, computation is dominated by the SCORE function applied to the pooled data  $\mathbf{X} \in \mathbb{R}^{m \times d}$ . Therefore, the overall complexity of Algorithm 1 is  $\mathcal{O}(dm^3)$ . See Figure 2.
Figure 2: (Left) F1 score of the output of Alg. 1 w.r.t. to the true set of shifted nodes. For different number of nodes, we observe how iSCAN recovers the true set of shifted nodes as the number of samples increases, thus empirically showing its consistency. (Right) Runtime vs number of nodes for different number of samples. We corroborate the linear dependence of the time complexity on  $d$ .
4 On Identifying Structural Differences
After estimating the set of shifted nodes  $\widehat{S}$  through Algorithm 1, it is of high interest to predict which causal relations between a shifted node and its parents have undergone changes across the environments. The meaning of a change in a causal relationship can vary based on the context and the estimation objective. This section primarily centers on structural changes, elaborated further below, while additional discussion about other types of changes is available in Appendix D.
Definition 4 (Structurally shifted edge). For a given shifted node  $X_{j}$ , an edge  $X_{i} \to X_{j}$  is called a structurally shifted edge if  $\exists h, h' \in [H]$  such that  $X_{i} \in \mathrm{PA}_{j}^{h}$  and  $X_{i} \notin \mathrm{PA}_{j}^{h'}$ .
In other words, a structurally shifted edge is an edge that exists in one environment but not in another, indicating a change in the underlying structure of the causal mechanism. To detect the structurally shifted edges, we will estimate the parents of each shifted node in  $\widehat{S}$  for all environments  $\mathcal{E}_h$ .
Remark 5. Note that under the sparse mechanism shift hypothesis [69], i.e.,  $|S| \ll d$ , estimating the parents of each shifted node is much more efficient than estimating the entire individual structures.
Kernel regression and variable selection. A potential strategy to estimate structurally shifted edges involves employing the estimated topological order  $\hat{\pi}$  obtained from Algorithm 1. If this estimated topological order remains valid across all environments, it can serve as a guide for the nonparametric variable selection process to identify the parents of a shifted node  $X_{j}$ . Specifically, we can regress the shifted node  $X_{j}$  on its predecessors  $\widehat{\mathrm{Pre}}(X_{j})$  and proceed with a nonparametric variable selection procedure. Here  $\widehat{\mathrm{Pre}}(X_{j})$  consists of the set of nodes that appear before  $X_{j}$  in the estimated topological order  $\hat{\pi}$ . To achieve that, there exist various methods under the hypothesis testing framework [42, 17, 63], and bandwidth selection procedures [40]. These methods offer consistency guarantees, but their time complexity might be problematic. Kernel regression, for example, has a time complexity of  $\mathcal{O}(m^3)$ , and requires an additional bandwidth selection procedure, usually with a time complexity of  $\mathcal{O}(m^2)$ . Consequently, it becomes imperative to find a more efficient method for identifying parents locally.
Feature ordering by conditional independence (FOCI). An alternative efficient approach for identifying the parents is to leverage the feature ordering method based on conditional independence proposed by Azadkia and Chatterjee [3]. This method provides a measure of conditional dependency between variables with a time complexity of  $\mathcal{O}(m\log m)$ . By applying this method, we can perform fast variable selection in a nonparametric setting. See Algorithm 4 in Appendix A.3.
Theorem 2 (Consistency of Algorithm 4). Under Assumption  $C$ , given in Appendix B.2, if the estimated topological order  $\hat{\pi}$  output from Algorithm 1 is valid for all environments, then the output  $\widehat{\mathrm{PA}}_j^h$  of Algorithm 4 is equal to the true parents  $\mathrm{PA}_j^h$  of node  $X_{j}$  with high probability, for all  $h\in [H]$ .
Motivated by Theorem 2, we next present Algorithm 2, a procedure to estimate the structurally shifted edges. Given the consistency of Alg. 1 and Alg. 2, it follows that combining both algorithms will correctly estimate the true set of shifted nodes and structural shifted edges, asymptotically.
Algorithm 2 Identifying structurally shifted edges
Input: Data  $\{X^h\}_{h\in [H]}$  , topological order  $\hat{\pi}$  , shifted nodes  $\widehat{S}$    
Output: Structurally shifted edges set  $\widehat{E}$    
1: Initialize  $\widehat{E} = \emptyset$    
2: for  $X_{j}$  in  $\widehat{S}$  do   
3: for  $h$  in  $[H]$  do   
4: Estimate  $\widehat{\mathrm{PA}}_j^h$  from Alg. 4 (FOCI) with input  $\{\widehat{\mathrm{Pre}} (\mathbf{X}_j^h),\mathbf{X}_j^h\}$    
5: if  $\exists X_{k},h,h^{\prime}$  such that  $X_{k}\in \widehat{\mathrm{PA}}_{j}^{h},X_{k}\notin \widehat{\mathrm{PA}}_{j}^{h^{\prime}}$  then   
6:  $\widehat{E}\gets \widehat{E}\cup (X_k,X_j)$
We conducted a comprehensive evaluation of our algorithms. Section 5.1 focuses on assessing the performance of iSCAN (Alg. 1) for identifying shifted variables. In Section 5.2, we apply iSCAN for identifying shifted nodes along with FOCI (Alg. 2) for estimating structural changes, on apoptosis data. Also, in App. C, we provide additional experiments including: (i) Localizing shifted nodes without structural changes (App. C.1), and where the functionals are sampled from Gaussian processes (App. C.1.1); (ii) Localizing shifted nodes and estimating structural changes when the underlying graphs are different; and (iii) Evaluating iSCAN using the elbow method for selecting shifted nodes (see App. C.3 and Remark 6). Code is publicly available at https://github.com/kevinsbello/iSCAN.
5.1 Synthetic experiments on shifted nodes
Graph models. We generated random graphs using the Erdős-Rényi (ER) and scale free (SF) models. For a given number of variables  $d$ , ER $k$  and SF $k$  indicate an average number of edges equal to  $kd$ .
Data generation. We first sampled a DAG,  $G^{1}$ , of  $d$  nodes according to either the ER or SF model for env.  $\mathcal{E}_1$ . For env.  $\mathcal{E}_2$ , we initialized its DAG structure from env.  $\mathcal{E}_1$  and produced structural changes by randomly selecting  $0.2 \cdot d$  nodes from the non-root nodes. This set of selected nodes  $S$  with cardinality  $|S| = 0.2d$ , correspond to the set of "shifted nodes". In env.  $\mathcal{E}_2$ , for each shifted node  $X_{j} \in S$ , we uniformly at random deleted at most three of its incoming edges, and use  $D_{j}$  to denote the parents whose edges to  $X_{j}$  were deleted; thus, the DAG  $G^{2}$  is a subgraph of  $G^{1}$ . Then, in  $\mathcal{E}_1$ , each  $X_{j}$  was defined as follows:
$$
X _ {j} = \sum_ {i \in \mathrm {P A} _ {j} ^ {1} \backslash D _ {j}} \sin (X _ {i} ^ {2}) + \sum_ {i \in D _ {j}} 4 \cos (2 X _ {i} ^ {2} - 3 X _ {i}) + N _ {j}
$$
In  $\mathcal{E}_2$ , each  $X_{j}$  was defined as follows:
$$
X _ {j} = \sum_ {i \in \mathrm {P A} _ {j} ^ {2}} \sin (X _ {i} ^ {2}) + N _ {j}
$$
Experiment details. For each simulation, we generated 500 data points per environment, i.e.,  $m_{1} = 500$ ,  $m_{2} = 500$  and  $m = 1000$ . The noise variances were set to 1. We conducted 30 simulations for each combination of graph type (ER or SF), noise type (Gaussian, Gumbel, and Laplace), and number of nodes ( $d \in \{10, 20, 30, 50\}$ ). The running time was recorded by executing the experiments on an Intel Xeon Gold 6248R Processor with 8 cores. For our method, we used  $\eta = 0.05$  for eq.(6) and eq.(7), and a threshold  $t = 2$  (see Alg. 3).
Evaluation. We compared the performance of iSCAN against several baselines, which include: DCI [82], the approach by [11], CITE [79], KCD [53], SCORE [65], and UT-IGSP [72]. Figure 3 illustrates the results for ER4 and SF4 graphs. We note that iSCAN consistently outperforms other baselines in terms of F1 score across all scenarios. Importantly, note how the performance of some baselines, like DCI, CITE, Budhathoki's, and SCORE, degrades faster for graphs with hub nodes, a property of SF graphs. In contrast, iSCAN performs similarly, as it is not dependent on structural assumptions on the individual DAGs. Additionally, it is worth noting that our method exhibits faster computational time than KCD, Budhathoki's, and SCORE, particularly for larger numbers of nodes.
In Appendix C.1, we provide experiments on sparser graphs such as ER2/SF2, and denser graphs such as ER6/SF6. We also include Precision and Recall in all plots in the supplement.
Figure 3: Experiments on ER4 and SF4 graphs. See the experiment details above. The points indicate the average values obtained from these simulations. The error bars depict the standard errors. Our method iSCAN (light blue) consistently outperformed baseline methods in terms of F1 score.
5.2 Experiments on apoptosis data
We conducted an analysis on an ovarian cancer dataset using iSCAN (Algorithm 1) to identify shifted nodes and Algorithm 2 to detect structurally shifted edges (SSEs). This dataset had previously been analyzed using the DPM method [88] in the undirected setting, and the DCI method [82] in the linear setting. By applying our method, we were able to identify the shifted nodes and SSEs in the dataset (see Figure 4a). Our analysis revealed the identification of two hub nodes in the apoptosis
pathway: BIRC3, and PRKAR2B. The identification of BIRC3 as a hub node was consistent with the results obtained by the DPM and DCI methods. Additionally, our analysis also identified PRKAR2B as a hub node, which was consistent with the result obtained by the DCI method. Indeed, BIRC3, in addition to its role in inhibiting TRAIL-induced apoptosis, has been investigated as a potential therapeutic target in cancer treatment including ovarian cancer[35, 81]; whereas PRKAR2B has been identified as an important factor in the progression of ovarian cancer cells. The latter serves as a key regulatory unit involved in the growth and development of cancer cells [84, 13].
Figure 4: Results on apoptosis data.
In this work, we showed a novel connection between score matching and identifying causal mechanism shifts among related heterogeneous datasets. This finding opens up a new and promising application for score function estimation techniques.
Our proposed technique consists of three modules. The first module evaluates the Jacobian of the score under the individual distributions and the mixture distribution. The second module identifies shifted features (variables) using the estimated Jacobians, allowing us to pinpoint the nodes that have undergone a mechanism shift. Finally, the third module aims to estimate structurally shifted edges, a.k.a. the difference DAG, by leveraging the information from the identified shifted nodes and the estimated topological order. It is important to note that our identifiability result in Theorem 1 is agnostic to the choice of the score estimator.
The strength of our result lies in its capability to recover the difference DAG in non-linear Additive Noise Models (ANMs) without making any assumptions about the parametric form of the functions or statistical independencies. This makes our method applicable in a wide range of scenarios where non-linear relationships and shifts in mechanisms are present.
6.1 Limitations and future work
While our work demonstrates the applicability of score matching in identifying causal mechanism shifts in the context of nonlinear ANMs, there are several limitations and areas for future exploration:
Extension to other families of SCMs: Currently, our method is primarily focused on ANMs where the noise distribution satisfies Assumption B, e.g., Gaussian distributions. It would be valuable to investigate the application of score matching in identifying causal mechanism shifts in other types of SCMs. Recent literature, such as [48], has extended score matching to additive Models with arbitrary noise for finding the topological order. Expanding our method to accommodate different noise models would enhance its applicability to a wider range of real-world scenarios.
Convergence rate analysis: Although the score matching estimator is asymptotically consistent, the convergence rate remains unknown in general. Understanding the convergence properties of the estimator is crucial for determining the sample efficiency and estimating the required number of samples to control the estimation error within a desired threshold. Further theoretical developments, such as [37], on score matching estimators would provide valuable insights into the performance and sample requirements of iSCAN.
K.B. was supported by NSF under Grant # 2127309 to the Computing Research Association for the CIFellows 2021 Project. B.A. was supported by NSF IIS-1956330, NIH R01GM140467, and the Robert H. Topel Faculty Research Fund at the University of Chicago Booth School of Business. P.R. was supported by ONR via N000141812861, and NSF via IIS-1909816, IIS-1955532, IIS-2211907. We are also grateful for the support of the University of Chicago Research Computing Center for assistance with the calculations carried out in this work.
[1] Aragam, B., Amini, A. and Zhou, Q. [2019], 'Globally optimal score-based learning of directed acyclic graphs in high-dimensions', Advances in Neural Information Processing Systems 32.
[2] Aragam, B. and Zhou, Q. [2015], 'Concave penalized estimation of sparse Gaussian Bayesian networks', The Journal of Machine Learning Research 16(1), 2273-2328.
[3] Azadkia, M. and Chatterjee, S. [2021], 'A simple measure of conditional dependence', The Annals of Statistics 49(6), 3070-3102.
[4] Azadkia, M., Taeb, A. and Buhlmann, P. [2021], 'A fast non-parametric approach for local causal structure learning', arXiv e-prints pp. arXiv-2111.
[5] Barabási, A.-L. and Albert, R. [1999], 'Emergence of scaling in random networks', science 286(5439), 509-512.
[6] Barabási, A.-L., Gulbahce, N. and Loscalzo, J. [2011], 'Network medicine: a network-based approach to human disease', Nature reviews genetics 12(1), 56-68.
[7] Barabasi, A.-L. and Oltvai, Z. N. [2004], 'Network biology: understanding the cell's functional organization', Nature reviews genetics 5(2), 101-113.
[8] Bello, K., Aragam, B. and Ravikumar, P. [2022], 'Dagma: Learning dags via m-matrices and a log-determinant acyclicity characterization', arXiv preprint arXiv:2209.08037.
[9] Boente, G. and Pardo-Fernández, J. C. [2022], 'Robust testing to compare regression curves', arXiv preprint arXiv:2205.12065.
[10] Brouillard, P., Lachapelle, S., Lacoste, A., Lacoste-Julien, S. and Drouin, A. [2020], 'Differentiable causal discovery from interventional data', Advances in Neural Information Processing Systems 33, 21865-21877.
[11] Budhathoki, K., Janzing, D., Bloebaum, P. and Ng, H. [2021], Why did the distribution change?, in 'International Conference on Artificial Intelligence and Statistics', PMLR, pp. 1666-1674.
[12] Buhlmann, P., Peters, J. and Ernest, J. [2014], 'Cam: Causal additive models, high-dimensional order search and penalized regression', The Annals of Statistics 42(6), 2526-2556.
[13] Chiaradonna, F., Balestrieri, C., Gaglio, D. and Vanoni, M. [2008], 'Ras and pka pathways in cancer: new insight from transcriptional analysis', Frontiers in Bioscience-Landmark 13(14), 5257-5278.
[14] Chickering, D. M. [1996], Learning bayesian networks is np-complete, in 'Learning from data', Springer, pp. 121-130.
[15] Chickering, D. M. [2003], 'Optimal structure identification with greedy search', JMLR 3, 507-554.
[16] Chickering, D. M., Heckerman, D. and Meek, C. [2004], 'Large-sample learning of Bayesian networks is NP-hard', Journal of Machine Learning Research 5, 1287-1330.
[17] Delgado, M. A. and Manteiga, W. G. [2001], 'Significance testing in nonparametric regression based on the bootstrap', The Annals of Statistics 29(5), 1469-1507.
[18] Demiralp, S. and Hoover, K. D. [2003], 'Searching for the causal structure of a vector autoregression', Oxford Bulletin of Economics and statistics 65, 745-767.
[19] Fazayeli, F. and Banerjee, A. [2016], Generalized Direct Change Estimation in Ising Model Structure, in 'International Conference on Machine Learning'.
[86] Zhang, K., Peters, J., Janzing, D. and Scholkopf, B. [2012], 'Kernel-based conditional independence test and application in causal discovery', arXiv preprint arXiv:1202.3775.
[87] Zhao, B., Wang, Y. S. and Kolar, M. [2020], 'Fudge: Functional differential graph estimation with fully and discretely observed curves', arXiv preprint arXiv:2003.05402.
[88] Zhao, S. D., Cai, T. T. and Li, H. [2014], 'Direct estimation of differential networks', Biometrika 101(2), 253-268.
[89] Zheng, X., Aragam, B., Ravikumar, P. K. and Xing, E. P. [2018], 'Dags with no tears: Continuous optimization for structure learning', NeurIPS.
SUPPLEMENTARY MATERIAL
A Practical Implementation
In this section, we present a more practical version of Alg. 1 that considers estimation errors, see Alg. 3. First, we provide more details of the score's Jacobian estimation.
A.1 Practical Version of SCORE
Let  $X = \{x^{1},\ldots ,x^{m}\}$  be a dataset of  $m$  possibly non-independent but identically distributed samples. From Li and Turner [45], we next present the estimator for the point-wise first-order partial derivative, corresponding to eq.(4):
$$
\hat {\boldsymbol {G}} = - \left(\boldsymbol {K} + \eta \boldsymbol {I}\right) ^ {- 1} \langle \nabla , \boldsymbol {K} \rangle \tag {6}
$$
where  $\pmb{H} = (h(x^{1}),\dots,h(x^{m}))\in \mathbb{R}^{d^{\prime}\times m},\overline{\nabla h} = \frac{1}{m}\sum_{k = 1}^{m}\nabla h(x^{k}),\pmb{K} = \pmb{H}^{\top}\pmb{H},K_{ij} = \kappa (x^{i},x^{j}) = \pmb {h}(x^{i})^{\top}\pmb {h}(x^{j}),\langle \nabla ,\pmb {K}\rangle = m\pmb{H}^{T}\overline{\nabla h},\langle \nabla ,\pmb{K}\rangle_{ij} = \sum_{k = 1}^{m}\nabla_{x_{j}^{k}}\kappa (x^{i},x^{j}),$  and  $\eta \geq 0$  is a regularization parameter. Here  $\hat{G}$  is used to approximate  $G\equiv (\nabla \log p(x^{1}),\ldots ,\nabla \log p(x^{m}))^{\top}\in$ $\mathbb{R}^{m\times d}$
From [65], we now present the estimator for the diagonal elements of the score's Jacobian at the sample points, i.e.  $J \equiv (\mathrm{diag}(\nabla^2 \log p(x^1)), \ldots (\mathrm{diag}(\nabla^2 \log p(x^m)))^\top \in \mathbb{R}^{m \times d}$ , the estimator of  $J$  is:
$$
\hat {\boldsymbol {J}} = - \operatorname {d i a g} \left(\hat {\boldsymbol {G}} \hat {\boldsymbol {G}} ^ {\top}\right) + (\boldsymbol {K} + \eta \boldsymbol {I}) ^ {- 1} \langle \nabla_ {\mathrm {d i a g}} ^ {2}, \boldsymbol {K} \rangle \tag {7}
$$
where  $\pmb{H} = (h(x^{1}),\dots,h(x^{m}))\in \mathbb{R}^{d^{\prime}\times m},\overline{\nabla_{\mathrm{diag}}^{2}\pmb{h}} = \frac{1}{m}\sum_{k = 1}^{m}\nabla_{\mathrm{diag}}^{2}\pmb {h}(x^{k}),(\nabla_{\mathrm{diag}}^{2}\pmb {h}(x))_{ij} = \frac{\partial^{2}h_{i}(x)}{\partial x_{j}^{2}},$ $\begin{array}{rlr}{\pmb {K}} & = & {\pmb{H}}^{\top}{\pmb{H}},\end{array}$ $\pmb {K}_{ij} = \kappa (x^i,x^j) = \pmb {h}(x^i)^{\top}\pmb {h}(x^j),\langle \nabla_{\mathrm{diag}}^2,\pmb {K}\rangle = m\pmb {H}^T\overline{\nabla_{\mathrm{diag}}^2\pmb{h}},$ $\langle \nabla_{\mathrm{diag}}^2,K\rangle_{ij} = \sum_{k = 1}^{m}\frac{\partial^2\kappa(x^i,x^k)}{(\partial x_j^k)^2}$  , and  $\eta \geq 0$  is a regularization parameter.
In the sequel, we use  $\mathrm{SCORE}(\mathbf{X})$  to denote the procedure to compute the sample variance for the estimator of the diagonal of the score's Jacobian via eq.(7).
A.2 Practical Version of Algorithm 1