SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We establish in Section 4 that the total variation distance between the true distribution of  $X|Z$  and the distribution of samples generated by the  $k$ -nearest-neighbor local sampling strategy tends to zero in probability as  $n$  goes to infinity. This evidence suggests that the latter is a close approximation of the former. Thus,  $(\widetilde{X}, Y, Z)$  approximates directly a distribution that encodes the null hypothesis. We then compute the CMI estimator based on it. This process is repeated  $B$  times, resulting in  $B$  realizations of the CMI estimator under the null hypothesis denoted by  $(\widehat{\mathbf{CMI}}^{(1)}, \ldots, \widehat{\mathbf{CMI}}^{(B)})$ . We can determine whether to reject the null hypothesis by comparing these estimators to the one of the original sample set  $A$ . Specifically, we calculate the  $p$ -value using (8), but with  $\widehat{\mathbf{CMI}}^{(b)}$  replaced by  $\widehat{\mathbf{CMI}}^{(b)}$ . If the resulting  $p$ -value is smaller than the prespecified significance level, we reject the null hypothesis. The pseudo code is presented in Algorithm 3. In Section 4, we will demonstrate through theoretical analysis that our test asymptotically achieves a valid control of type I error and is consistent against all the alternatives in  $H_{1}$ .
[30] employed the 1-NN sampling strategy to approximate the distribution of  $X|Z$ . To ensure substantial dissimilarity between the pseudo-sample datasets generated across repetitions, their method requires dividing the dataset into two parts. As a consequence, the dataset used for calculating the test statistics consists of only one-third of the total samples. This limitation can result in reduced statistical power of the test, especially when working with small datasets. In contrast, our proposed procedure eliminates the need for dataset splitting and allows the entire dataset to be used in computing CMI, thereby avoiding the loss of testing power.
In this section, we present our main theoretical results. All the detailed proofs for these results can be found in the Supplementary Materials. Let  $P_{1}$  and  $P_{2}$  be two probability distributions defined on the same probability space. The total variation distance between  $P_{1}$  and  $P_{2}$  is defined as  $d_{TV}(P_1,P_2) = \sup_{A\subset \Omega}|P_1(A) - P_2(A)|$ , where the supremum is taken over all measurable subsets  $A$  of the sample space  $\Omega$ . We will show in Theorem 2 that the distribution of  $\widetilde{X}$  generated by the  $k$ -nearest-neighbor local sampling strategy is very close to the true conditional distribution in terms of the total variation distance.
Lemma 1 plays a crucial role in proving Theorem 2. It demonstrates that the  $k$ -nearest neighbor of  $Z$  has a similar property to the work of [13], where it has been shown that the nearest neighbor of  $Z$  converges almost surely (a.s.) to  $Z$  as the sample size  $n$  approaches infinity.
Lemma 1. Let  $Z$  and  $Z_{1},\ldots ,Z_{n}$  be i.i.d. random vectors from  $p(z)$ . For a given positive integer  $k$ , let  $Z_{n}^{(k)}$  be the  $k$ -nearest neighbor of  $Z$  from the set  $\{Z_1,\dots,Z_n\}$ . Then,  $Z_{n}^{(k)} \xrightarrow{a.s.} Z$  as  $n\to \infty$ .
We make the following regularity conditions, which have been introduced in [19], [20] and [41]. To simplify the notation, we may drop the subscripts when referring to probability density functions.
Assumption 1 (Smoothness on  $p(x|z)$ ). For any  $z \in \mathbb{R}^{d_z}$  and any  $a$  such that  $\| a - z \|_2 \leq \epsilon$ , we assume  $0 \leq \lambda_{\max}(I_a(z)) \leq \beta$ , where  $d_z$  is the dimension of the random vector  $Z$ ,  $\| \cdot \|_2$  denotes the  $l_2$  norm,  $0 < \beta < \infty$ , and  $I_a(z)$ , called the generalized curvature matrix, is defined with entries
$$
I _ {a} (z) _ {i j} = E \left(- \frac {\partial^ {2} \log p (X | Z = \widetilde {z})}{\partial \widetilde {z} _ {i} \partial \widetilde {z} _ {j}} \Big | _ {\widetilde {z} = a} \Bigg | Z = z\right) = \left(\frac {\partial^ {2}}{\partial \widetilde {z} _ {i} \partial \widetilde {z} _ {j}} \int \log \frac {p (x | z)}{p (x | \widetilde {z})} p (x | z) d x\right) \Bigg | _ {\widetilde {z} = a}.
$$
Assumption 2 (Smoothness on  $p(z)$ ). Assume that the probability density function  $p(z)$  is twice continuously differentiable. Let  $H_{p}(z)$  denote the Hessian matrix of  $p(z)$  with respect to  $z$ . We assume that  $\| H_{p}(z)\|_{2}\leq c_{d_{z}}$  holds almost everywhere, where  $c_{d_z}$  depends only on the dimension  $d_{z}$  of the random vector  $Z$ .
Note that Assumption 1 can be viewed as an extension of the requirement that the maximum eigenvalue of the Fisher information matrix for  $x$  with respect to  $z$  is bounded. Assumptions 1 and 2 can be validated when  $(X, Z)$  follows a multivariate Gaussian distribution (MVD) and when  $Z$  follows a MVD, respectively.
For  $Z$ , we define  $Z_{n}^{(l)}$  as its  $l$ -nearest neighbor for  $l = 1, \dots, k$ . According to Algorithm 3, each  $Z_{n}^{(l)}$  is selected with probability  $1 / k$ . Let  $\xi$  be a random variable with a probability mass function  $P(\xi = l) = 1 / k$  for  $l = 1, \dots, k$ , and  $\xi$  is independent of both  $Y$  and  $Z$ . Given  $Z$ , we denote by  $\widetilde{X}$  the sample generated by the  $k$ -nearest-neighbor local sampling mechanism, which follows the distribution  $\widehat{p}(x|Z) = p(x|Z_{n}^{(1)})^{1\{\xi = 1\}} \times \ldots \times p(x|Z_{n}^{(k)})^{1\{\xi = k\}}$ . The sketch proof of Theorem 2 proceeds as follows. We first apply Pinsker's inequality, which relates the total variation distance between  $p(x|Z)$  and  $\widehat{p}(x|Z)$  to their KL divergence. We then establish a connection between the KL divergence and the discrepancy between  $Z$  and  $Z_{n}^{(l)}$ . Finally, by Lemma 1, we derive Theorem 2.
Theorem 2. Under Assumptions 1 and 2, we have  $d_{TV}\{p(x|Z), \widehat{p}(x|Z)\} = o_p(1)$  as  $n \to \infty$ .
Remark 1. The constant  $\beta$  in Assumption 1 is used to establish an upper bound on the KL divergence between  $p(x|Z)$  and  $p(x|Z_n^{(l)})$ , which can be used to bound  $d_{TV}\{p(x|Z), \widehat{p}(x|Z)\}$  through Pinsker's inequality.
In Theorem 3, we bound the excess type I error conditionally on  $\mathbf{Y}$  and  $\mathbf{Z}$  by the total variation distance between  $\widehat{p}(\cdot|\mathbf{Z})$  and  $p(\cdot|\mathbf{Z})$ .
Theorem 3. Assume  $H_0: X \perp Y|Z$  is true. Under Assumptions 1 and 2, for any significance level  $\alpha \in (0,1)$ , the  $p$ -value obtained from Algorithm 3 satisfies  $P(p \leq \alpha | Y, Z) \leq \alpha + d_{TV} \{ p(\cdot | Z), \widehat{p}(\cdot | Z) \}$ .
The type I error rate can be unconditionally controlled based on Theorems 2 and 3, which implies that  $P(p \leq \alpha | H_0) \leq \alpha + o(1)$  as  $n$  approaches infinity. Now we turn to analyze the power of our test in asymptotic scenarios in Theorem 4. Some intuitive explanations on both type I error rate and power can be found in the third paragraph of Section 3.2.
Theorem 4. For any  $\alpha \in (0,1)$  and the number of repetitions  $B_{n}$  satisfying  $B_{n}\to \infty$ , let  $p$  be the  $p$ -value calculated in Algorithm 3. Under Assumptions 1 and 2 and the Assumptions stated in the Supplementary Materials,  $P(p\leq \alpha |H_1)\rightarrow 1$  as  $n\to \infty$ .
Remark 2. The condition  $B_{n} \to \infty$  is mild and has been made in [7, 43]. The consistency of our test heavily relies on the consistency of the CMI estimator, which hinges on whether the joint density of  $(X, Y', Z)$  generated by the 1-NN sampling (Algorithm 1), denoted as  $\phi$ , approximates  $p(x, z) p(y|z)$  (denoted as  $g$ ) well, in terms of TV distance.  $\beta$  and  $c_{d_z}$  in Assumptions 1 and 2 are used to bound  $d_{TV}(\phi, g)$ .
Remark 3. According to Theorem 4, our test can achieve consistency against all the alternatives stated in  $H_{1}$ . In contrast, the method proposed by [43] only achieves consistency against a subset of the alternatives in  $H_{1}$ , and [30] does not provide any theoretical results on the testing power.
In this section, we present a comparative evaluation of our proposed method against the state-of-the-art (SOTA) methods on synthetic datasets. Our code is publicly available at: https://github.com/LeeShuai-kenwitch/NNLSCIT. We specifically compare our method with six commonly used competitive CI testing methods: KCIT [47], LPCIT[40], CCIT [41], CMIknn [38], GCIT [5], and NNSCIT [30]. We set the number of repetitions  $B = 200$  and the neighbor order  $k = 7$  for our tests. The XGBoost classifier was used in all of our experiments. Further elaboration on the choice of  $k$  is given in Figure 3 in the Supplementary Materials. We set the significance level to  $\alpha = 0.05$  and report the type I error rate and the testing power under  $H_{1}$  for all methods evaluated in our experiments. All the results are presented as an average over 200 independent trials. We provide additional simulation studies and real data analysis in the Supplementary Materials.
Scenario I: the post-nonlinear model. The datasets used in our experiments are generated using the post-nonlinear model similar to those in [47, 16, 5, 40]. Specifically, we define  $(X,Y,Z)$  under  $H_0$  and  $H_{1}$  as follows:
$$
H _ {0}: X = f _ {1} (\bar {Z} + 0. 5 \epsilon_ {x}), Y = f _ {2} (\bar {Z} + 0. 5 \epsilon_ {y}),
$$
$$
H _ {1}: X = f _ {1} (\bar {Z} + 0. 5 \epsilon_ {x}) + 0. 5 \epsilon_ {b}, Y = f _ {2} (\bar {Z} + 0. 5 \epsilon_ {y}) + 0. 5 \epsilon_ {b}, \tag {9}
$$
where  $\overline{Z}$  represents the sample mean of  $Z = (z_{1},\dots ,z_{d_{z}})$ , all  $z_{l}$  in  $Z$ ,  $\epsilon_x,\epsilon_y$  and  $\epsilon_{b}$  are i.i.d. samples generated from the standard Gaussian or the standard Laplace distribution, and  $f_{1}$  and  $f_{2}$  are randomly sampled from the set  $\{x,x^2,x^3,\tanh (x),\cos (x)\}$ .
We conduct a comparative analysis of various tests from two perspectives. The results are shown in Figure 1. First, we fix the sample size at  $n = 500$  and vary the dimension of  $Z$  from 30 to 100. In both the Gaussian and Laplace cases, our test shows good and stable performance in controlling type I error and achieving high power under  $H_{1}$  as the dimension of  $Z$  increases. LPCIT and NNSCIT have satisfactory performance in controlling type I error, but LPCIT loses power under  $H_{1}$  when the dimension exceeds 70, and NNSCIT has inadequate power for all dimensions. Although GCIT, CCIT, and KCIT have adequate power under  $H_{1}$ , they inflate the type I error in almost all scenarios. CMIknn shows weak performance on both type I error and testing power. Figure 6 in the Supplementary Materials contains additional results on the performance of various tests in low dimensions of  $Z$  ranging from 5 to 30, further demonstrating the superiority of our test. Second, we vary the sample size from 300 to 2000 while fixing the dimension of  $Z$  at 80. Our test maintains good control of the type I error and achieves high power, with the power approaching 1 when the sample size exceeds 500. However, LPCIT, NNSCIT, and CMIknn lose power under  $H_{1}$ , while GCIT, CCIT, and KCIT either always or sometimes fail to control the type I error well.
In the Supplementary Materials, Figure 7 shows the timing performance of all methods for a single test. Our test is found to be highly computationally efficient even when dealing with large sample sizes and high-dimensional conditioning sets. In contrast, CMIknn and CCIT for sample sizes exceeding 1000, and LPCIT for dimension of  $Z$  higher than 50 are impractical due to their prohibitively long running time.
Scenario II: the heavy tailed model. We compare the performance of our test with the SOTA tests under the heavy tailed error distributions, as described in [10]. The data  $(X,Y,Z)$  is generated
Figure 1: Comparison of the type I error (lower is better) and power (higher is better) of our method with six SOTA methods on the post-nonlinear model under Gaussian or Laplace distributions in Scenario I. Left: The results when varying the dimension of  $Z$ . Right: The results when varying the sample size.
according to the following model:
$$
H _ {0}: X = \bar {Z} + \epsilon_ {1}, Y = \bar {Z} + \epsilon_ {2},
$$
$$
H _ {1}: X = \bar {Z} + \epsilon_ {1}, Y = \bar {Z} + \epsilon_ {1} + \epsilon_ {2}, \tag {10}
$$
where  $\overline{Z}$  is the sample mean of  $Z = (z_{1},\ldots ,z_{d_{z}})$ ,  $z_{l}$ 's are i.i.d. samples generated from the standard Gaussian distribution, and  $\epsilon_{1}$  and  $\epsilon_{2}$  are independently generated from the standard Cauchy distribution. We keep the sample size fixed at  $n = 500$ .
Figure 2: Comparison of the type I error (lower is better) and power (higher is better) of our method with six SOTA methods in Scenarios II and III. Left: The results in Scenario II. Right: The results in Scenario III.
Scenario III: the chain structure. We further use the chain structure  $Y \to Z \to X$  in [30] to validate the proposed test. The detailed procedure for generating the simulated data can be found in the Supplementary Materials. The sample size is fixed at  $n = 500$ .
Figure 2 demonstrates that our test effectively controls the type I error and obtains adequate power in both Scenarios II and III under various dimensions of  $Z$ .
In this paper, we introduce a novel method for conducting the conditional independence testing. Theoretical analysis shows that our test is asymptotically valid and consistent against all alternatives in  $H_{1}$ . Extensive experiments on both synthetic and real datasets demonstrate that our method consistently outperforms commonly used SOTA methods. Moreover, our approach maintains computational efficiency even when the sample size and/or dimension of the conditioning set increase. Our method has the potential to enhance the applicability of causal discovery to real-world problems, such as gene regulatory networks or complex social networks, and to facilitate the discovery of relationships and patterns in complex systems. Ethically, we believe that our rather fundamental work has minimal potential for misuse.
Dr. Ziqi Chen's work was partially supported by the National Key R&D Program of China (2021YFA1000100 and 2021YFA1000101), National Natural Science Foundation of China (NSFC) (12271167, 72331005 and 11871477), Natural Science Foundation of Shanghai (21ZR1418800) and Basic Research Project of Shanghai Science and Technology Commission (22JC1400800). Dr. Christina Dan Wang's work was partially supported by the National Natural Science Foundation of China (NSFC) (12271363 and 11901395). We thank the anonymous reviewers for their helpful comments.
[39] Karen Sachs, Omar Perez, Dana Pe'er, Douglas A Lauffenburger, and Garry P Nolan. Causal protein-signaling networks derived from multiparameter single-cell data. Science, 308(5721):523-529, 2005.
[40] Meyer Sctbon, Laurent Meunier, and Yaniv Romano. An asymptotic test for conditional independence using analytic kernel embeddings. In International Conference on Machine Learning, pages 19328-19346. PMLR, 2022.
[41] Rajat Sen, Ananda Theertha Suresh, Karthikeyan Shanmugam, Alexandros G Dimakis, and Sanjay Shakkottai. Model-powered conditional independence test. In Advances in Neural Information Processing Systems, 30, 2017.
[42] Rajen D Shah and Jonas Peters. The hardness of conditional independence testing and the generalised covariance measure. The Annals of Statistics, 48(3):1514-1538, 2020.
[43] Chengchun Shi, Tianlin Xu, Wicher Bergsma, and Lexin Li. Double generative adversarial networks for conditional independence testing. Journal of Machine Learning Research, 22(285):1-32, 2021.
[44] Peter Spirtes, Clark N Glymour, and Richard Scheines. Causation, prediction, and search. MIT Press, 2000.
[45] Eric V Strobl, Kun Zhang, and Shyam Visweswaran. Approximate kernel-based conditional independence tests for fast non-parametric causal discovery. Journal of Causal Inference, 7(1), 2019.
[46] Xueqin Wang, Wenliang Pan, Wenhao Hu, Yuan Tian, and Heping Zhang. Conditional distance correlation. Journal of the American Statistical Association, 110(512):1726-1734, 2015.
[47] Kun Zhang, Jonas Peters, Dominik Janzing, and Bernhard Schölkopf. Kernel-based conditional independence test and application in causal discovery. In Proceedings of the 27th Conference on Uncertainty in Artificial Intelligence, pages 804–813, 2011.
[48] Shengyu Zhu, Ignavier Ng, and Zhitang Chen. Causal discovery with reinforcement learning. In International Conference on Learning Representations, 2020.
Supplementary Materials for "K-Nearest-Neighbor Local Sampling Based Conditional Independence Testing"
Proof. Recall that  $Z$  is a random vector taking values in Euclidean space  $(\mathbb{R}^{d_z},\| \cdot \| _2)$ , where  $d_{z}$  is the dimension of  $Z$  and  $\| \cdot \| _2$  is Euclidean distance.  $Z_{1},Z_{2},\ldots ,Z_{n}$  are i.i.d. random vectors according to  $p(z)$ . For a fixed  $z\in \mathbb{R}^{d_z}$ , we denote by  $Z_{n}^{(1)}(z),\dots,Z_{n}^{(n)}(z)$  a reordering of  $Z_{1},Z_{2},\dots,Z_{n}$  according to the increasing values of  $\| Z_i - z\| _2$ , that is,
$$
\left\| Z _ {n} ^ {(1)} (z) - z \right\| _ {2} \leq \dots \leq \left\| Z _ {n} ^ {(n)} (z) - z \right\| _ {2}.
$$
Define the set  $G = \{z \in \mathbb{R}^{d_z} \mid \forall \delta > 0, P(\{\omega : Z(\omega) \in S_z(\delta)\}) > 0\}$ , where  $S_z(\delta) = \{x \in \mathbb{R}^{d_z} \mid \| x - z \|_2 \leq \delta\}$ . For convenience, we omit  $\omega$  in probability in the following paper. For example, write  $P(Z \in S_z(\delta))$  instead of  $P(\{\omega : Z(\omega) \in S_z(\delta)\})$ . By definition, for  $z \in G, \forall \delta > 0, P(Z \in S_z(\delta)) > 0$ . Let  $G^c$  be the complement of  $G$ . Then, for  $z \in G^c, \exists r_z > 0$ , s.t.  $\forall r < r_z, P(Z \in S_z(r)) = 0$ . Note that  $P(\| Z_n^{(k)}(z) - z \|_2 > \delta) = P(Z_n^{(k)}(z) \notin S_z(\delta))$ .
In order to prove  $\| Z_n^{(k)}(Z) - Z \|_2 \to 0$  a.s., it is sufficient to prove  $\forall \delta > 0$ ,  $\lim_{n \to \infty} P(\sup_{m \geq n} \| Z_m^{(k)}(Z) - Z \|_2 > \delta) = 0$ . We can obtain
$$
\begin{array}{l} P \left(\sup  _ {m \geq n} \| Z _ {m} ^ {(k)} (Z) - Z \| _ {2} > \delta\right) \leq P \left(\left\{\sup  _ {m \geq n} \| Z _ {m} ^ {(k)} (Z) - Z \| _ {2} > \delta \right\} \cap \{Z \in G \}\right) + P \left(Z \in G ^ {c}\right) \\ = \int_ {G} P \left(\sup  _ {m \geq n} \| Z _ {m} ^ {(k)} (z) - z \| _ {2} > \delta\right) p (z) d z + P \left(Z \in G ^ {c}\right) \\ \leq \int_ {G} \sum_ {m \geq n} P \left(Z _ {m} ^ {(k)} (z) \notin S _ {z} (\delta)\right) p (z) d z + P \left(Z \in G ^ {c}\right). \tag {11} \\ \end{array}
$$
First, consider the first term of (11). We have
$$
\begin{array}{l} P \left(Z _ {m} ^ {(k)} (z) \notin S _ {z} (\delta)\right) = P \left(Z _ {m} ^ {(1)} (z), Z _ {m} ^ {(2)} (z), \dots , Z _ {m} ^ {(m)} (z) \notin S _ {z} (\delta)\right) \\ + P \left(Z _ {m} ^ {(1)} (z) \in S _ {z} (\delta), Z _ {m} ^ {(2)} (z), \dots , Z _ {m} ^ {(m)} (z) \notin S _ {z} (\delta)\right) \\ + P \left(Z _ {m} ^ {(1)} (z), Z _ {m} ^ {(2)} (z) \in S _ {z} (\delta), Z _ {m} ^ {(3)} (z), \dots , Z _ {m} ^ {(m)} (z) \notin S _ {z} (\delta)\right) \\ + \dots + P \left(Z _ {m} ^ {(1)} (z), \dots , Z _ {m} ^ {(k - 1)} (z) \in S _ {z} (\delta), Z _ {m} ^ {(k)} (z), \dots , Z _ {m} ^ {(m)} (z) \notin S _ {z} (\delta)\right). \\ \end{array}
$$
By setting  $P(Z\in S_z(\delta)) = \gamma$  , we have
$$
\begin{array}{l} P \left(Z _ {m} ^ {(k)} (z) \notin S _ {z} (\delta)\right) = (1 - \gamma) ^ {m} + C _ {m} ^ {1} \gamma (1 - \gamma) ^ {m - 1} + C _ {m} ^ {2} \gamma^ {2} (1 - \gamma) ^ {m - 2} + \dots \\ + C _ {m} ^ {k - 1} \gamma^ {k - 1} (1 - \gamma) ^ {m - k + 1}. \tag {12} \\ \end{array}
$$
Consider the  $j$ -th term of (12). Let  $C_1 \coloneqq \gamma^j / j!$ ,  $C_2 \coloneqq C_1 e^j$  and  $C_3 \coloneqq C_2 e^j (1 - \gamma)^{-j}$ . By using Stirling's approximation, we have
$$
\begin{array}{l} \lim  _ {m \rightarrow \infty} C _ {m} ^ {j} \gamma^ {j} (1 - \gamma) ^ {m - j} = \lim  _ {m \rightarrow \infty} \frac {m !}{(m - j) ! j !} \gamma^ {j} (1 - \gamma) ^ {m - j} \\ = \lim  _ {m \rightarrow \infty} C _ {1} \frac {\sqrt {2 \pi m} \left(\frac {m}{e}\right) ^ {m}}{\sqrt {2 \pi (m - j)} \left(\frac {m - j}{e}\right) ^ {m - j}} (1 - \gamma) ^ {m - j} \\ = \lim  _ {m \rightarrow \infty} C _ {2} \sqrt {\frac {m}{(m - j)}} \frac {m ^ {m}}{(m - j) ^ {m - j}} (1 - \gamma) ^ {m - j} \\ = \lim  _ {m \rightarrow \infty} C _ {2} e ^ {j} (m + j) ^ {j} (1 - \gamma) ^ {m} \\ = \lim  _ {m \rightarrow \infty} C _ {2} e ^ {j} m ^ {j} (1 - \gamma) ^ {m - j} \\ = \lim  _ {m \rightarrow \infty} C _ {3} m ^ {j} (1 - \gamma) ^ {m}. \\ \end{array}
$$
Thus, there exists  $C_4 > 0$  such that  $P(Z_m^{(k)}(z) \notin S_z(\delta)) \leq C_4 m^{k-1} (1 - \gamma)^m$  when  $m$  is large enough. It holds that
$$
\frac {C _ {4} m ^ {k - 1} (1 - \gamma) ^ {m}}{m ^ {- 2}} = C _ {4} m ^ {k + 1} (1 - \gamma) ^ {m} \rightarrow 0, \quad \text {a s} \quad m \rightarrow + \infty .
$$
Thus, for  $z \in G$  and  $n$  large enough,  $\forall \delta > 0$ , we have  $P(Z_{m}^{(k)}(z) \notin S_{z}(\delta)) = o(m^{-2})$  and
$$
\sum_ {m \geq n} P (Z _ {m} ^ {(k)} (z) \notin S _ {z} (\delta)) \leq \sum_ {m \geq n} \frac {1}{m ^ {2}},
$$
which shows  $\lim_{n\to \infty}\sum_{m\geq n}P(Z_m^{(k)}(z)\notin S_z(\delta)) = 0$  for  $z\in G$ . So by Lebesgue dominated convergence theorem, we obtain  $\lim_{n\to \infty}\int_G\sum_{m\geq n}P(Z_m^{(k)}(z)\notin S_z(\delta))p(z)dz = 0$ .
Second, we consider the second term of (11). To prove that  $P(Z \in G^c) = 0$ , we aim to construct a countable open cover of  $G^c$  and show that the probability of the random vector  $Z$  falling into each of these open balls is zero. By the property of  $G^c$ , for every  $z \in G^c$ , there exists  $r_z > 0$  such that for all  $r < r_z$ ,  $P(Z \in S_z(r)) = 0$ . Furthermore, using the separability of Euclidean space and the density of the rational number set, we can approximate  $z$  using points from  $\mathbb{Q}^{d_z}$  with  $\mathbb{Q}$  being the rational number set. Therefore, for every  $z \in G^c$ , there exist  $x \in \mathbb{Q}^{d_z} \cap S_z(\frac{r_z}{3})$  and  $r \in \mathbb{Q} \cap (\frac{r_z}{3}, \frac{2r_z}{3})$  such that  $z \in S_x(r) \subseteq S_z(r_z)$ . Because  $P(Z \in S_z(r_z)) = 0$ , we conclude that  $P(Z \in S_x(r)) = 0$ . Define
$$
\mathcal {F} := \left\{S _ {x} (r) | \exists z \in G ^ {c}, \text {s u c h t h a t} z \in S _ {x} (r) \subseteq S _ {z} \left(r _ {z}\right) \text {w i t h} x \in \mathbb {Q} ^ {d _ {z}} \text {a n d} r \in \mathbb {Q} \right\}.
$$
Note that the elements in set  $\mathcal{F}$  are mutually distinct. By the construction of  $S_{x}(r)$ ,  $\mathcal{F}$  forms a countable open cover of  $G^{c}$ , and the probability of  $Z$  falling into each open ball in  $\mathcal{F}$  is zero. Using the monotonicity and countable additivity properties of probability, we have  $P(Z\in G^{c})\leq P(Z\in \cup_{S_{x}(r)\in \mathcal{F}}S_{x}(r))\leq \sum_{S_{x}(r)\in \mathcal{F}}P(Z\in S_{x}(r)) = 0$ . Thus, we conclude that  $P(Z\in G^{c}) = 0$ .
We therefore conclude that,  $\forall \delta > 0$ ,  $\lim_{n\to \infty}P(\sup_{m\geq n}\| Z_m^{(k)}(Z) - Z\| _2 > \delta) = 0$ . This finishes the proof.
Proof. By Pinsker's inequality, we have
$$
d _ {T V} \left\{p (x | Z), \widehat {p} (x | Z) \right\} \leq \sqrt {D _ {K L} \left\{p (x | Z) , \widehat {p} (x | Z) \right\} / 2}.
$$
Note that  $I\{\xi = 1\} + \ldots + I\{\xi = k\} = 1$ . By the definition of  $\widehat{p}(x|Z)$ , we obtain
$$
\begin{array}{l} D _ {K L} \{p (x | Z), \widehat {p} (x | Z) \} = \int p (x | Z) \log \left\{\frac {p (x | Z)}{p (x | Z _ {n} ^ {(1)}) ^ {I \{\xi = 1 \}} \times \ldots \times p (x | Z _ {n} ^ {(k)}) ^ {I \{\xi = k \}}} \right\} d x \\ = \int p (x | Z) \log \prod_ {l = 1} ^ {k} \frac {p (x | Z) ^ {I \{\xi = l \}}}{p (x | Z _ {n} ^ {(l)}) ^ {I \{\xi = l \}}} d x \\ = \sum_ {l = 1} ^ {k} I \{\xi = l \} \int p (x | Z) \log \frac {p (x | Z)}{p \left(x \mid Z _ {n} ^ {(l)}\right)} d x \\ = \sum_ {l = 1} ^ {k} I \left\{\xi = l \right\} D _ {K L} \left\{p (x | Z) | | p (x | Z _ {n} ^ {(l)}) \right\}. \\ \end{array}
$$
Then, by Taylor's expansion, we have