SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} D _ {K L} \left\{p (x | Z) \mid \mid p (x \mid Z _ {n} ^ {(l)}) \right\} \\ = D _ {K L} \left\{p (x | Z) | | p (x | Z) \right\} + \frac {\partial}{\partial z ^ {\prime}} D _ {K L} \left\{p (x | Z) | | p (x | z ^ {\prime}) \right\} \Bigg | _ {z ^ {\prime} = Z} \left(Z _ {n} ^ {(l)} - Z\right) \\ + \frac {1}{2} (Z _ {n} ^ {(l)} - Z) ^ {T} \frac {\partial^ {2}}{\partial z ^ {\prime} \partial z ^ {\prime T}} D _ {K L} \{p (x | Z) | | p (x | z ^ {\prime}) \} \Bigg | _ {z ^ {\prime} = a} (Z _ {n} ^ {(l)} - Z), \\ \end{array}
$$
where  $a = \lambda Z + (1 - \lambda)Z_n^{(l)}$  with  $0\leq \lambda \leq 1$
Note that  $D_{KL}\{p(x|Z)||p(x|Z)\} = \int p(x|Z)\log \frac{p(x|Z)}{p(x|Z)} dx = 0$ . By Lemma 1 and Assumptions 1 and 2, we have
$$
\begin{array}{l} \frac {\partial}{\partial z ^ {\prime}} D _ {K L} \{p (x | Z) | | p (x | z ^ {\prime}) \} \Bigg | _ {z ^ {\prime} = Z} = - \int p (x | Z) \cdot \frac {\partial}{\partial z ^ {\prime}} \log p (x | z ^ {\prime}) \Bigg | _ {z ^ {\prime} = Z} d x \\ = - \frac {\partial}{\partial z ^ {\prime}} \int p (x | z ^ {\prime}) d x \Bigg | _ {z ^ {\prime} = Z} = 0 \\ \end{array}
$$
$$
\frac {\partial^ {2}}{\partial z ^ {\prime} \partial z ^ {\prime T}} D _ {K L} \{p (x | Z) | | p (x | z ^ {\prime}) \} \Bigg | _ {z ^ {\prime} = a} = - \int p (x | Z) \cdot \frac {\partial^ {2}}{\partial z ^ {\prime} \partial z ^ {\prime T}} \log p (x | z ^ {\prime}) \Bigg | _ {z ^ {\prime} = a} d x = I _ {a} (Z).
$$
This means
$$
D _ {K L} \{p (x | Z) | | p (x | Z _ {n} ^ {(l)}) \} = \frac {1}{2} (Z _ {n} ^ {(l)} - Z) ^ {T} I _ {a} (Z) (Z _ {n} ^ {(l)} - Z).
$$
Note that  $Z_{n}^{(l)} \to Z$  a.s. implies that  $Z_{n}^{(l)}$  converges to  $Z$  in probability. Then  $\forall \delta > 0$ , for  $\epsilon$  defined in Assumption 1, we have
$$
\begin{array}{l} P \left(D _ {K L} \left\{p (x \mid Z) \mid \mid p (x \mid Z _ {n} ^ {(l)}) \right\} > \delta\right) \leq P \left(\left\{D _ {K L} \left\{p (x \mid Z) \mid \mid p (x \mid Z _ {n} ^ {(l)}) \right\} > \delta \right\} \cap \left\{\left\| Z _ {n} ^ {(l)} - Z \right\| _ {2} \leq \epsilon \right\}\right) \\ + P \left(\| Z _ {n} ^ {(l)} - Z \| _ {2} > \epsilon\right) \\ \leq P \left(\frac {1}{2} \beta \| Z _ {n} ^ {(l)} - Z \| _ {2} ^ {2} > \delta\right) + P \left(\| Z _ {n} ^ {(l)} - Z \| _ {2} > \epsilon\right) \\ = o (1). \\ \end{array}
$$
Thus,  $D_{KL}\{p(x|Z)||p(x|Z_n^{(l)})\} = o_p(1)$
Because  $I\{\xi = l\} \leq 1$  for  $l = 1,2,\ldots ,k$ , and  $k$  is finite, we obtain
$$
D _ {K L} \{p (x | Z), \widehat {p} (x | Z) \} = \sum_ {l = 1} ^ {k} I \{\xi = l \} D _ {K L} \{p (x | Z) | | p (x | Z _ {n} ^ {(l)}) \} = o _ {p} (1).
$$
Finally, we conclude that  $d_{TV}\{p(x|Z),\widehat{p} (x|Z)\} \leq \sqrt{D_{KL}\{p(x|Z),\widehat{p} (x|Z)\} / 2} = o_p(1)$
To prove Theorem 3, the following two lemmas are needed.
Lemma 5. Let  $\hat{X}$  be drawn from  $\widehat{p}(\cdot | Z)$ , independently of  $Y$ .  $\hat{X}^{(1)}, \ldots, \hat{X}^{(B)}$  are i.i.d. samples drawn from the  $k$ -nearest-neighbor local sampling mechanism based on  $(\hat{X}, Y, Z)$ . For any statistic  $T$ , the  $B + 1$  statistics
$$
T (\dot {\boldsymbol {X}}, \boldsymbol {Y}, \boldsymbol {Z}), T (\dot {\boldsymbol {X}} ^ {(1)}, \boldsymbol {Y}, \boldsymbol {Z}), \dots , T (\dot {\boldsymbol {X}} ^ {(B)}, \boldsymbol {Y}, \boldsymbol {Z})
$$
are exchangeable conditionally on  $\mathbf{Y}$  and  $\mathbf{Z}$ .
Proof. We have that the  $B + 1$  triples  $(\dot{\pmb{X}},\pmb {Y},\pmb {Z}),(\dot{\pmb{X}}^{(1)},\pmb {Y},\pmb {Z}),\ldots ,(\dot{\pmb{X}}^{(B)},\pmb {Y},\pmb {Z})$  are i.i.d. samples drawn from the same mechanism after conditionally on  $\dot{\pmb{X}}_{()}$ $\pmb{Y}$  and  $\pmb{Z}$ , where  $\dot{\pmb{X}}_{()}$  is the order statistic of  $\dot{\pmb{X}}$ . Note that,  $T$  is measurable. Thus,  $T(\dot{\pmb{X}},\pmb {Y},\pmb {Z}),T(\dot{\pmb{X}}^{(1)},\pmb {Y},\pmb {Z}),\ldots ,T(\dot{\pmb{X}}^{(B)},\pmb {Y},\pmb {Z})$  are i.i.d. after conditionally on  $\dot{\pmb{X}}_{()}$ $\pmb{Y}$  and  $\pmb{Z}$ . Conditionally on  $\dot{\pmb{X}}_{()}$ $\pmb{Y}$  and  $\pmb{Z}$ , denote their cumulative conditional distribution as  $F(\cdot |\dot{\pmb{X}}_{()},\pmb {Y},\pmb {Z})$ . Denote  $\dot{\pmb{X}}$  as  $\dot{\pmb{X}}^{(0)}$ . Then, for any  $t_0,\dots ,t_B\in \mathbb{R}$  and
any permutation  $\pi = (\pi_{(0)},\dots ,\pi_{(B)})$  of the indices  $\{0,1,\ldots ,B\}$ , we have
$$
\begin{array}{l} P (T (\hat {X} ^ {(0)}, \mathbf {Y}, \mathbf {Z}) \leq t _ {0}, T (\hat {X} ^ {(1)}, \mathbf {Y}, \mathbf {Z}) \leq t _ {1}, \dots , T (\hat {X} ^ {(B)}, \mathbf {Y}, \mathbf {Z}) \leq t _ {B} | \mathbf {Y}, \mathbf {Z}) \\ = E _ {\hat {X} _ {()} | \boldsymbol {Y}, \boldsymbol {Z}} \left\{P \left(T \left(\hat {\boldsymbol {X}} ^ {(0)}, \boldsymbol {Y}, \boldsymbol {Z}\right) \leq t _ {0}, T \left(\hat {\boldsymbol {X}} ^ {(1)}, \boldsymbol {Y}, \boldsymbol {Z}\right) \leq t _ {1}, \dots , T \left(\hat {\boldsymbol {X}} ^ {(B)}, \boldsymbol {Y}, \boldsymbol {Z}\right) \leq t _ {B} \mid \hat {\boldsymbol {X}} _ {()}, \boldsymbol {Y}, \boldsymbol {Z}\right) \right\} \\ = E _ {\hat {X} (_ {0}) | \boldsymbol {Y}, \boldsymbol {Z}} \{P (T (\hat {\boldsymbol {X}} ^ {(0)}, \boldsymbol {Y}, \boldsymbol {Z}) \leq t _ {0} | \hat {\boldsymbol {X}} (_ {0}), \boldsymbol {Y}, \boldsymbol {Z}), \dots , P (T (\hat {\boldsymbol {X}} ^ {(B)}, \boldsymbol {Y}, \boldsymbol {Z}) \leq t _ {B} | \hat {\boldsymbol {X}} (_ {0}), \boldsymbol {Y}, \boldsymbol {Z}) \} \\ = E _ {\hat {X} (_ {0}) | \mathbf {Y}, \mathbf {Z}} \left\{\prod_ {i = 0} ^ {B} F (t _ {i} | \hat {X} (_ {0}), \mathbf {Y}, \mathbf {Z}) \right\} \\ = E _ {\hat {X} (_ {0}) | \boldsymbol {Y}, \boldsymbol {Z}} \left\{P \left(T \left(\hat {\boldsymbol {X}} ^ {\left(\pi_ {(0)}\right)}, \boldsymbol {Y}, \boldsymbol {Z}\right) \leq t _ {0}, \dots , T \left(\hat {\boldsymbol {X}} ^ {\left(\pi_ {(B)}\right)}, \boldsymbol {Y}, \boldsymbol {Z}\right) \leq t _ {B} \mid \hat {X} (_ {0}), \boldsymbol {Y}, \boldsymbol {Z}\right) \right\} \\ = P \left(T \left(\hat {\boldsymbol {X}} ^ {\left(\pi_ {(0)}\right)}, \boldsymbol {Y}, \boldsymbol {Z}\right) \leq t _ {0}, \dots , T \left(\hat {\boldsymbol {X}} ^ {\left(\pi_ {(B)}\right)}, \boldsymbol {Y}, \boldsymbol {Z}\right) \leq t _ {B} | \boldsymbol {Y}, \boldsymbol {Z}\right). \\ \end{array}
$$
Thus, the desired result follows.
Let  $\stackrel{d}{=}$  denotes equality in distribution. We present the following Lemma:
Lemma 6. For any two bi-tuples  $(\mathbf{U},\mathbf{V})$  and  $(\mathbf{U}',\mathbf{V}')$ , if  $\forall \mathbf{u}$ ,  $(\mathbf{V}|\mathbf{U} = \mathbf{u}) \stackrel{\mathrm{d}}{=} (\mathbf{V}'|\mathbf{U}' = \mathbf{u})$ , we have  $d_{TV}(\{\mathbf{U},\mathbf{V}$ ,  $(\mathbf{U}',\mathbf{V}')\} = d_{TV}(\mathbf{U},\mathbf{U}')$ .
Proof. Denote the joint density functions of  $(\pmb{U},\pmb{V})$  and  $(\pmb{U}',\pmb{V}')$  by  $p_{\pmb{U},\pmb{V}}(\pmb{u},\pmb{v})$  and  $p_{\pmb{U}',\pmb{V}'}(\pmb{u}',\pmb{v}')$ , respectively. According to the equivalent definition of the TV distance, we obtain
$$
\begin{array}{l} d _ {T V} \left\{\left(\boldsymbol {U}, \boldsymbol {V}\right), \left(\boldsymbol {U} ^ {\prime}, \boldsymbol {V} ^ {\prime}\right) \right\} = \frac {1}{2} \iint \left| p _ {\boldsymbol {U}, \boldsymbol {V}} (\boldsymbol {u}, \boldsymbol {v}) - p _ {\boldsymbol {U} ^ {\prime}, \boldsymbol {V} ^ {\prime}} (\boldsymbol {u}, \boldsymbol {v}) \right| d \boldsymbol {u} d \boldsymbol {v} \\ = \frac {1}{2} \iint \left| p _ {\boldsymbol {V} | \boldsymbol {U}} (\boldsymbol {v} | \boldsymbol {u}) p _ {\boldsymbol {U}} (\boldsymbol {u}) - p _ {\boldsymbol {V} ^ {\prime} | \boldsymbol {U} ^ {\prime}} (\boldsymbol {v} | \boldsymbol {u}) p _ {\boldsymbol {U} ^ {\prime}} (\boldsymbol {u}) \right| d \boldsymbol {u} d \boldsymbol {v} \\ = \frac {1}{2} \iint p _ {\boldsymbol {V} | \boldsymbol {U}} (\boldsymbol {v} | \boldsymbol {u}) \left| p _ {\boldsymbol {U}} (\boldsymbol {u}) - p _ {\boldsymbol {U} ^ {\prime}} (\boldsymbol {u}) \right| d \boldsymbol {u} d \boldsymbol {v} \\ = \frac {1}{2} \int \left[ \int p _ {\boldsymbol {V} | \boldsymbol {U}} (\boldsymbol {v} | \boldsymbol {u}) d \boldsymbol {v} \right] \left| p _ {\boldsymbol {U}} (\boldsymbol {u}) - p _ {\boldsymbol {U} ^ {\prime}} (\boldsymbol {u}) \right| d \boldsymbol {u} \\ = \frac {1}{2} \int \left| p _ {\boldsymbol {U}} (\boldsymbol {u}) - p _ {\boldsymbol {U} ^ {\prime}} (\boldsymbol {u}) \right| d \boldsymbol {u} \\ = d _ {T V} \left(\boldsymbol {U}, \boldsymbol {U} ^ {\prime}\right). \\ \end{array}
$$
Now we present the proof of Theorem 3:
Proof. Let  $\widetilde{\pmb{X}}^{(1)},\dots,\widetilde{\pmb{X}}^{(B)}$  be i.i.d. drawn from the  $k$ -nearest-neighbor local sampling mechanism, see Algorithm 3. Now let  $\dot{\pmb{X}}$  be an additional sample drawn from  $\widehat{p} (\cdot |\pmb {Z})$  independently of  $\pmb{Y}$ . Let  $\dot{\pmb{X}}^{(1)},\dots,\dot{\pmb{X}}^{(B)}$  be i.i.d. drawn from the  $k$ -nearest-neighbor local sampling mechanism after we observe  $\dot{\pmb{X}}$  instead of  $\pmb{X}$ . Because  $(\dot{\pmb{X}}^{(1)},\dots,\dot{\pmb{X}}^{(B)})$ , conditionally on  $\dot{\pmb{X}},\pmb{Y}$  and  $\pmb{Z}$ , is generated from the same mechanism as  $(\widetilde{\pmb{X}}^{(1)},\dots,\widetilde{\pmb{X}}^{(B)})$ , conditionally on  $\pmb{X},\pmb{Y}$  and  $\pmb{Z}$ , for all  $\pmb{x}\in \mathbb{R}^n$ , we have
$$
\left. \left(\left(\widetilde {\boldsymbol {X}} ^ {(1)}, \dots , \widetilde {\boldsymbol {X}} ^ {(B)}\right) \mid \boldsymbol {X} = \boldsymbol {x}, \boldsymbol {Y}, \boldsymbol {Z}\right) \stackrel {{\mathrm {d}}} {{=}} \left(\left(\dot {\boldsymbol {X}} ^ {(1)}, \dots , \dot {\boldsymbol {X}} ^ {(B)}\right) \mid \dot {\boldsymbol {X}} = \boldsymbol {x}, \boldsymbol {Y}, \boldsymbol {Z}\right). \right.
$$
Then, by applying Lemma 6, we obtain
$$
\begin{array}{l} d _ {T V} \left\{\left(\boldsymbol {X}, \widetilde {\boldsymbol {X}} ^ {(1)}, \dots , \widetilde {\boldsymbol {X}} ^ {(B)} | \boldsymbol {Y}, \boldsymbol {Z}\right), \left(\dot {\boldsymbol {X}}, \dot {\boldsymbol {X}} ^ {(1)}, \dots , \dot {\boldsymbol {X}} ^ {(B)} | \boldsymbol {Y}, \boldsymbol {Z}\right) \right\} \\ = d _ {T V} \left\{\left(\boldsymbol {X} \mid \boldsymbol {Y}, \boldsymbol {Z}\right), \left(\hat {\boldsymbol {X}} \mid \boldsymbol {Y}, \boldsymbol {Z}\right) \right\} = d _ {T V} \left\{p (\cdot \mid \boldsymbol {Z}), \hat {p} (\cdot \mid \boldsymbol {Z}) \right\}. \\ \end{array}
$$
Define  $\chi_{\alpha}^{B} := \left\{(\pmb{x}, \pmb{x}^{(1)}, \dots, \pmb{x}^{(B)}) \Big| \left[1 + \sum_{b=1}^{B} 1\{T(\pmb{x}^{(b)}, \pmb{Y}, \pmb{Z}) \geq T(\pmb{x}, \pmb{Y}, \pmb{Z})\}\right] / (1 + B) \leq \alpha \right\}$ ,
where  $1(\cdot)$  is the indicator function. Note that in our case, the statistic  $T$  is selected to be  $\widehat{\mathrm{CMI}}$ . Then,
$$
\begin{array}{l} P (p \leq \alpha | \boldsymbol {Y}, \boldsymbol {Z}) = P ((\boldsymbol {X}, \widetilde {\boldsymbol {X}} ^ {(1)}, \dots , \widetilde {\boldsymbol {X}} ^ {(B)}) \in \chi_ {\alpha} ^ {B} | \boldsymbol {Y}, \boldsymbol {Z}) \\ = P \left(\left(\mathring {\boldsymbol {X}}, \mathring {\boldsymbol {X}} ^ {(1)}, \dots , \mathring {\boldsymbol {X}} ^ {(B)}\right) \in \chi_ {\alpha} ^ {B} | \boldsymbol {Y}, \boldsymbol {Z}\right) + P \left(\left(\boldsymbol {X}, \widetilde {\boldsymbol {X}} ^ {(1)}, \dots , \widetilde {\boldsymbol {X}} ^ {(B)}\right) \in \chi_ {\alpha} ^ {B} | \boldsymbol {Y}, \boldsymbol {Z}\right) \\ - P \left(\left(\dot {\boldsymbol {X}}, \dot {\boldsymbol {X}} ^ {(1)}, \dots , \dot {\boldsymbol {X}} ^ {(B)}\right) \in \chi_ {\alpha} ^ {B} | \boldsymbol {Y}, \boldsymbol {Z}\right) \\ \leq P \left(\left(\hat {\boldsymbol {X}}, \hat {\boldsymbol {X}} ^ {(1)}, \dots , \hat {\boldsymbol {X}} ^ {(B)}\right) \in \chi_ {\alpha} ^ {B} | \boldsymbol {Y}, \boldsymbol {Z}\right) \\ + d _ {T V} \left\{\left(\boldsymbol {X}, \widetilde {\boldsymbol {X}} ^ {(1)}, \dots , \widetilde {\boldsymbol {X}} ^ {(B)} | \boldsymbol {Y}, \boldsymbol {Z}\right), \left(\acute {\boldsymbol {X}}, \acute {\boldsymbol {X}} ^ {(1)}, \dots , \acute {\boldsymbol {X}} ^ {(B)} | \boldsymbol {Y}, \boldsymbol {Z}\right) \right\} \\ = P \left(\left(\hat {\boldsymbol {X}}, \hat {\boldsymbol {X}} ^ {(1)}, \dots , \hat {\boldsymbol {X}} ^ {(B)}\right) \in \chi_ {\alpha} ^ {B} | \boldsymbol {Y}, \boldsymbol {Z}\right) + d _ {T V} \left\{p (\cdot | \boldsymbol {Z}), \hat {p} (\cdot | \boldsymbol {Z}) \right\}. \\ \end{array}
$$
Applying Lemma 5 and the property of rank test, we obtain  $P((\hat{\mathbf{X}}, \hat{\mathbf{X}}^{(1)}, \dots, \hat{\mathbf{X}}^{(B)}) \in \chi_{\alpha}^{B} | \mathbf{Y}, \mathbf{Z}) \leq \alpha$ . Finally, we have  $P(p \leq \alpha | \mathbf{Y}, \mathbf{Z}) \leq \alpha + d_{TV} \{p(\cdot | \mathbf{Z}), \widehat{p}(\cdot | \mathbf{Z})\}$ .
Because the TV distance is bounded by 1, marginalizing the above inequality over  $Y$  and  $Z$  and applying Theorem 2 and Lebesgue dominated convergence theorem lead to
$$
P (p \leq \alpha | H _ {0}) \leq \alpha + E \left(d _ {T V} \left\{p (\cdot | \mathbf {Z}), \widehat {p} (\cdot | \mathbf {Z}) \right\}\right) = \alpha + o (1).
$$
Here we present the assumptions given in [34] that ensure the consistency of CMI estimator. We denote the point  $(x,y,z)$  as  $\omega \in \mathbb{R}^{d_x}\times \mathbb{R}^{d_y}\times \mathbb{R}^{d_z}$ . Let  $f(\omega) = p(x,y,z)$  be the joint density function of  $(X,Y,Z)$ ,  $g(\omega) = p(x,z)p(y|z)$  be the joint density function of  $(X,Y,Z)$  under  $H_0$ , and  $\phi (\omega) = \phi (x,y,z)$  be the joint density of  $(X,Y',Z)$  produced by Algorithm 1. For a given point  $\omega$ ,  $\gamma (\omega) = P(l = 1|\omega)$  represents the predicted positive label probability generated by a classifier. When the prediction is from a classifier with parameter  $\theta$ , it is denoted as  $\gamma_{\theta}(\omega)$ . For convenience, the parameter  $\theta$  in  $\gamma$  is dropped when it is understood from the context. According to Algorithm 2,  $P(l = 1) = P(l = 0) = 1 / 2$ . Define the population binary-cross entropy loss over the joint distribution of data and label as  $\mathrm{BCE}(\gamma) = -(E_{Wl}(l\log \gamma (W) + (1 - l)\log (1 - \gamma (W)))$
Assumption (A1):  $f(\cdot)$  and  $\phi (\cdot)$  admit densities in a compact subset  $\mathcal{W}\subset \mathbb{R}^{d_x}\times \mathbb{R}^{d_y}\times \mathbb{R}^{d_z}$ .
Assumption (A2): For some constant  $\alpha, \beta > 0, \alpha \leq f(\omega), \phi(\omega) \leq \beta, \forall \omega$ .
Assumption (A3): Clip predictions such that  $\gamma (\omega)\in [\tau ,1 - \tau ],\forall \omega$  , with  $0 <   \tau \leq \alpha /(\alpha +\beta)$
Assumption (A4): The classifier class  $\mathcal{C}_{\theta}$  is parametrized by  $\theta$  within a compact domain  $\Theta \subset \mathbb{R}^h$ . There exists a constant  $K$  such that  $||\theta||_2 \leq K$ , and the classifier's output is  $L$ -Lipschitz with respect to  $\theta$ .
Assumption (A5):  $\int p(z)^{1 - 1 / d}dz\leq C_5,\forall d\geq 2$  , where  $C_5$  is a constant.
We denote the CMI estimator  $\widehat{\mathbf{CMI}}$  based on Algorithm 2 by  $\widehat{D}_{KL}^{(n)}(f||\phi)$ . The true CMI of  $(X,Y,Z)$  is  $\mathbf{CMI} = I(X;Y|Z) = D_{KL}(f||g)$ . Then we have the following Lemma:
Lemma 7. Under Assumptions 1 and 2 and (A1)-(A5), we have  $\widehat{D}_{KL}^{(n)}(f||\phi)\xrightarrow{P}D_{KL}(f||g)$ .
Proof. By the definition of convergence in probability, it is sufficient to prove  $\forall \delta > 0$ ,  $\forall \eta > 0$ ,  $\exists N$ , when  $n > N$ ,  $P(|\widehat{D}_{KL}^{(n)}(f||\phi) - D_{KL}(f||g)| > \delta) < \eta$ .
$$
\begin{array}{l} P \left(\left| \hat {D} _ {K L} ^ {(n)} (f | | \phi) - D _ {K L} (f | | g) \right| > \delta\right) \\ = P (| \widehat {D} _ {K L} ^ {(n)} (f | | \phi) - D _ {K L} (f | | \phi) + D _ {K L} (f | | \phi) - D _ {K L} (f | | g) | > \delta). \\ \end{array}
$$
Applying Theorem 1 in [34], we have  $\widehat{D}_{KL}^{(n)}(f||\phi) - D_{KL}(f||\phi)\stackrel {P}{\to}0$ , which means for  $\delta /2 > 0$  and  $\eta >0,\exists N_1$ , when  $n > N_{1}$ ,  $P(|\widehat{D}_{KL}^{(n)}(f||\phi) - D_{KL}(f||\phi)| > \delta /2) < \eta$ .
Now consider the term  $D_{KL}(f||\phi) - D_{KL}(f||g)$ . Let  $\gamma'$  be the point-wise minimizer of binary-cross entropy loss based on  $f(\omega)$  and  $\phi(\omega)$ , and  $\gamma''$  be the point-wise minimizer of binary-cross entropy
loss based on  $f(\omega)$  and  $g(\omega)$ . Applying Lemma 3 in [34], we have  $\gamma'(\omega) / \{1 - \gamma'(\omega)\} = f(\omega) / \phi(\omega)$  and  $\gamma''(\omega) / \{1 - \gamma''(\omega)\} = f(\omega) / g(\omega)$ . Then, by the definition of KL divergence, it follows that
$$
\begin{array}{l} | D _ {K L} (f | | \phi) - D _ {K L} (f | | g) | = \left| E _ {f (\omega)} \log \frac {f (\omega)}{\phi (\omega)} - E _ {f (\omega)} \log \frac {f (\omega)}{g (\omega)} \right| \\ = \left| E _ {f (\omega)} \left(\log \frac {\gamma^ {\prime} (\omega)}{1 - \gamma^ {\prime} (\omega)} - \log \frac {\gamma^ {\prime \prime} (\omega)}{1 - \gamma^ {\prime \prime} (\omega)}\right) \right| \\ \leq E _ {f (\omega)} \left| \log \frac {1 - \gamma^ {\prime} (\omega)}{\gamma^ {\prime} (\omega)} - \log \frac {1 - \gamma^ {\prime \prime} (\omega)}{\gamma^ {\prime \prime} (\omega)} \right| \\ \leq \frac {1 - \tau}{\tau} E _ {f (\omega)} \left| \frac {1 - \gamma^ {\prime} (\omega)}{\gamma^ {\prime} (\omega)} - \frac {1 - \gamma^ {\prime \prime} (\omega)}{\gamma^ {\prime \prime} (\omega)} \right| \\ = \frac {1 - \tau}{\tau} E _ {f (\omega)} \left| \frac {\phi (\omega) - g (\omega)}{f (\omega)} \right| \\ = \frac {1 - \tau}{\tau} \iiint | \phi (x, y, z) - g (x, y, z) | d x d y d z \\ = \frac {2 (1 - \tau)}{\tau} d _ {T V} (\phi , g). \\ \end{array}
$$
The second inequality follows from Lagrange's mean value theorem and Assumption (A3).
Applying Theorem 1 in [41],  $\forall \epsilon_1 \leq \epsilon$  with  $\epsilon$  being defined in Assumption 1, we have  $d_{TV}(\phi, g) \leq b(n)$ , where
$$
b (n) = \frac {1}{2} \sqrt {\frac {\beta}{4} \frac {C _ {5} 2 ^ {1 / d _ {z}} \Gamma (1 / d _ {z})}{(n \gamma_ {d _ {z}}) ^ {1 / d _ {z}} d _ {z}} + \frac {\beta \epsilon_ {1} G (2 c _ {d _ {z}} \epsilon_ {1} ^ {2})}{4}} + \exp \left(- \frac {1}{2} n \gamma_ {d _ {z}} c _ {d _ {z}} \epsilon_ {1} ^ {d _ {z} + 2}\right) + G (2 c _ {d _ {z}} \epsilon_ {1} ^ {2}).
$$
Here,  $\beta$  is defined in Assumption 1,  $C_5$  is defined in Assumption (A5),  $d_z$  is the dimension of  $Z$ ,  $\Gamma(\cdot)$  is the gamma function,  $\gamma_{d_z}$  is the volume of the unit radius  $l_2$  ball in  $\mathbb{R}^{d_z}$ ,  $c_{d_z}$  is defined in Assumption 2, and  $\forall \delta > 0$ ,  $G(\delta) = P(p(Z) \leq \delta)$ .
Because  $\epsilon_{1}$  can be arbitrary small, we conclude that  $\lim_{n\to \infty}b(n) = 0$ . So we arrive at  $\lim_{n\to \infty}|D_{KL}(f||\phi) - D_{KL}(f||g)| = 0$ , which means for  $\delta /2 > 0$ ,  $\exists N_2$ , when  $n > N_2$ ,  $|D_{KL}(f||\phi) - D_{KL}(f||g)| < \delta /2$ .
Then for  $\delta > 0$  and  $\eta > 0$ , take  $N = \max(N_1, N_2)$ , when  $n > N$ ,
$$
\begin{array}{l} P \left(\left| \widehat {D} _ {K L} ^ {(n)} (f | | \phi) - D _ {K L} (f | | \phi) + D _ {K L} (f | | \phi) - D _ {K L} (f | | g) \right| > \delta\right) \\ \leq P \left(\left| \hat {D} _ {K L} ^ {(n)} (f | | \phi) - D _ {K L} (f | | \phi) \right| + \left| D _ {K L} (f | | \phi) - D _ {K L} (f | | g) \right| > \delta\right) \\ \leq P \left(\left| \widehat {D} _ {K L} ^ {(n)} (f | | \phi) - D _ {K L} (f | | \phi) \right| > \delta / 2\right) <   \eta \\ \end{array}
$$
holds. This finish the proof.
We therefore conclude that  $\widehat{\mathbf{CMI}}$  is a consistent estimator of CMI. When considering  $\widehat{\mathbf{CMI}}^{(b)}$  based on the sample  $(\widetilde{\boldsymbol{X}}^{(b)},\boldsymbol {Y},\boldsymbol {Z})$ $(b = 1,\dots ,B)$  drawn from the  $k$  -nearest-neighbor local sampling mechanism as depicted in Algorithm 3, we can state: Under Assumptions 1, 2, (A1)-(A2) with  $f(\omega)$  and  $\phi (\omega)$  replaced by densities of the distribution of  $(\widetilde{X},Y,Z)$  and the corresponding 1-NN distribution, respectively, and Assumptions (A3)-(A5),  $\forall b = 1,\ldots ,B$ $\widehat{\mathbf{CMI}}^{(b)}$  is a consistent estimator of  $\mathbf{CMI}^{(b)}$ , where  $\mathbf{CMI}^{(b)} = I(\widetilde{X}^{(b)};Y|Z)$ . Now let's present the proof of Theorem 4.
Proof. Write  $P(\cdot | H_1)$  as  $P_{H_1}(\cdot)$ . By Markov inequality, it follows that
$$
\begin{array}{l} P _ {H _ {1}} (p > \alpha) = P _ {H _ {1}} \left(\frac {1 + \Sigma_ {b = 1} ^ {B _ {n}} 1 (\widehat {\mathbf {C M I}} ^ {(b)} \geq \widehat {\mathbf {C M I}})}{1 + B _ {n}} > \alpha\right) \\ \leq \frac {1}{\alpha \left(1 + B _ {n}\right)} E _ {H _ {1}} \left(1 + \sum_ {b = 1} ^ {B _ {n}} 1 \left(\widehat {\mathbf {C M I}} ^ {(b)} \geq \widehat {\mathbf {C M I}}\right)\right) \\ = \frac {1}{\alpha \left(1 + B _ {n}\right)} + \frac {B _ {n}}{\alpha \left(1 + B _ {n}\right)} P _ {H _ {1}} \left(\widehat {\mathbf {C M I}} ^ {(1)} \geq \widehat {\mathbf {C M I}}\right) \\ \leq \frac {1}{\alpha (1 + B _ {n})} + \frac {1}{\alpha} P _ {H _ {1}} \left(\widehat {\mathbf {C M I}} ^ {(1)} \geq \widehat {\mathbf {C M I}}\right). \\ \end{array}
$$
Because  $\widetilde{X}^{(1)}\perp Y|Z,\mathbf{CMI}^{(1)} = I(\widetilde{X}^{(1)};Y|Z) = 0.$  Then,  $\forall \delta >0$
$$
\begin{array}{l} P _ {H _ {1}} \left(\widehat {\mathbf {C M I}} ^ {(1)} \geq \widehat {\mathbf {C M I}}\right) \leq P _ {H _ {1}} \left(\left\{\widehat {\mathbf {C M I}} \leq \widehat {\mathbf {C M I}} ^ {(1)} \right\} \cap \left\{\left| \widehat {\mathbf {C M I}} ^ {(1)} - \mathbf {C M I} ^ {(1)} \right| \leq \delta \right\}\right) \\ + P _ {H _ {1}} \left(\left| \widehat {\mathbf {C M I}} ^ {(1)} - \mathbf {C M I} ^ {(1)} \right| > \delta\right) \\ \leq P _ {H _ {1}} \left(\widehat {\mathbf {C M I}} \leq \delta\right) + P _ {H _ {1}} \left(\left| \widehat {\mathbf {C M I}} ^ {(1)} - \mathbf {C M I} ^ {(1)} \right| > \delta\right). \\ \end{array}
$$
Next, we have
$$
\begin{array}{l} P _ {H _ {1}} (\widehat {\mathbf {C M I}} \leq \delta) \leq P _ {H _ {1}} \left(\left\{\widehat {\mathbf {C M I}} \leq \delta \right\} \cap \left\{\left| \widehat {\mathbf {C M I}} - \mathbf {C M I} \right| \leq \delta \right\}\right) + P _ {H _ {1}} \left(\left| \widehat {\mathbf {C M I}} - \mathbf {C M I} \right| > \delta\right) \\ \leq P _ {H _ {1}} \left(\mathrm {C M I} - \delta \leq \widehat {\mathrm {C M I}} \leq \delta\right) + P _ {H _ {1}} \left(\left| \widehat {\mathrm {C M I}} - \mathrm {C M I} \right| > \delta\right). \\ \end{array}
$$
Thus, we conclude that
$$
\begin{array}{l} P _ {H _ {1}} (p \leq \alpha) \geq 1 - \frac {1}{\alpha \left(1 + B _ {n}\right)} - \frac {1}{\alpha} \left[ P _ {H _ {1}} \left(\mathrm {C M I} - \delta \leq \widehat {\mathrm {C M I}} \leq \delta\right) \right. \\ \left. + P _ {H _ {1}} \left(\left| \widehat {\mathbf {C M I}} - \mathbf {C M I} \right| > \delta\right) + P _ {H _ {1}} \left(\left| \widehat {\mathbf {C M I}} ^ {(1)} - \mathbf {C M I} ^ {(1)} \right| > \delta\right) \right]. \\ \end{array}
$$
Under  $H_{1}$ ,  $\mathrm{CMI} > 0$ . Take  $\delta = \mathrm{CMI} / 4 > 0$ , we obtain
$$
\lim  _ {n \to \infty} P _ {H _ {1}} (p \leq \alpha) \to 1.
$$
B Additional Empirical Results
B.1 The choice of the neighbor order  $k$
To investigate the impact of the parameter  $k$  on our proposed approach, we employ a linear uniform model. To accomplish this, we generate synthetic data in the following manner:
$$
H _ {0}: X = \epsilon_ {x}, Y = \epsilon_ {y}, \text {a n d} Z \sim \operatorname {U n i f o r m} (- 1, 1),
$$
$$
H _ {1}: X = \epsilon_ {x}, Y = \alpha X + 0. 5 \epsilon_ {y}, \text {a n d} Z \sim \operatorname {U n i f o r m} (- 1, 1), \tag {13}
$$
where  $\epsilon_{x}$  and  $\epsilon_{y}$  are generated independently from the uniform distribution over the interval  $[-1, 1]$ . The parameter  $\alpha$  is randomly generated within the range of  $[0, 2]$ . As is shown in Figure 3, our method achieves effective control of type I error and exhibits the highest power under  $H_{1}$  across all dimensions when  $k = 7$ . Therefore, we consistently set  $k = 7$  in all experiments.
B.2 Empirical results for Scenario (13)