Title: K-Nearest-Neighbor Local Sampling Based Conditional Independence Testing
Venue: NeurIPS 2023
Accepted: yes
Keywords: Conditional Independence testing

causal inference

conditional mutual information

k-nearest neighbor

conditional randomization test

conditional permutation test
Abstract: Conditional independence (CI) testing is a fundamental task in statistics and machine learning, but its effectiveness is hindered by the challenges posed by high-dimensional conditioning variables and limited data samples. This article introduces a novel testing approach to address these challenges and enhance control of the type I error while achieving high power under alternative hypotheses. The proposed approach incorporates a computationally efficient classifier-based conditional mutual information (CMI) estimator, capable of capturing intricate dependence structures among variables. To approximate a distribution encoding the null hypothesis, a $k$-nearest-neighbor local sampling strategy is employed. An important advantage of this approach is its ability to operate without assumptions about distribution forms or feature dependencies. Furthermore, it eliminates the need to derive asymptotic null distributions for the estimated CMI and avoids dataset splitting, making it particularly suitable for small datasets. The method presented in this article demonstrates asymptotic control of the type I error and consistency against all alternative hypotheses. Extensive analyses using both synthetic and real data highlight the computational efficiency of the proposed test. Moreover, it outperforms existing state-of-the-art methods in terms of type I and II errors, even in scenarios with high-dimensional conditioning sets. Additionally, the proposed approach exhibits robustness in the presence of heavy-tailed data.

K -Nearest-Neighbor Local Sampling Based Conditional Independence Testing

Abstract

Conditional independence (CI) testing is a fundamental task in statistics and machine learning, but its effectiveness is hindered by the challenges posed by high-dimensional conditioning variables and limited data samples. This article introduces a novel testing approach to address these challenges and enhance control of the type I error while achieving high power under alternative hypotheses. The proposed approach incorporates a computationally efficient classifier-based conditional mutual information (CMI) estimator, capable of capturing intricate dependence structures among variables. To approximate a distribution encoding the null hypothesis, a k -nearest-neighbor local sampling strategy is employed. An important advantage of this approach is its ability to operate without assumptions about distribution forms or feature dependencies. Furthermore, it eliminates the need to derive asymptotic null distributions for the estimated CMI and avoids dataset splitting, making it particularly suitable for small datasets. The method presented in this article demonstrates asymptotic control of the type I error and consistency against all alternative hypotheses. Extensive analyses using both synthetic and real data highlight the computational efficiency of the proposed test. Moreover, it outperforms existing state-of-the-art methods in terms of type I and II errors, even in scenarios with high-dimensional conditioning sets. Additionally, the proposed approach exhibits robustness in the presence of heavy-tailed data.

1 Introduction

Testing for conditional independence (CI) is a crucial and challenging task in statistics and machine learning, with wide-ranging applications in graphical models [26, 17], causal inference [44, 37, 22], and variable selection [11, 24]. The objective is to determine whether two random variables, X and Y , are independent given a set of conditioning variables Z , based on observations of the joint

distribution p X,Y,Z ( x, y, z ) . Specifically, the hypothesis to be tested is:

However, CI testing becomes challenging due to the high-dimensionality of the conditioning variables Z , and the limited availability of data samples [5, 35, 40, 1]. Existing tests may struggle to control the type I error, particularly when handling high-dimensional conditioning variable sets with complex dependency structures [6, 40]. Moreover, even when a test is valid, the limited data availability can make it exceedingly challenging to distinguish between null and alternative hypotheses, resulting in low testing power [40].

In this article, we present a novel conditional independence (CI) testing method based on a k -nearestneighbor local sampling scheme. Our approach incorporates two essential components: a conditional mutual information (CMI) estimator utilizing classification techniques, and a k -nearest-neighbor local sampling strategy to approximate the conditional distribution p X | Z ( x | z ) . This approximation enables us to simulate data sets from a distribution that represents the null hypothesis, allowing us to estimate the CMI using these simulated data sets. By comparing the CMI estimator based on real data with those computed from the simulated data sets, we make informed decisions regarding the hypothesis testing. Theoretical analysis demonstrates that our proposed test achieves a valid control of the type I error asymptotically and exhibits consistency against all alternatives in H 1 . Synthetic and real data analyses showcase that our test outperforms previous methods in terms of both the type I error and power under H 1 , even when dealing with high-dimensional conditioning sets and/or small datasets. Furthermore, our approach remains computationally efficient as the dimension of the conditioning set and/or sample size increase. Additionally, our method is robust even in the presence of heavy-tailed data.

2 Related Work

2.1 CMI estimation

In our work, we utilize CMI as a measure of conditional dependence. CMI offers a strong theoretical guarantee for CI testing, where I ( X ; Y | Z ) = 0 ⇐⇒ X ⊥ ⊥ Y | Z [14]. It has the ability to capture complex dependence structures among variables, even in non-linear scenarios. Estimators based on k -nearest neighbors and kernel methods have been widely employed for CMI estimation [27, 38, 23, 19]. However, these methods may encounter efficiency issues when dealing with highdimensional conditioning variables, known as the curse of dimensionality [34, 32]. [32] approached the CMI estimation problem by formulating it as a minimax optimization problem and proposed the use of generative adversarial networks (GANs) to optimize and obtain the CMI estimator, which can handle high-dimensional scenarios. However, the training of GANs is often challenging, with the risk of collapse if hyperparameters and regularizers are not carefully chosen [15]. Recently, [34, 30] proposed a classifier-based CMI estimator capable of handling high dimensions. They initially developed an estimator for the Kullback-Leibler (KL) divergence using a classifier and then derived mutual information (MI) estimators based on it. CMI is defined as the difference between two MI values, and the CMI estimator is obtained by computing the difference between two MI estimators. In contrast, our method directly utilizes the classifier-based KL-divergence estimator to obtain the CMI estimator. This approach also handles high dimensions effectively and is computationally more efficient compared to the method employed by [34, 30].

2.2 CI Testing

In recent years, a considerable body of literature on CI testing has emerged. Here, we provide a brief overview of some existing methods, and for a more comprehensive review, we refer readers to [28].

One important category of CI testing methods involves using different measures for CI [3, 46, 2]. For example, [18] proposed a CI testing method based on the empirical Hilbert-Schmidt norm of the conditional cross-covariance operator. Another approach, KCIT, was introduced by [47], which utilizes the partial association of reproducing kernel Hilbert spaces to measure conditional independence. [40] proposed a novel kernel-based CI testing method using the l p distance between two well-chosen analytic kernel mean embeddings evaluated at a finite set of locations. [42] proposed the generalized covariance measure for CI testing based on a regression method. However, obtaining the exact distribution of the test statistic derived from the conditional independence measure under

H 0 can be challenging. Instead, researchers have employed alternative methods to derive asymptotic distributions of the test statistics under the null hypothesis [47, 46, 45, 40]. Nevertheless, the effectiveness of asymptotic distributions can be compromised when the sample size is small or the dimension of Z is high [16, 38]. As a result, tests based on asymptotic distributions may exhibit inflated type-I errors or inadequate power when dealing with small sample sizes or high-dimensional Z [47, 40]. In our work, we choose to utilize CMI as a metric for conditional dependence.

To overcome the reliance on asymptotic null distributions, [11] proposed the model-X framework, assuming knowledge of the conditional distribution p X | Z ( x | z ) . Under this assumption, a set of test statistics can be computed that are exchangeable under the null hypothesis, either through direct resampling [11] or permutation methods [8]. However, in practice, the distribution of X | Z is rarely available. Therefore, accurately approximating the distribution of X | Z becomes crucial for maintaining the type I error control of these tests. To address this challenge, [5] developed the generative conditional independence test (GCIT) using Wasserstein GANs to approximate the distribution of X | Z , while [43] proposed using Sinkhorn GANs for the same purpose. However, as highlighted in [5, 30], limited data and noise may lead to inaccurate learning of conditional distributions using neural networks, resulting in inflated type I errors for GCIT and the double GANsbased CI test (DGCIT). To mitigate this issue, [30] proposed using the 1-nearest neighbor method to generate samples from the approximated conditional distribution of X given Z . However, their method requires splitting the dataset into two parts, with the testing dataset used for calculating the test statistics comprising only one-third of the total samples. This will reduce the statistical power of the test, particularly when working with small datasets. [38] utilized a Kozachenko-Leonenko estimator for CMI as the test statistic and proposed a permutation scheme based on k -nearest neighbors to generate samples under the null distribution. However, the curse of dimensionality adversely affects the performance of the Kozachenko-Leonenko CMI estimator, leading to poor performance when the conditioning variable set Z has high dimensions. Additionally, no theoretical guarantee is provided. [31] and [25] employed a binning strategy, discretizing Z into a finite number of bins based on the proximity of conditioning variables Z , followed by the "permute-within-groups" strategy. However, selecting bins in high-dimensional settings presents a significant challenge [8], and their methods are limited to handling conditioning variables Z with very few dimensions.

In our work, we propose the utilization of a k -nearest-neighbor local sampling strategy as an appealing alternative to the binning strategy. Specifically, our strategy generates samples locally based on the k -nearest neighbors of the conditioning variables Z . We demonstrate that the distribution of samples generated from this k -nearest-neighbor local sampling scheme closely approximates the true conditional distribution p X | Z ( x | z ) in terms of total variation distance. One significant advantage of our proposed method is that it allows the entire dataset to be used for computing the testing statistics, eliminating the need for dataset splitting. This feature makes our method more effective compared to [30], which requires dataset splitting, particularly when dealing with small datasets. Moreover, our method does not require the derivation of asymptotic null distributions for the estimated CMI, and it can easily handle high-dimensional conditioning variables Z . We provide theoretical and empirical evidence that our test achieves a valid control of the type I error and attains high power under the alternative hypothesis H 1 .

3 K -Nearest-Neighbor Local Sampling Based CI Testing

3.1 Classifier-based CMI estimator

The CMI for a triplet of random variables/vectors ( X,Y,Z ) is defined as:

where p X,Y,Z ( x, y, z ) is the joint density of ( X,Y,Z ) , p X,Z ( x, z ) is the joint density of ( X,Z ) , and p Y | Z ( y | z ) is the conditional density of Y given Z = z . One approach to estimate CMI is by directly estimating the joint and conditional densities from the available data and plugging them into (2). However, accurately estimating the density functions can be challenging, especially in high-dimensional settings where it is more difficult than directly estimating CMI (2) [34]. CMI is a special case of the Kullback-Leibler (KL) divergence, and thus we have:

Algorithm 1 1-Nearest-Neighbor sampling ( 1-NN( V 1 , V 2 , n ) )

Input : Datasets V 1 and V 2 , both with sample size n and V = V 1 ∪ V 2 consisting of 2 n independently and identically distributed (i.i.d.) samples from p X,Y,Z ( x, y, z ) .

Output : A new data set V ′ consists of n samples.

1: Let V ′ = ∅ .

2: for ( X,Y,Z ) in V 2 do

4: V ′ = V ′ ∪ { ( X,Y ′ , Z ) } .

3: Go to V 1 to find the sample ( X ′ , Y ′ , Z ′ ) such that Z ′ is the 1-nearest neighbor of Z in terms of the l 2 norm

5: end for

6: return V ′

where D KL ( f || g ) denotes the KL divergence between two distribution functions F and G , with density functions f ( x ) and g ( x ) , respectively. The Donsker-Varadhan (DV) representation of D KL ( f || g ) is given by:

where the function class S includes all functions with finite expectations. The optimal function in (4) is given by s ∗ ( x ) = log { f ( x ) /g ( x ) } [4], which leads to:

Building upon the classifier-based CMI estimation method proposed by [34], we propose a CMI estimation method using the 1-nearest-neighbor (1-NN) sampling algorithm [41, 30]. The pseudocode for estimating CMI is outlined in Algorithm 2. The main objective is to empirically estimate (5) with f = p X,Y,Z ( x, y, z ) and g = p X,Z ( x, z ) p Y | Z ( y | z ) , which requires samples from both p X,Y,Z ( x, y, z ) and p X,Z ( x, z ) p Y | Z ( y | z ) . The available data only consists of samples from p X,Y,Z ( x, y, z ) . However, generating samples from p X,Z ( x, z ) p Y | Z ( y | z ) requires knowledge of the unknown conditional distribution p Y | Z ( y | z ) . To address this challenge, we propose utilizing the 1-NN sampling algorithm [41, 30], which is outlined in Algorithm 1. Empirical and theoretical results presented in [30] indicate that the 1-NN sampling algorithm can accurately approximate the conditional distribution.

Next, we formalize the classifier-based CMI estimator. We consider a data set V consisting of 2 n i.i.d. samples { W i := ( X i , Y i , Z i ) } 2 n i =1 with ( X i , Y i , Z i ) ∼ p X,Y,Z ( x, y, z ) . The data set V is divided into two equally sized parts V 1 and V 2 , where | V 1 | = | V 2 | = n . For data sets V 1 and V 2 , we use the 1-NN sampling algorithm 1 to generate a new data set V ′ with n samples. We assign labels l = 1 for all samples in V 2 and l = 0 for all samples in V ′ . In this supervised classification task, a binary classifier can be trained using an advanced binary classification model, such as XGBoost [41, 12] or deep neural networks [21]. The classifier produces predicted probability α m = P ( l = 1 | W m ) for a given sample W m , leading to an estimator of the likelihood ratio on W m given by ̂ L ( W m ) = α m / (1 -α m ) . It follows from (3) and (5) that an estimator of I ( X ; Y | Z ) is given by

where d = ⌊ n/ 3 ⌋ with ⌊ t ⌋ being the largest integer not greater than t , W f i is a sample in V test f and W g j is a sample in V test g , where V test f and V test g are defined in Algorithm 2. According to Theorem 1 in [34], ̂ I ( X ; Y | Z ) is a consistent estimator of I ( X ; Y | Z ) .

In contrast to the approach taken by [34, 30], which estimate CMI by using the difference between two mutual informations (i.e., I ( X ; Y | Z ) = I ( X ; Y, Z ) -I ( X ; Z ) ) and therefore require training two binary classifiers, our method achieves CMI estimation with just a single binary classifier. This leads to significantly improved computational efficiency. As depicted in Algorithm 3, calculating a single p -value requires the computation of ( B +1 ) CMIs. Hence, in practical applications involving real data analysis, especially when dealing with large sample sizes or high-dimensional conditional variables, efficient computation of the CMI becomes crucial.

Algorithm 2 Classifier-based CMI Estimator

Input : Dataset V containing 2 n i.i.d. samples drawn from p X,Y,Z ( x, y, z ) . Output : An estimator of CMI.

1: Equally split V into two parts V 1 and V 2 , each containing n samples.

2: Apply Algorithm 1 to generate a new dataset V ′ with | V ′ | = n .

3: Form the labeled datasets V f = { ( W f i , l = 1) : W f i ∈ V 2 } and V g = { ( W g j , l = 0) : W g j ∈ V ′ } .

4: Divide V f into training and testing subsets V train f and V test f , at a ratio of 2:1.

5: Similarly, split V g into training and testing subsets V train g and V test g , at a ratio of 2:1.

6: Merge the datasets to form V train = V train f ∪ V train g and V test = V test f ∪ V test g .

7: Train the classifier C using V train .

8: For each w ∈ V test 0 , where V test 0 includes all features in V test , compute the classifier-based predicted probability P ( l = 1 | w ) .

9: Calculate ̂ I ( X ; Y | Z ) as per formula (6).

10: return ̂ I ( X ; Y | Z ) .

3.2 The Proposed CI Testing Procedure

While we have proposed a consistent estimator for CMI, accurately estimating it as zero under the null hypothesis, H 0 , is practically unattainable due to random errors in the sample data. These errors lead to deviations of the estimator from the actual value. To enhance the effectiveness of CI testing based on the CMI estimator, we propose a test that takes into account the statistical variation inherent in the CMI estimator.

Under H 0 , we can express the following representation:

The null model in (7) preserves the dependence between X and Z , while breaking any dependence between X and Y . Therefore, if a direct causal link exists between X and Y , replacing X with a null sample X ∗ ∼ p X | Z ( x | z ) would disrupt this relationship. Thus, we can conclude that X ∗ and Y are conditionally independent given Z , i.e., I ( X ∗ ; Y | Z ) = 0 .

Consider n i.i.d. copies A = { ( x i , y i , z i ) : i = 1 , . . . , n } of ( X,Y,Z ) . When the distribution of X | Z is known, conditional on Z = ( z 1 , . . . , z n ) T , we can independently draw a pseudo sample x ( b ) i ∼ p X | Z ( x | z i ) for each i across b = 1 , . . . , B such that all X ( b ) := ( x ( b ) 1 , . . . , x ( b ) n ) T are independent of Y := ( y 1 , . . . , y n ) T and also X := ( x 1 , . . . , x n ) T , where B is the number of repetitions. Under H 0 , ( X ( b ) , Y , Z ) d = ( X , Y , Z ) for all b , where d = denotes equality in distribution. We denote the CMI estimator of I ( X ; Y | Z ) based on ( X ( b ) , Y , Z ) as ˜ CMI ( b ) and denote the estimator based on ( X , Y , Z ) as ̂ CMI. We can approximate the p -value by

where 1( · ) is the indicator function. To begin with, we demonstrate that the test based on (8) can effectively control the type I error. Specifically, under the null hypothesis H 0 , the ( B +1 ) triples ( X , Y , Z ) , ( X (1) , Y , Z ) , . . . , ( X ( B ) , Y , Z ) are exchangeable, and thus the p -value is valid and satisfies P ( p ≤ α | H 0 ) ≤ α for any given α ∈ (0 , 1) [11, 8, 43]. Furthermore, it intuitively suggests that our test can achieve high power under the alternative hypothesis H 1 . As n approaches infinity, ˜ CMI ( b ) converges to zero in probability. But under H 1 , we know that I ( X ; Y | Z ) > 0 . As a result, ̂ CMI should be positive with high probability. Consequently, the p -value calculated using (8) is very small with high probability, indicating the consistency of our test against all alternatives stated in H 1 .

However, p X | Z ( x | z ) is rarely known in practice [5, 43]. We propose using the k -nearest-neighbor local sampling strategy to approximate p X | Z ( x | z ) . The approximated distribution is denoted as ̂ p X | Z ( x | z ) . To generate pseudo samples from ̂ p X | Z ( x | z ) , we follow the steps below:

:

Obtain the set of indices of the k -nearest neighbors of z i based on the l 2 norm. Denote this set as M i = { j ∈ { 1 , ..., n } : ∥ z j -z i ∥ 2 ≤ d k i } , where d k i is the distance of z i to its k -nearest neighbor;

Shuffle M i and let j be the first element of M i . Then, define the pseudo sample ˜ x i as x j ;

Repeat the above two steps for all i ∈ { 1 , 2 , . . . , n } to obtain the resultant pseudo sample vector ˜ X = (˜ x 1 , . . . , ˜ x n ) T .

We establish in Section 4 that the total variation distance between the true distribution of X | Z and the distribution of samples generated by the k -nearest-neighbor local sampling strategy tends to zero in probability as n goes to infinity. This evidence suggests that the latter is a close approximation of the former. Thus, ( ˜ X , Y , Z ) approximates directly a distribution that encodes the null hypothesis. We then compute the CMI estimator based on it. This process is repeated B times, resulting in B realizations of the CMI estimator under the null hypothesis denoted by ( ̂ CMI (1) , . . . , ̂ CMI ( B ) ) . We can determine whether to reject the null hypothesis by comparing these estimators to the one of the original sample set A . Specifically, we calculate the p -value using (8), but with ˜ CMI ( b ) replaced by ̂ CMI ( b ) . If the resulting p -value is smaller than the prespecified significance level, we reject the null hypothesis. The pseudo code is presented in Algorithm 3. In Section 4, we will demonstrate through theoretical analysis that our test asymptotically achieves a valid control of type I error and is consistent against all the alternatives in H 1 .

[30] employed the 1-NN sampling strategy to approximate the distribution of X | Z . To ensure substantial dissimilarity between the pseudo-sample datasets generated across repetitions, their method requires dividing the dataset into two parts. As a consequence, the dataset used for calculating the test statistics consists of only one-third of the total samples. This limitation can result in reduced statistical power of the test, especially when working with small datasets. In contrast, our proposed procedure eliminates the need for dataset splitting and allows the entire dataset to be used in computing ̂ CMI, thereby avoiding the loss of testing power.

4 Theoretical Results

In this section, we present our main theoretical results. All the detailed proofs for these results can be found in the Supplementary Materials. Let P 1 and P 2 be two probability distributions defined on the same probability space. The total variation distance between P 1 and P 2 is defined as d TV ( P 1 , P 2 ) = sup A ⊂ Ω | P 1 ( A ) -P 2 ( A ) | , where the supremum is taken over all measurable subsets A of the sample space Ω . We will show in Theorem 2 that the distribution of ˜ X generated by the k -nearest-neighbor local sampling strategy is very close to the true conditional distribution in terms of the total variation distance.

Lemma 1 plays a crucial role in proving Theorem 2. It demonstrates that the k -nearest neighbor of Z has a similar property to the work of [13], where it has been shown that the nearest neighbor of Z converges almost surely (a.s.) to Z as the sample size n approaches infinity.

Lemma 1. Let Z and Z 1 , . . . , Z n be i.i.d. random vectors from p ( z ) . For a given positive integer k , let Z ( k ) n be the k -nearest neighbor of Z from the set { Z 1 , . . . , Z n } . Then, Z ( k ) n a.s. -→ Z as n →∞ .

We make the following regularity conditions, which have been introduced in [19], [20] and [41]. To simplify the notation, we may drop the subscripts when referring to probability density functions.

Assumption 1 (Smoothness on p ( x | z ) ). For any z ∈ R d z and any a such that ∥ a -z ∥ 2 ≤ ϵ , we assume 0 ≤ λ max ( I a ( z )) ≤ β , where d z is the dimension of the random vector Z , ∥·∥ 2 denotes the l 2 norm, 0 < β < ∞ , and I a ( z ) , called the generalized curvature matrix, is defined with entries

Assumption 2 (Smoothness on p ( z ) ). Assume that the probability density function p ( z ) is twice continuously differentiable. Let H p ( z ) denote the Hessian matrix of p ( z ) with respect to z . We assume that ∥ H p ( z ) ∥ 2 ≤ c d z holds almost everywhere, where c d z depends only on the dimension d z of the random vector Z .

Note that Assumption 1 can be viewed as an extension of the requirement that the maximum eigenvalue of the Fisher information matrix for x with respect to z is bounded. Assumptions 1 and 2 can be validated when ( X,Z ) follows a multivariate Gaussian distribution (MVD) and when Z follows a MVD, respectively.

For Z , we define Z ( l ) n as its l -nearest neighbor for l = 1 , ..., k . According to Algorithm 3, each Z ( l ) n is selected with probability 1 /k . Let ξ be a random variable with a probability mass function P ( ξ = l ) = 1 /k for l = 1 , ..., k , and ξ is independent of both Y and Z . Given Z , we denote by ˜ X the sample generated by the k -nearest-neighbor local sampling mechanism, which follows the distribution ̂ p ( x | Z ) = p ( x | Z (1) n ) 1 { ξ =1 } × . . . × p ( x | Z ( k ) n ) 1 { ξ = k } . The sketch proof of Theorem 2 proceeds as follows. We first apply Pinsker's inequality, which relates the total variation distance between p ( x | Z ) and ̂ p ( x | Z ) to their KL divergence. We then establish a connection between the KL divergence and the discrepancy between Z and Z ( l ) n . Finally, by Lemma 1, we derive Theorem 2.

Theorem 2. Under Assumptions 1 and 2, we have d TV { p ( x | Z ) , ̂ p ( x | Z ) } = o p (1) as n →∞ .

Remark 1. The constant β in Assumption 1 is used to establish an upper bound on the KL divergence between p ( x | Z ) and p ( x | Z ( l ) n ) , which can be used to bound d TV { p ( x | Z ) , ̂ p ( x | Z ) } through Pinsker's inequality.

In Theorem 3, we bound the excess type I error conditionally on Y and Z by the total variation distance between ̂ p ( ·| Z ) and p ( ·| Z ) .

Theorem 3. Assume H 0 : X ⊥ ⊥ Y | Z is true. Under Assumptions 1 and 2, for any significance level α ∈ (0 , 1) , the p -value obtained from Algorithm 3 satisfies P ( p ≤ α | Y , Z ) ≤ α + d TV { p ( ·| Z ) , ̂ p ( ·| Z ) } .

The type I error rate can be unconditionally controlled based on Theorems 2 and 3, which implies that P ( p ≤ α | H 0 ) ≤ α + o (1) as n approaches infinity. Now we turn to analyze the power of our test in asymptotic scenarios in Theorem 4. Some intuitive explanations on both type I error rate and power can be found in the third paragraph of Section 3.2.

Theorem 4. For any α ∈ (0 , 1) and the number of repetitions B n satisfying B n →∞ , let p be the p -value calculated in Algorithm 3. Under Assumptions 1 and 2 and the Assumptions stated in the Supplementary Materials, P ( p ≤ α | H 1 ) → 1 as n →∞ .

Remark 2. The condition B n →∞ is mild and has been made in [7, 43]. The consistency of our test heavily relies on the consistency of the CMI estimator, which hinges on whether the joint density of ( X,Y ′ , Z ) generated by the 1-NN sampling (Algorithm 1), denoted as ϕ , approximates p ( x, z ) p ( y | z ) (denoted as g ) well, in terms of TV distance. β and c d z in Assumptions 1 and 2 are used to bound d TV ( ϕ, g ) .

Remark 3. According to Theorem 4, our test can achieve consistency against all the alternatives stated in H 1 . In contrast, the method proposed by [43] only achieves consistency against a subset of the alternatives in H 1 , and [30] does not provide any theoretical results on the testing power.

5 Empirical Results

In this section, we present a comparative evaluation of our proposed method against the state-of-the-art (SOTA) methods on synthetic datasets. Our code is publicly available at: https://github.com/LeeShuaikenwitch/NNLSCIT. We specifically compare our method with six commonly used competitive CI testing methods: KCIT [47], LPCIT[40], CCIT [41], CMIknn [38], GCIT [5], and NNSCIT [30]. We set the number of repetitions B = 200 and the neighbor order k = 7 for our tests. The XGBoost classifier was used in all of our experiments. Further elaboration on the choice of k is given in Figure 3 in the Supplementary Materials. We set the significance level to α = 0 . 05 and report the type I error rate and the testing power under H 1 for all methods evaluated in our experiments. All the results are presented as an average over 200 independent trials. We provide additional simulation studies and real data analysis in the Supplementary Materials.

Scenario I: the post-nonlinear model. The datasets used in our experiments are generated using the post-nonlinear model similar to those in [47, 16, 5, 40]. Specifically, we define ( X,Y,Z ) under H 0 and H 1 as follows:

where Z represents the sample mean of Z = ( z 1 , . . . , z d z ) , all z l in Z , ϵ x , ϵ y and ϵ b are i.i.d. samples generated from the standard Gaussian or the standard Laplace distribution, and f 1 and f 2 are randomly sampled from the set { x, x 2 , x 3 , tanh( x ) , cos( x ) } .

We conduct a comparative analysis of various tests from two perspectives. The results are shown in Figure 1. First, we fix the sample size at n = 500 and vary the dimension of Z from 30 to 100 . In both the Gaussian and Laplace cases, our test shows good and stable performance in controlling type I error and achieving high power under H 1 as the dimension of Z increases. LPCIT and NNSCIT have satisfactory performance in controlling type I error, but LPCIT loses power under H 1 when the dimension exceeds 70 , and NNSCIT has inadequate power for all dimensions. Although GCIT, CCIT, and KCIT have adequate power under H 1 , they inflate the type I error in almost all scenarios. CMIknn shows weak performance on both type I error and testing power. Figure 6 in the Supplementary Materials contains additional results on the performance of various tests in low dimensions of Z ranging from 5 to 30, further demonstrating the superiority of our test. Second, we vary the sample size from 300 to 2000 while fixing the dimension of Z at 80 . Our test maintains good control of the type I error and achieves high power, with the power approaching 1 when the sample size exceeds 500 . However, LPCIT, NNSCIT, and CMIknn lose power under H 1 , while GCIT, CCIT, and KCIT either always or sometimes fail to control the type I error well.

In the Supplementary Materials, Figure 7 shows the timing performance of all methods for a single test. Our test is found to be highly computationally efficient even when dealing with large sample sizes and high-dimensional conditioning sets. In contrast, CMIknn and CCIT for sample sizes exceeding 1000 , and LPCIT for dimension of Z higher than 50 are impractical due to their prohibitively long running time.

Scenario II: the heavy tailed model. We compare the performance of our test with the SOTA tests under the heavy tailed error distributions, as described in [10]. The data ( X,Y,Z ) is generated

30

40

50

60

70

80

90

100

Dimension of Z

0.00

0.05

0.10

0.15

0.20

0.25

0.30

0.35

Type I error

Gauss, samples=500

GCIT

CCIT

KCIT

CMIknn

LPCIT

NNSCIT

Ours

300 500

1000

1500

2000

Sample size

0.00

0.05

0.10

0.15

0.20

0.25

0.30

0.35

Type I error

Gauss, dz=80

30

40

50

60

70

80

90

100

Dimension of Z

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

Power

Gauss, samples=500

300 500

1000

1500

2000

Sample size

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

Power

Gauss, dz=80

30

40

50

60

70

80

90

100

Dimension of Z

0.00

0.05

0.10

0.15

0.20

0.25

0.30

0.35

0.40

0.45

Type I error

Laplace, samples=500

GCIT

CCIT

KCIT

CMIknn

LPCIT

NNSCIT

Ours

300 500

1000

1500

2000

Sample size

0.00

0.05

0.10

0.15

0.20

0.25

0.30

0.35

0.40

0.45

Type I error

Laplace, dz=80

30

40

50

60

70

80

90

100

Dimension of Z

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

Power

Laplace, samples=500

300 500

1000

1500

2000

Sample size

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

Power

Laplace, dz=80

according to the following model:

where Z is the sample mean of Z = ( z 1 , . . . , z d z ) , z l 's are i.i.d. samples generated from the standard Gaussian distribution, and ϵ 1 and ϵ 2 are independently generated from the standard Cauchy distribution. We keep the sample size fixed at n = 500 .

5

10

15

20

25

30

40

50

Dimension of Z

0.00

0.05

0.10

0.15

0.20

0.25

0.30

0.35

0.40

Type I error

Cauchy, samples=500

GCIT

CCIT

KCIT

CMIknn

LPCIT

NNSCIT

Ours

5

10

15

20

25

30

40

50

Dimension of Z

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

Type I error

Gauss, samples=500

5

10

15

20

25

30

40

50

Dimension of Z

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

Power

Cauchy, samples=500

5

10

15

20

25

30

40

50

Dimension of Z

0.4

0.5

0.6

0.7

0.8

0.9

1.0

Power

Gauss, samples=500

Scenario III: the chain structure. We further use the chain structure Y → Z → X in [30] to validate the proposed test. The detailed procedure for generating the simulated data can be found in the Supplementary Materials. The sample size is fixed at n = 500 .

Figure 2 demonstrates that our test effectively controls the type I error and obtains adequate power in both Scenarios II and III under various dimensions of Z .

6 Conclusion

In this paper, we introduce a novel method for conducting the conditional independence testing. Theoretical analysis shows that our test is asymptotically valid and consistent against all alternatives in H 1 . Extensive experiments on both synthetic and real datasets demonstrate that our method consistently outperforms commonly used SOTA methods. Moreover, our approach maintains computational efficiency even when the sample size and/or dimension of the conditioning set increase. Our method has the potential to enhance the applicability of causal discovery to real-world problems, such as gene regulatory networks or complex social networks, and to facilitate the discovery of relationships and patterns in complex systems. Ethically, we believe that our rather fundamental work has minimal potential for misuse.

Acknowledgments

Dr. Ziqi Chen's work was partially supported by the National Key R&D Program of China (2021YFA1000100 and 2021YFA1000101), National Natural Science Foundation of China (NSFC) (12271167, 72331005 and 11871477), Natural Science Foundation of Shanghai (21ZR1418800) and Basic Research Project of Shanghai Science and Technology Commission (22JC1400800). Dr. Christina Dan Wang's work was partially supported by the National Natural Science Foundation of China (NSFC) (12271363 and 11901395). We thank the anonymous reviewers for their helpful comments.

References

Chunrong Ai, Li-Hsien Sun, Zheng Zhang, and Liping Zhu. Testing unconditional and conditional independence via mutual information. Journal of Econometrics , 2022.

Mona Azadkia and Sourav Chatterjee. A simple measure of conditional dependence. The Annals of Statistics , 49(6):3070-3102, 2021.

Kunihiro Baba, Ritei Shibata, and Masaaki Sibuya. Partial correlation and conditional correlation as measures of conditional independence. Australian & New Zealand Journal of Statistics , 46(4):657-664, 2004.

Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeshwar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and Devon Hjelm. Mutual information neural estimation. In International Conference on Machine Learning , pages 531-540. PMLR, 2018.

Alexis Bellot and Mihaela van der Schaar. Conditional independence testing using generative adversarial networks. In Advances in Neural Information Processing Systems , 32, 2019.

Wicher Pieter Bergsma. Testing conditional independence for continuous random variables . Eurandom, 2004.

Thomas B Berrett and Richard J Samworth. Nonparametric independence testing via mutual information. Biometrika , 106(3):547-566, 2019.

Thomas B Berrett, Yi Wang, Rina Foygel Barber, and Richard J Samworth. The conditional permutation test for independence while controlling for confounders. Journal of the Royal Statistical Society: Series B (Statistical Methodology) , 82(1):175-197, 2020.

Catherine L Blake and Christopher J Merz. Uci repository of machine learning databases, 1998.

Zhanrui Cai, Runze Li, and Yaowu Zhang. A distribution free conditional independence test with applications to causal discovery. Journal of Machine Learning Research , 23(85):1-41, 2022.

Emmanuel Candès, Yingying Fan, Lucas Janson, and Jinchi Lv. Panning for gold:'model-x' knockoffs for high dimensional controlled variable selection. Journal of the Royal Statistical Society: Series B (Statistical Methodology) , 80(3):551-577, 2018.

Tianqi Chen and Carlos Guestrin. Xgboost: A scalable tree boosting system. In Proceedings of the 22Nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , pages 785-794. ACM, 2016.

Thomas Cover and Peter Hart. Nearest neighbor pattern classification. IEEE transactions on information theory , 13(1):21-27, 1967.

Thomas M Cover and Joy A Thomas. Elements of Information Theory . John Wiley & Sons, 2012.

Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. In Advances in Neural Information Processing Systems , 34, 2021.

Gary Doran, Krikamol Muandet, Kun Zhang, and Bernhard Schölkopf. A permutation-based kernel conditional independence test. In Proceedings of the 30th Conference on Uncertainty in Artificial Intelligence , pages 132-141, 2014.

Sebastian Engelke and Adrien S Hitz. Graphical models for extremes. Journal of the Royal Statistical Society: Series B (Statistical Methodology) , 82(4):871-932, 2020.

Kenji Fukumizu, Arthur Gretton, Xiaohai Sun, and Bernhard Schölkopf. Kernel measures of conditional dependence. In Advances in Neural Information Processing Systems , 20, 2007.

Weihao Gao, Sewoong Oh, and Pramod Viswanath. Breaking the bandwidth barrier: Geometrical adaptive entropy estimation. In Advances in Neural Information Processing Systems , 29, 2016.

Weihao Gao, Sewoong Oh, and Pramod Viswanath. Demystifying fixed k -nearest neighbor information estimators. In Information Theory (ISIT), 2017 IEEE International Symposium on , pages 1267-1271, 2017.

Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep learning . MIT Press, 2016.

Guido W Imbens and Donald B Rubin. Causal inference in statistics, social, and biomedical sciences . Cambridge University Press, 2015.

Kirthevasan Kandasamy, Akshay Krishnamurthy, Barnabas Poczos, Larry Wasserman, and James m Robins. Nonparametric von mises estimators for entropies, divergences and mutual informations. In Advances in Neural Information Processing Systems , 28, 2015.

Eugene Katsevich and Chiara Sabatti. Multilayer knockoff filter: Controlled variable selection at multiple resolutions. The Annals of Applied Statistics , 13(1):1-33, 2019.

Ilmun Kim, Matey Neykov, Sivaraman Balakrishnan, and Larry Wasserman. Local permutation tests for conditional independence. The Annals of Statistics , 50(6):3388-3414, 2022.

Daphne Koller and Nir Friedman. Probabilistic graphical models: principles and techniques . MIT Press, 2009.

Alexander Kraskov, Harald Stögbauer, and Peter Grassberger. Estimating mutual information. Physical Review E , 69(6):066138, 2004.

Chun Li and Xiaodan Fan. On nonparametric conditional independence tests for continuous variables. Wiley Interdisciplinary Reviews: Computational Statistics , 12(3):e1489, 2020.

Honghao Li, Vincent Cabeli, Nadir Sella, and Hervé Isambert. Constraint-based causal structure learning with consistent separating sets. In Advances in Neural Information Processing Systems , 32, 2019.

Shuai Li, Ziqi Chen, Hongtu Zhu, Christina Dan Wang, and Wang Wen. Nearest-neighbor sampling based conditional independence testing. arXiv preprint arXiv:2304.04183 , 2023.

Dimitris Margaritis. Distribution-free learning of bayesian network structure in continuous domains. In Proceedings of the 20th National Conference on Artificial Intelligence , volume 5, pages 825-830, 2005.

Arnab Mondal, Arnab Bhattacharjee, Sudipto Mukherjee, Himanshu Asnani, Sreeram Kannan, and AP Prathosh. C-mi-gan: Estimation of conditional mutual information using minmax formulation. In Proceedings of the 36th Conference on Uncertainty in Artificial Intelligence , pages 849-858, 2020.

Joris M Mooij and Tom Heskes. Cyclic causal discovery from continuous equilibrium data. In Proceedings of the 29th Conference on Uncertainty in Artificial Intelligence , pages 431-439, 2013.

Sudipto Mukherjee, Himanshu Asnani, and Sreeram Kannan. Ccmi: Classifier based conditional mutual information estimation. In Proceedings of the 36th Conference on Uncertainty in Artificial Intelligence , pages 1083-1093, 2020.

Matey Neykov, Sivaraman Balakrishnan, and Larry Wasserman. Minimax optimal conditional independence testing. The Annals of Statistics , 49(4):2151-2177, 2021.

Ignavier Ng, AmirEmad Ghassami, and Kun Zhang. On the role of sparsity and dag constraints for learning linear dags. In Advances in Neural Information Processing Systems , 33, 2020.

Judea Pearl. Causality . Cambridge University Press, 2009.

Jakob Runge. Conditional independence testing based on a nearest-neighbor estimator of conditional mutual information. In Proceedings of the 21th International Conference on Artificial Intelligence and Statistics , pages 938-947, 2018.

Karen Sachs, Omar Perez, Dana Pe'er, Douglas A Lauffenburger, and Garry P Nolan. Causal protein-signaling networks derived from multiparameter single-cell data. Science , 308(5721):523-529, 2005.

Meyer Scetbon, Laurent Meunier, and Yaniv Romano. An asymptotic test for conditional independence using analytic kernel embeddings. In International Conference on Machine Learning , pages 19328-19346. PMLR, 2022.

Rajat Sen, Ananda Theertha Suresh, Karthikeyan Shanmugam, Alexandros G Dimakis, and Sanjay Shakkottai. Model-powered conditional independence test. In Advances in Neural Information Processing Systems , 30, 2017.

Rajen D Shah and Jonas Peters. The hardness of conditional independence testing and the generalised covariance measure. The Annals of Statistics , 48(3):1514-1538, 2020.

Chengchun Shi, Tianlin Xu, Wicher Bergsma, and Lexin Li. Double generative adversarial networks for conditional independence testing. Journal of Machine Learning Research , 22(285):1-32, 2021.

Peter Spirtes, Clark N Glymour, and Richard Scheines. Causation, prediction, and search . MIT Press, 2000.

Eric V Strobl, Kun Zhang, and Shyam Visweswaran. Approximate kernel-based conditional independence tests for fast non-parametric causal discovery. Journal of Causal Inference , 7(1), 2019.

Xueqin Wang, Wenliang Pan, Wenhao Hu, Yuan Tian, and Heping Zhang. Conditional distance correlation. Journal of the American Statistical Association , 110(512):1726-1734, 2015.

Kun Zhang, Jonas Peters, Dominik Janzing, and Bernhard Schölkopf. Kernel-based conditional independence test and application in causal discovery. In Proceedings of the 27th Conference on Uncertainty in Artificial Intelligence , pages 804-813, 2011.

Shengyu Zhu, Ignavier Ng, and Zhitang Chen. Causal discovery with reinforcement learning. In International Conference on Learning Representations , 2020.

Supplementary Materials for " K -Nearest-Neighbor Local Sampling Based Conditional Independence Testing"

A Theoretical results

A.1 Proof of Lemma 1

Proof. Recall that Z is a random vector taking values in Euclidean space ( R d z , ∥·∥ 2 ) , where d z is the dimension of Z and ∥ · ∥ 2 is Euclidean distance. Z 1 , Z 2 , . . . , Z n are i.i.d. random vectors according to p ( z ) . For a fixed z ∈ R d z , we denote by Z (1) n ( z ) , . . . , Z ( n ) n ( z ) a reordering of Z 1 , Z 2 , . . . , Z n according to the increasing values of ∥ Z i -z ∥ 2 , that is,

Define the set G = { z ∈ R d z | ∀ δ > 0 , P ( { ω : Z ( ω ) ∈ S z ( δ ) } ) > 0 } , where S z ( δ ) = { x ∈ R d z | ∥ x -z ∥ 2 ≤ δ } . For convenience, we omit ω in probability in the following paper. For example, write P ( Z ∈ S z ( δ )) instead of P ( { ω : Z ( ω ) ∈ S z ( δ ) } ) . By definition, for z ∈ G , ∀ δ > 0 , P ( Z ∈ S z ( δ )) > 0 . Let G c be the complement of G . Then, for z ∈ G c , ∃ r z > 0 , s.t. ∀ r < r z , P ( Z ∈ S z ( r )) = 0 . Note that P ( ∥ Z ( k ) n ( z ) -z ∥ 2 > δ ) = P ( Z ( k ) n ( z ) / ∈ S z ( δ )) .

In order to prove ∥ Z ( k ) n ( Z ) -Z ∥ 2 → 0 a.s., it is sufficient to prove ∀ δ > 0 , lim n →∞ P (sup m ≥ n ∥ Z ( k ) m ( Z ) -Z ∥ 2 > δ ) = 0 . We can obtain

First, consider the first term of (11). We have

By setting P ( Z ∈ S z ( δ )) = γ , we have

Consider the j -th term of (12). Let C 1 := γ j /j ! , C 2 := C 1 e j and C 3 := C 2 e j (1 -γ ) -j . By using Stirling's approximation, we have

Thus, there exists C 4 > 0 such that P ( Z ( k ) m ( z ) / ∈ S z ( δ )) ≤ C 4 m k -1 (1 -γ ) m when m is large enough. It holds that

Thus, for z ∈ G and n large enough, ∀ δ > 0 , we have P ( Z ( k ) m ( z ) / ∈ S z ( δ )) = o ( m -2 ) and

which shows lim n →∞ ∑ m ≥ n P ( Z ( k ) m ( z ) / ∈ S z ( δ )) = 0 for z ∈ G . So by Lebesgue dominated convergence theorem, we obtain lim n →∞ ∫ G ∑ m ≥ n P ( Z ( k ) m ( z ) / ∈ S z ( δ )) p ( z ) dz = 0 .

Second, we consider the second term of (11). To prove that P ( Z ∈ G c ) = 0 , we aim to construct a countable open cover of G c and show that the probability of the random vector Z falling into each of these open balls is zero. By the property of G c , for every z ∈ G c , there exists r z > 0 such that for all r < r z , P ( Z ∈ S z ( r )) = 0 . Furthermore, using the separability of Euclidean space and the density of the rational number set, we can approximate z using points from Q d z with Q being the rational number set. Therefore, for every z ∈ G c , there exist x ∈ Q d z ∩ S z ( r z 3 ) and r ∈ Q ∩ ( r z 3 , 2 r z 3 ) , such that z ∈ S x ( r ) ⊆ S z ( r z ) . Because P ( Z ∈ S z ( r z )) = 0 , we conclude that P ( Z ∈ S x ( r )) = 0 . Define

Note that the elements in set F are mutually distinct. By the construction of S x ( r ) , F forms a countable open cover of G c , and the probability of Z falling into each open ball in F is zero. Using the monotonicity and countable additivity properties of probability, we have P ( Z ∈ G c ) ≤ P ( Z ∈ ∪ S x ( r ) ∈F S x ( r )) ≤ ∑ S x ( r ) ∈F P ( Z ∈ S x ( r )) = 0 . Thus, we conclude that P ( Z ∈ G c ) = 0 .

We therefore conclude that, ∀ δ > 0 , lim n →∞ P (sup m ≥ n ∥ Z ( k ) m ( Z ) -Z ∥ 2 > δ ) = 0 . This finish the proof.

A.2 Proof of Theorem 2

Proof. By Pinsker's inequality, we have

Note that I { ξ = 1 } + . . . + I { ξ = k } = 1 . By the definition of ̂ p ( x | Z ) , we obtain

Then, by Taylor's expansion, we have

where a = λZ +(1 -λ ) Z ( l ) n with 0 ≤ λ ≤ 1 . Note that D KL { p ( x | Z ) || p ( x | Z ) } = ∫ p ( x | Z ) log p ( x | Z ) p ( x | Z ) dx = 0 . By Lemma 1 and Assumptions 1 and 2 , we have

and

This means

Note that Z ( l ) n → Z a.s. implies that Z ( l ) n converges to Z in probability. Then ∀ δ > 0 , for ϵ defined in Assumption 1, we have

Thus, D KL { p ( x | Z ) || p ( x | Z ( l ) n ) } = o p (1) .

Because I { ξ = l } ≤ 1 for l = 1 , 2 , ..., k , and k is finite, we obtain

Finally, we conclude that d TV { p ( x | Z ) , ̂ p ( x | Z ) } ≤ √ D KL { p ( x | Z ) , ̂ p ( x | Z ) } / 2 = o p (1) .

A.3 Proof of Theorem 3

To prove Theorem 3, the following two lemmas are needed.

Lemma 5. Let ´ X be drawn from ̂ p ( ·| Z ) , independently of Y . ´ X (1) , . . . , ´ X ( B ) are i.i.d. samples drawn from the k -nearest-neighbor local sampling mechanism based on ( ´ X , Y , Z ) . For any statistic T , the B +1 statistics

are exchangeable conditionally on Y and Z .

Proof. Wehave that the B +1 triples ( ´ X , Y , Z ) , ( ´ X (1) , Y , Z ) , . . . , ( ´ X ( B ) , Y , Z ) are i.i.d. samples drawn from the same mechanism after conditionally on ´ X () , Y and Z , where ´ X () is the order statistic of ´ X . Note that, T is measurable. Thus, T ( ´ X , Y , Z ) , T ( ´ X (1) , Y , Z ) , . . . , T ( ´ X ( B ) , Y , Z ) are i.i.d. after conditionally on ´ X () , Y and Z . Conditionally on ´ X () , Y and Z , denote their cumulative conditional distribution as F ( ·| ´ X () , Y , Z ) . Denote ´ X as ´ X (0) . Then, for any t 0 , . . . , t B ∈ R and

any permutation π = ( π (0) , . . . , π ( B ) ) of the indices { 0 , 1 , . . . , B } , we have

Thus, the desired result follows.

Let d = denotes equality in distribution. We present the following Lemma:

Proof. Denote the joint density functions of ( U , V ) and ( U ′ , V ′ ) by p U , V ( u , v ) and p U ′ , V ′ ( u ′ , v ′ ) , respectively. According to the equivalent definition of the TV distance, we obtain

Now we present the proof of Theorem 3:

Proof. Let ˜ X (1) , . . . , ˜ X ( B ) be i.i.d. drawn from the k -nearest-neighbor local sampling mechanism, see Algorithm 3. Now let ´ X be an additional sample drawn from ̂ p ( ·| Z ) independently of Y . Let ´ X (1) , . . . , ´ X ( B ) be i.i.d. drawn from the k -nearest-neighbor local sampling mechanism after we observe ´ X instead of X . Because ( ´ X (1) , . . . , ´ X ( B ) ) , conditionally on ´ X , Y and Z , is generated from the same mechanism as ( ˜ X (1) , . . . , ˜ X ( B ) ) , conditionally on X , Y and Z , for all x ∈ R n , we have

Then, by applying Lemma 6, we obtain

Define χ B α := { ( x , x (1) , . . . , x ( B ) ) ∣ ∣ ∣ [ 1 + ∑ B b =1 1 { T ( x ( b ) , Y , Z ) ≥ T ( x , Y , Z ) } ] / (1 + B ) ≤ α } , where 1( · ) is the indicator function. Note that in our case, the statistic T is selected to be ̂ CMI. Then,

it follows that

Applying Lemma 5 and the property of rank test, we obtain P (( ´ X , ´ X (1) , . . . , ´ X ( B ) ) ∈ χ B α | Y , Z ) ≤ α . Finally, we have P ( p ≤ α | Y , Z ) ≤ α + d TV { p ( ·| Z ) , ̂ p ( ·| Z ) } .

Because the TV distance is bounded by 1, marginalizing the above inequality over Y and Z and applying Theorem 2 and Lebesgue dominated convergence theorem lead to

A.4 Proof of Theorem 4

Here we present the assumptions given in [34] that ensure the consistency of CMI estimator. We denote the point ( x, y, z ) as ω ∈ R d x × R d y × R d z . Let f ( ω ) = p ( x, y, z ) be the joint density function of ( X,Y,Z ) , g ( ω ) = p ( x, z ) p ( y | z ) be the joint density function of ( X,Y,Z ) under H 0 , and ϕ ( ω ) = ϕ ( x, y, z ) be the joint density of ( X,Y ′ , Z ) produced by Algorithm 1. For a given point ω , γ ( ω ) = P ( l = 1 | ω ) represents the predicted positive label probability generated by a classifier. When the prediction is from a classifier with parameter θ , it is denoted as γ θ ( ω ) . For convenience, the parameter θ in γ is dropped when it is understood from the context. According to Algorithm 2, P ( l = 1) = P ( l = 0) = 1 / 2 . Define the population binary-cross entropy loss over the joint distribution of data and label as BCE ( γ ) = -( E Wl ( l log γ ( W ) + (1 -l ) log(1 -γ ( W ))) .

Assumption (A1) : f ( · ) and ϕ ( · ) admit densities in a compact subset W ⊂ R d x × R d y × R d z .

Assumption (A2) : For some constant α , β > 0 , α ≤ f ( ω ) , ϕ ( ω ) ≤ β , ∀ ω .

Assumption (A3) : Clip predictions such that γ ( ω ) ∈ [ τ, 1 -τ ] , ∀ ω , with 0 < τ ≤ α/ ( α + β ) .

Assumption (A4) : The classifier class C θ is parametrized by θ within a compact domain Θ ⊂ R h . There exists a constant K such that || θ || 2 ≤ K , and the classifier's output is L -Lipschitz with respect to θ .

Assumption (A5) : ∫ p ( z ) 1 -1 /d dz ≤ C 5 , ∀ d ≥ 2 , where C 5 is a constant.

We denote the CMI estimator ̂ CMI based on Algorithm 2 by ̂ D ( n ) KL ( f || ϕ ) . The true CMI of ( X,Y,Z ) is CMI = I ( X ; Y | Z ) = D KL ( f || g ) . Then we have the following Lemma:

Lemma 7. Under Assumptions 1 and 2 and (A1)-(A5), we have ̂ D ( n ) KL ( f || ϕ ) P → D KL ( f || g ) .

Proof. By the definition of convergence in probability, it is sufficient to prove ∀ δ > 0 , ∀ η > 0 , ∃ N , when n > N , P ( | ̂ D ( n ) KL ( f || ϕ ) -D KL ( f || g ) | > δ ) < η .

Note that

Applying Theorem 1 in [34], we have ̂ D ( n ) KL ( f || ϕ ) -D KL ( f || ϕ ) P → 0 , which means for δ/ 2 > 0 and η > 0 , ∃ N 1 , when n > N 1 , P ( | ̂ D ( n ) KL ( f || ϕ ) -D KL ( f || ϕ ) | > δ/ 2) < η .

Now consider the term D KL ( f || ϕ ) -D KL ( f || g ) . Let γ ′ be the point-wise minimizer of binary-cross entropy loss based on f ( ω ) and ϕ ( ω ) , and γ ′′ be the point-wise minimizer of binary-cross entropy

loss based on f ( ω ) and g ( ω ) . Applying Lemma 3 in [34], we have γ ′ ( ω ) / { 1 -γ ′ ( ω ) } = f ( ω ) /ϕ ( ω ) and γ ′′ ( ω ) / { 1 -γ ′′ ( ω ) } = f ( ω ) /g ( ω ) . Then, by the definition of KL divergence, it follows that

The second inequality follows from Lagrange's mean value theorem and Assumption (A3).

Applying Theorem 1 in [41], ∀ ϵ 1 ≤ ϵ with ϵ being defined in Assumption 1, we have d TV ( ϕ, g ) ≤ b ( n ) , where

Here, β is defined in Assumption 1, C 5 is defined in Assumption (A5), d z is the dimension of Z , Γ( · ) is the gamma function, γ d z is the volume of the unit radius l 2 ball in R d z , c d z is defined in Assumption 2, and ∀ δ > 0 , G ( δ ) = P ( p ( Z ) ≤ δ ) .

Because ϵ 1 can be arbitrary small, we conclude that lim n →∞ b ( n ) = 0 . So we arrive at lim n →∞ | D KL ( f || ϕ ) -D KL ( f || g ) | = 0 , which means for δ/ 2 > 0 , ∃ N 2 , when n > N 2 , | D KL ( f || ϕ ) -D KL ( f || g ) | < δ/ 2 .

Then for δ > 0 and η > 0 , take N = max( N 1 , N 2 ) , when n > N ,

holds. This finish the proof.

We therefore conclude that ̂ CMI is a consistent estimator of CMI. When considering ̂ CMI ( b ) based on the sample ( ˜ X ( b ) , Y , Z ) ( b = 1 , . . . , B ) drawn from the k -nearest-neighbor local sampling mechanism as depicted in Algorithm 3, we can state: Under Assumptions 1, 2, (A1)-(A2) with f ( ω ) and ϕ ( ω ) replaced by densities of the distribution of ( ˜ X,Y,Z ) and the corresponding 1-NN distribution, respectively, and Assumptions (A3)-(A5), ∀ b = 1 , . . . , B, ̂ CMI ( b ) is a consistent estimator of CMI ( b ) , where CMI ( b ) = I ( ˜ X ( b ) ; Y | Z ) . Now let's present the proof of Theorem 4.

Proof. Write P ( ·| H 1 ) as P H 1 ( · ) . By Markov inequality, it follows that

Because ˜ X (1) ⊥ ⊥ Y | Z , CMI (1) = I ( ˜ X (1) ; Y | Z ) = 0 . Then, ∀ δ > 0 ,

Next, we have

Thus, we conclude that

Under H 1 , CMI > 0 . Take δ = CMI / 4 > 0 , we obtain

B Additional Empirical Results

B.1 The choice of the neighbor order k

To investigate the impact of the parameter k on our proposed approach, we employ a linear uniform model. To accomplish this, we generate synthetic data in the following manner:

where ϵ x and ϵ y are generated independently from the uniform distribution over the interval [ -1 , 1] . The parameter α is randomly generated within the range of [0 , 2] . As is shown in Figure 3, our method achieves effective control of type I error and exhibits the highest power under H 1 across all dimensions when k = 7 . Therefore, we consistently set k = 7 in all experiments.

B.2 Empirical results for Scenario (13)

We demonstrate the effectiveness of our approach and compare it with alternative methods in Scenario (13). The results are shown in Figure 4, which pertains to high-dimensional Z , and Figure 5, which focuses on low-dimensional Z . The results consistently demonstrate that our test achieves favorable performance in terms of the type I error and power under H 1 . Although LPCIT, CMIknn, and NNSCIT effectively control the type I error, they exhibit noticeably lower power compared to our method, when the dimension exceeds 60, often by a substantial margin. Furthermore, KCIT, GCIT, and CCIT all yield high power under H 1 , but they either always or sometimes suffer from inflated type I errors.

5

30

50

80

100

Dimension of Z

0.00

0.02

0.04

0.06

0.08

0.10

Type I error

Uniform, samples=500

k=2

k=5

k=6

k=7

k=10

k=100

5

30

50

80

100

Dimension of Z

0.85

0.88

0.91

0.94

0.97

1.00

Power

Uniform, samples=500

30

40

50

60

70

80

90

100

Dimension of Z

0.00

0.05

0.10

0.15

0.20

0.25

0.30

0.35

Type I error

Uniform, samples=500

GCIT

CCIT

KCIT

CMIknn

LPCIT

NNSCIT

Ours

300 500

1000

1500

2000

Sample size

0.00

0.05

0.10

0.15

0.20

0.25

0.30

0.35

Type I error

Uniform, dz=80

30

40

50

60

70

80

90

100

Dimension of Z

0.4

0.5

0.6

0.7

0.8

0.9

1.0

Power

Uniform, samples=500

300 500

1000

1500

2000

Sample size

0.4

0.5

0.6

0.7

0.8

0.9

1.0

Power

Uniform, dz=80

B.3 Additional empirical results for Scenario I

In Figure 6, we present the type I error and power under H 1 in low dimensions of Z ranging from 5 to 30 for Scenario I (Eq. (9)) with Gaussian or Laplace noises. It can be observed that our test and LPCIT consistently achieve good and stable performance in terms of type I error and power under H 1 , when the dimensionality of Z is lower than 30 . On the other hand, GCIT, CCIT, and KCIT exhibit high power under H 1 but fail to control the type I error. NNSCIT and CMIknn demonstrate relatively good control of type I errors but lack sufficient power under H 1 .

B.4 Computational efficiency analysis

Figure 7 shows the timing performance of all methods for a single test under Scenario I with Laplace noises. Our test is found to be highly computationally efficient even when dealing with large

5

10

15

20

25

30

Dimension of Z

0.0

0.1

0.2

0.3

0.4

Type I error

Uniform, samples=500

GCIT

CCIT

KCIT

CMIknn

LPCIT

NNSCIT

Ours

5

10

15

20

25

30

Dimension of Z

0.8

0.9

1.0

Power

Uniform, samples=500

5

10

15

20

25

30

Dimension of Z

0.0

0.1

0.2

0.3

0.4

0.5

0.6

Type I error

Gauss, samples=500

GCIT

CCIT

KCIT

CMIknn

LPCIT

NNSCIT

Ours

5

10

15

20

25

30

Dimension of Z

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

Type I error

Laplace, samples=500

5

10

15

20

25

30

Dimension of Z

0.5

0.6

0.7

0.8

0.9

1.0

Power

Gauss, samples=500

5

10

15

20

25

30

Dimension of Z

0.4

0.5

0.6

0.7

0.8

0.9

1.0

Power

Laplace, samples=500

sample sizes and high-dimensional conditioning sets. In contrast, CMIknn and CCIT for sample sizes exceeding 1000, and LPCIT for dimension of Z higher than 50 are impractical due to their prohibitively long running time.

B.5 The detailed experimental setup for Scenario III

For the chain structure Y → Z → X , we generate synthetic data as follows:

5 10 15 20 25 30

40

50

60

70

80

90

100

Dimension of Z

0

30

60

90

120

150

180

210

240

270

300

Run time

samples=500

GCIT

CCIT

KCIT

CMIknn

LPCIT

NNSCIT

Ours

300 500

1000

1500

2000

Sample size

0

30

60

90

120

150

180

210

240

270

300

330

Run time

dz=80

where a and b are both d z -dimensional, the entries of a and b are both randomly and uniformly sampled from [0 , 0 . 3] , ϵ 1 is generated from a d z -dimensional standard multivariate Gaussian distribution, and ϵ 2 is sampled from a standard Gaussian distribution.

C Real Data Analysis

In order to showcase the superior performance of our test, we conduct a comparative evaluation against other state-of-the-art (SOTA) CI tests using real datasets. We assess the effectiveness of our method along with six SOTA approaches on two specific datasets: the ABALONE dataset [9] and the Flow-Cytometry dataset [39].

C.1 Real ABALONE dataset

The ABALONE dataset [9] comprises measurements obtained from a study conducted to predict the age of abalones based on their physical characteristics. The dataset is publicly available at the UCI Machine Learning Repository and can be downloaded from https://archive.ics.uci.edu/ml/datasets/abalone. In our evaluation, we consider the graph structure recovered by [31] as the ground truth, as depicted in Figure 4 of their paper. This graph represents the causal relationships among the 8 variables in the dataset. We specifically select 35 CI relations and 35 non-CI relations from this graph. The philosophy used is that a node X is independent of all other nodes Y in the graph when conditioned on its parents, children, and parents of children [26, 41]. Additionally, if there exists a direct edge between node X and node Y in the graph, they are never conditionally independent given any other set of variables. As a result, the conditioning set Z can be arbitrarily selected from the remaining nodes. The dataset consists of 4177 samples, and d z varies from 1 to 6.

In order to evaluate the performance of various tests, we utilize precision, recall, and F-score as evaluation metrics. Precision is calculated as TP/(TP+FP), where TP represents the number of true CI instances correctly identified, and FP represents the number of non-CI instances incorrectly identified as CI. Recall is calculated as TP/(TP+FN), where FN represents the number of CI instances not identified. The F-score is then computed as the harmonic mean of precision and recall, given by 2 × precision × recall / (precision + recall) [29]. TN represents the number of correctly identified true

non-CI instances. Table 1 presents the results for all methods. It should be noted that we do not record the results for GCIT as it does not correctly identify any CI relations. Our approach successfully identifies 31 CI relations and 32 non-CI relations, achieving the highest F-score among the testing methods, while maintaining high precision and recall.

C.2 Real Flow-Cytometry dataset

The Flow-Cytometry dataset is a widely used benchmark in the field of causal structure learning [36, 48]. This dataset captures the expression levels of proteins and phospholipids in human cells [39]. The data can be obtained from the website https://www.science.org/doi/10.1126/science.1105809. In our evaluation, we consider the consensus graph proposed in [33] as the ground truth, which has also been adopted by [41] for verifying CI relations. Figure 5(a) in [33] illustrates the causal relationships among the 11 proteins in the dataset. Following the philosophy outlined in Section C.1, we select 50 CI relations and 40 non-CI relations from this graph. The number of samples is 1755 and d z varies from 1 to 9.

Table 2 presents the results for all tests. Our method outperforms other approaches by correctly identifying 47 CI relations and achieving the highest recall and F-score.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper argues that conditional independence (CI) testing becomes challenging because of high-dimensional conditioning variables and limited data samples. To address these issues, the authors propose a testing approach incorporating a class-based conditional mutual information (CMI) and a $k$-nearest-neighbor local sampling strategy. Theoretical analysis demonstrates its asymptotic control of type I error and consistency against all alternative hypotheses. Extensive empirical results on synthetic and real data show that the proposed method achieves computational efficiency, decent performance under different scenarios, and robustness towards heavy-tailed data.\n\nSTRENGTHS:\nThis idea is intrigued by two significant challenges of CI testing, i.e., high-dimensional conditioning variables and limited data samples. \nThe authors do a good job of motivating this work, and the proposed test approach is simple but effective.\nExtensive empirical results show that the proposed method achieves a better trade-off between type I error rate and testing power.\nIn addition, the paper is very well written and easy to understand.\n\nWEAKNESSES:\nMy main doubts/concerns regarding the paper are the following:\n- Line 104-105 claim that the $k$-nearest-neighbor local sampling strategy is an alternative to the binning strategy. However, this paper does not provide a comparison between the proposed testing approach and the methods employing a binning strategy [1,2].\n- In Figure 3, the results on synthetic data show that the performance is sensitive to the selection of $k$. Therefore, it is unreasonable to apply $k=7$ in all experiments, especially in real data.\n\nMinor:\n- Empirical analysis shows that the proposed method achieves a better trade-off between type I error rate and testing power. It is hard to say that the proposed method outperforms existing SOTA methods.\n\n[1] I. Kim, et al., \"Local Permutation Tests for Conditional Independence\", The Annals of Statistic 2022.\n\n[2] D. Margaritis, \"Distribution-free Learning of Bayesian Network Structure in Continuous Domains\", AAAI 2005.\n\nQUESTIONS:\n1. Can the authors add more discussions on the difference between the $k$-nearest-neighbor local sampling strategy and the binning strategy? What are the advantages of the $k$-nearest-neighbor local sampling strategy?\n2. Can the authors provide the optimal $k$ on real data?\n\nLIMITATIONS:\nYes, the authors have addressed the societal impact of their work.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors present a method for conditional independence (CI) testing based on a conditional mutual information (CMI) estimator. \nThis CMI estimator is based on a classifier over 1-NN samples. From the CMI estimates, the authors present a hypothesis testing procedure for CI based on multiple iterations over estimates of CMI on K-NN samples.\n\nThe authors present a theoretical analysis of their approach and the empirical results show that their approach has low Type 1 error and high power. The empirical section shows also that their approach remains consistent as the number of dimensions increase for the conditioning variables.\n\nSTRENGTHS:\nThe paper is well written and technically sound. The contribution builds on previous approaches but is an original method for the important task of CI. \nThe method is very promising given the theoretical and empirical results providing balanced detection of TP and TN links in real-world datasets as shown by the top F-scores. \nThe use of a single classifier for the CMI estimation is advantageous as the CI test is usually executed multiple times for graph detection.\n\nWEAKNESSES:\nI find the paper really compelling and useful in many other ML tasks. However, I believe the empirical evaluation could have gone a step further.\nIn particular, I believe a distillation procedure would be helpful to show which part of your approach contributes to the significant results. More precisely, it would be great to see how good your CMI estimator is. A comparison against two-classifier approaches and even a synthetic dataset where you have access to the ground truth conditional distributions would be very useful.\nThis could show how tight the bound is given your estimator. Furthermore, it would be nice to see the performance of your approach given different classifiers, e.g., DNNs or Random forest methods as mentioned in your paper.\n\nIn the presentation, I would make the figures larger.\n\nQUESTIONS:\nI might have missed it but it wasn't clear to me, which classifier you used in your experiments?\n\nWhat do you mean by $V_0^{test}$ in line 7 of alg. 2? is that any instance over the union of test sets in f and g?\n\nHow long does your approach take compared to the others? After all, a large number of B repetitions could significantly add up.\n\nLIMITATIONS:\nI don't expect any potential negative societal impact from this work", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}

comments: SUMMARY:
In this paper, the authors propose a novel CMI estimator with a classifier-based approach. The estimated CMI is then used for conditional independence testing. This approach achieves asymptotic type I & II control and does not require prior assumptions on the distribution, types of correlations, or knowledge of the null distribution. Empirical results show promising performance.

STRENGTHS:
1. This paper is well-written. 
2. The proposed CMI is very natural for testing CI. The algorithm is easy to implement given any classifier.  
3.  The proposed method is backed up by theory (Theorem 3 & 4).

WEAKNESSES:
1.  As authors have pointed out, prior works for estimating MI (e.g. Suzuki et al., 2008 and [3]) using density ratio do exist, and a conditional version of such an estimator is not exactly a huge leap forward (but no small feat either)

2. The control of type I & II is asymptotic, and it is unclear how useful they are when the sample size is small. The claim of Theorem 3 and 4 is unsurprising (due to the consistency of KNN and smoothness of conditional distribution). If the proof is non-trivial, please point out.  

Suzuki et al., 2008 Approximating Mutual Information by Maximum Likelihood Density Ratio Estimation

QUESTIONS:
1. Why is the CMI estimated using classifiers rather than directly optimising (4) like f-GAN? Why not directly use the maximum of (4) to approximate the CMI? This way, you would not need to split the dataset into training and testing sets right?

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a novel method to test conditional independence (CI) by estimating conditional mutual information (CMI). A kNN local sampling strategy is employed to empirically sample from an unknown conditional distribution. The method is proved to have type-I error controlled, and power converging to 1.\n\nThe main contribution of the novel method consists of three points. First, it addresses challenges posed by high-dimensional conditioning variables and limited data samples. Second, it operates without assumptions about distribution forms or feature dependencies. Third, it avoids dataset splitting. Empirically, it is computationally efficient.\n\nSTRENGTHS:\n(1) Good originality. Using kNN as a sampling method to circumvent prior knowledge on conditional distribution $p_{X|Z}(x|z)$ is a novel idea.\n\n(2) The algorithm is a high-quality one, in terms of enhanced computational efficiency.\n\n(3) The simulation settings is a nice touch to showcase that the novel method operates without assumptions about distribution forms.\n\nWEAKNESSES:\n(1) The paper claims an improvement upon the current literature under high-dimensional settings. But this contribution is not reflected in either theoretical results or simulations. Specifically, in simulation the sample size is 500, while dimension of $Z$ is at most 100. The results are still in low-dimensional regime.\n\nQUESTIONS:\n(1) The paper claims novelty in avoiding sample splitting. It seems to me that Algorithm 1 & 2 are still splitting data into $V_1$ and $V_2$. Does this splitting not weaken test power?\n\n(2) More runtime analysis would be appreciated. The paper's claim on enhanced computational complexity sounds legit, but requires more evidence to support.\n\n(3) How does $\\beta$ and $c_{d_Z}$ in Assumptions 1 and 2 affect the convergence rates in Theorems 2 and 4?\n\n(4) Line 180: \"Furthermore, it intuitively suggests that our test can achieve high power under $H_1$\". This claim was based on $\\hat{\\text{CMI}}$ should be positive whp. This quantity, however, is replaced by $\\tilde{\\text{CMI}}^{(b)}$ in practice. It the latter still positive whp?\n\nLIMITATIONS:\nThe paper claims that their novel method operates without assumptions about distribution forms or feature dependencies. However, the two assumptions on smoothness do not sound mild.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-26"}