Double Generative Adversarial Networks for Conditional Independence Testing

1 INTRODUCTION . Conditional independence ( CI ) is a fundamental concept in statistics and machine learning . Testing conditional independence is a key building block and plays a central role in a wide variety of statistical learning problems , for instance , causal inference ( Pearl , 2009 ) , graphical models ( Koller & Friedman , 2009 ) , dimension reduction ( Li , 2018 ) , among others . In this article , we aim at testing whether two random variables X and Y are conditionally independent given a set of confounding variables Z . That is , we test the hypotheses : H0 : X ⊥ Y | Z versus H1 : X 6⊥ Y | Z , ( 1 ) given the observed data of n i.i.d . copies { ( Xi , Yi , Zi ) } 1≤i≤n of ( X , Y , Z ) . For our problem , X , Y and Z can all be multivariate . However , the main challenge arises when the confounding set of variables Z is high-dimensional . As such , we primarily focus on the scenario with a univariate X and Y , and a multivariateZ . Meanwhile , our proposed method can be extended to the multivariateX and Y scenario as well . Another challenge is the limited sample size compared to the dimensionality ofZ . As a result , many existing tests are ineffective , with either an inflated type-I error , or not having enough power to detect the alternatives . See Section 2 for a detailed review . We propose a double generative adversarial networks ( GANs , Goodfellow et al. , 2014 ) -based inference procedure for the CI testing problem ( 1 ) . Our proposal involves two key components , a double GANs framework to learn two generators that approximate the conditional distribution of X given Z and Y given Z , and a maximum of generalized covariance measures of multiple combinations of the transformation functions of X and Y . We first establish that our test statistic is doubly-robust , which offers additional protections against potential misspecification of the conditional distributions ( see Theorems 1 and 2 ) . Second , we show the resulting test achieves a valid control of the type-I error asymptotically , and more importantly , under the conditions that are much weaker and practically more feasible ( see Theorem 3 ) . Finally , we prove the power of our test approaches one asymptotically ( see Theorem 4 ) , and demonstrate it is more powerful than the competing tests empirically . 2 RELATED WORKS . There has been a growing literature on conditional independence testing in recent years ; see ( Li & Fan , 2019 ) for a review . Broadly speaking , the existing testing methods can be cast into four main categories , the metric-based tests , e.g. , ( Su & White , 2007 ; 2014 ; Wang et al. , 2015 ) , the conditional randomization-based tests ( Candes et al. , 2018 ; Bellot & van der Schaar , 2019 ) , the kernel-based tests ( Fukumizu et al. , 2008 ; Zhang et al. , 2011 ) , and the regression-based tests ( Hoyer et al. , 2009 ; Zhang et al. , 2018 ; Shah & Peters , 2018 ) . There are other types of tests , e.g. , Bergsma ( 2004 ) ; Doran et al . ( 2014 ) ; Sen et al . ( 2017 ; 2018 ) ; Berrett et al . ( 2019 ) , to mention a few . The metric-based tests typically employ some kernel smoothers to estimate the conditional characteristic function or the distribution function of Y given X and Z. Kernel smoothers , however , are known to suffer from the curse of dimensionality , and as such , these tests are not suitable when the dimension of Z is high . The conditional randomization-based tests require the knowledge of the conditional distribution of X|Z ( Candes et al. , 2018 ) . If unknown , the type-I error rates of these tests rely critically on the quality of the approximation of this conditional distribution . Kernel-based test is built upon the notion of maximum mean discrepancy ( MMD , Gretton et al. , 2012 ) , and could have inflated type-I errors . The regression-based tests have valid type-I error control , but may suffer from inadequate power . Next , we discuss in detail the conditional randomization-based tests , in particular , the work of Bellot & van der Schaar ( 2019 ) , the regression-based and the MMD-based tests , since our proposal is closely related to them . 2.1 CONDITIONAL RANDOMIZATION-BASED TESTS . The family of conditional randomization-based tests is built upon the following basis . If the conditional distribution PX|Z ofX given Z is known , then one can independently drawX ( 1 ) i ∼ PX|Z=Zi for i = 1 , . . . , n , and these samples are independent of the observed samples Xi ’ s and Yi ’ s . Write X = ( X1 , . . . , Xn ) > , X ( 1 ) = ( X ( 1 ) 1 , . . . , X ( 1 ) n ) > , Y = ( Y1 , . . . , Yn ) > , and Z = ( Z1 , . . . , Zn ) > . Here we use boldface letters to denote data matrices that consist of n samples . The joint distributions of ( X , Y , Z ) and ( X ( 1 ) , Y , Z ) are the same under H0 . Any large difference between the two distributions can be interpreted as the evidence against H0 . Therefore , one can repeat the process M times , and generate X ( m ) i ∼ PX|Z=Zi , i = 1 , . . . , n , m = 1 , . . . , M . Write X ( m ) = ( X ( m ) 1 , . . . , X ( m ) n ) > . Then , for any given test statistic ρ = ρ ( X , Y , Z ) , its associated p-value is p = [ 1 + ∑M m=1 I { ρ ( X ( m ) , Y , Z ) ≥ ρ ( X , Y , Z ) } ] / ( 1 + M ) , where I ( · ) is the indicator function . Since the triplets ( X , Y , Z ) , ( X ( 1 ) , Y , Z ) , . . . , ( X ( M ) , Y , Z ) are exchangeable underH0 , the p-value is valid , and it satisfies that Pr ( p ≤ α|H0 ) ≤ α+ o ( 1 ) for any 0 < α < 1 . In practice , however , PX|Z is rarely known , and Bellot & van der Schaar ( 2019 ) proposed to approximate it using GANs . Specifically , they learned a generator GX ( · , · ) from the observed data , then took Zi and a noise variable v ( m ) i , X as input to obtain a sample X̃ ( m ) i , which minimizes the divergence between the distributions of ( Xi , Zi ) and ( X̃ ( m ) i , Zi ) . The p-value is then computed by replacing X ( m ) by X̃ ( m ) = ( X̃ ( m ) 1 , . . . , X̃ ( m ) n ) > . They called this test GCIT , short for generative conditional independence test . By Theorem 1 of Bellot & van der Schaar ( 2019 ) , the excess type-I error of this test is upper bounded by Pr ( p ≤ α|H0 ) − α ≤ EdTV ( P̃X|Z , PX|Z ) = E sup A |Pr ( X ∈ A|Z ) − Pr ( X̃ ( m ) ∈ A|Z ) | ≡ D , ( 2 ) where dTV is the total variation norm between two probability distributions , the supremum is taken over all measurable sets , and the expectations in ( 2 ) are taken with respect to Z . By definition , the quantity D on the right-hand-side of ( 2 ) measures the quality of the conditional distribution approximation . Bellot & van der Schaar ( 2019 ) argued that this error term is negligible due to the capacity of deep neural nets in estimating conditional distributions . To the contrary , we find this approximation error is usually not negligible , and consequently , it may inflate the type-I error and invalidate the test . We consider a simple example to further elaborate this . Example 1 . Suppose X is one-dimensional , and follows a simple linear regression model , X = Z > β0 + ε , where the error ε is independent of Z and ε ∼ N ( 0 , σ20 ) for some σ20 > 0 . Suppose we know a priori that the linear regression model holds . We thus estimate β0 by ordinary least squares , and denote the resulting estimator by β̂ . For simplicity , suppose σ20 is known too . For this simple example , we have the following result regarding the approximation error term D. Proposition 1 Suppose the linear regression model holds . The derived distribution P̃X|Z is N ( Zβ̂ , σ20In ) , where In is the n× n identity matrix . Then D is not o ( 1 ) . To facilitate the understanding of the convergence behavior of D , we sketch a few lines of the proof of Proposition 1 . A detailed proof is given in Appendix F.1 . Let P̃X|Z=Zi denote the conditional distribution of X̃ ( m ) i given Zi , which is N ( Z > i β̂ , σ 2 0 ) in this example . If D = o ( 1 ) , then , D̃ ≡ n1/2 √ Ed2TV ( P̃X|Z=Zi , PX|Z=Zi ) = o ( 1 ) . ( 3 ) In other words , the validity of GCIT requires the root mean squared total variation distance in ( 3 ) to converge at a faster rate than n−1/2 . However , this rate can not be achieved in general . In our simple Example 1 , we have D̃ ≥ c for some universal constant c > 0 . Consequently , D in ( 2 ) is not o ( 1 ) . Proposition 1 shows that , even if we know a priori that the linear model holds , D is not to decay to zero as n grows to infinity . In practice , we do not have such prior model information . Then it would be even more difficult to estimate the conditional distribution PX|Z . Therefore , using GANs to approximate PX|Z does not guarantee a negligible approximation error , nor the validity of the test . 2.2 REGRESSION-BASED TESTS . The family of regression-based tests is built upon a key quantity , the generalized covariance measure , GCM ( X , Y ) = 1 n n∑ i=1 { Xi − Ê ( Xi|Zi ) } { Yi − Ê ( Yi|Zi ) } , where Ê ( X|Z ) and Ê ( Y |Z ) are the predicted condition mean E ( X|Z ) and E ( Y |Z ) , respectively , by any supervised learner . When the prediction errors of Ê ( X|Z ) and Ê ( Y |Z ) satisfy certain convergence rates , Shah & Peters ( 2018 ) proved that GCM is asymptotically normal . Under H0 , the asymptotic mean of GCM is zero , and its asymptotic standard deviation can be consistently estimated by some standard error estimator , denoted by ŝ ( GCM ) . Therefore , at level α , we reject H0 , if |GCM|/ŝ ( GCM ) exceeds the upper α/2th quantile of a standard normal distribution . Such a test is valid . However , it may not have sufficient power to detect H1 . This is because the asymptotic mean of GCM equals GCM∗ ( X , Y ) = E { X−E ( X|Z ) } { Y −E ( Y |Z ) } . The regressionbased tests require |GCM∗| to be nonzero under H1 to have power . However , there is no guarantee of this requirement . We again consider a simple example to elaborate . Example 2 . Suppose X∗ , Y and Z are independent random variables . Besides , X∗ has mean zero , and X = X∗g ( Y ) for some function g. For this example , we have E ( X|Z ) = E ( X ) , since both X∗ and Y are independent of Z , and so is X . Besides , E ( X ) = E ( X∗ ) E { g ( Y ) } = 0 , since X∗ is independent of Y and E ( X∗ ) = 0 . As such , GCM∗ ( X , Y ) = E { X − E ( X ) } { Y − E ( Y |Z ) } = 0 for any function g. On the other hand , X and Y are conditionally dependent given Z , as long as g is not a constant function . Therefore , for this example , the regression-based tests would fail to discriminate betweenH0 andH1 .

Summary: This paper considers the problem of conditional independence testing, especially when the variables are high-dimensional. The authors proposed a double GAN based algorithm. Two GANs are designed to learn the conditional probability distributions P_{X|Z} and P_{Y|Z}, then used to generate samples to compute the test statistic. It is proved that the error of the test statistic is O_p(n^{-2k} \log n) when the total variation error of the GANS is O(n^{-k}). So to ensure that the test statistic converges, only o(\log^{-1/2} n) rate is required for the total variation error of the GANs.