Double Generative Adversarial Networks for Conditional Independence Testing
Authors: Chengchun Shi
Tianlin Xu
Wicher Pieter Bergsma
Lexin Li
Venue:  (2021)
Keywords: conditional independence testing
generative adversarial networks
high dimensionality
statistical inference

Abstract: In this article, we consider the problem of high-dimensional conditional independence testing, which is a key building block in statistics and machine learning. We propose a double generative adversarial networks (GAN)-based inference procedure. We first introduce a double GANs framework to learn two generators, and integrate the two generators to construct a doubly-robust test statistic. We next consider multiple generalized covariance measures, and take their maximum as our test statistic. Finally, we obtain the empirical distribution of our test statistic through multiplier bootstrap. We show that our test controls type-I error, while the
power approaches one asymptotically. More importantly, these theoretical guarantees are obtained under much weaker and practically more feasible conditions compared to existing tests. We demonstrate the efficacy of our test through both synthetic and real datasets.

Under review as a conference paper at ICLR 2021
DOUBLE GENERATIVE ADVERSARIAL NETWORKS FOR CONDITIONAL INDEPENDENCE TESTING
Anonymous authors Paper under double-blind review
ABSTRACT
In this article, we consider the problem of high- sigh-dimensional conditional indepen- dence testing, which is key building block statistics and machine learning. We propose a double generative adversarial networks (GANs)-based inference proce- dure. We first introduce a double GANs framework to learn two generators, and integrate the two generators to construct doubly-robust test statistic. We next consider multiple generalized covariance measures, and take their maximum as test statistic. Finally, obtain the empirical distribution of test statistic through multiplier bootstrap. We show that our test controls type-I error, while the power approaches one asymptotically. More importantly, these theoretical guar- antees are obtained under much weaker and practically more feasible conditions compared to existing tests. We demonstrate the efficacy of our test through both synthetic and real data examples.
INTRODUCTION
Conditional independence (CI) is a fundamental concept in statistics and machine learning. Testing conditional independence key building block and plays central role in wide variety sta- tistical learning problems, for instance, causal inference (Pearl 2009). graphical models (Koller & FFieddaan] 2009), dimension reduction (Li) 2018). among others. In this article, we aim testing whether two random variables X and are conditionally independent given set of confounding variables Z. That is, we test the hypotheses:
Ho YYIZ
versus
H4
X
L Z,
given the observed data of n i.i.d. copies {(X,, hsssn of (X, Z). For our problem, X, Y and Z can all be multivariate. However, the main challenge arises when the confounding set of variables Z high-dimensional. As such, we primarily focus on the scenario with a univariate X and Y. and multivariate Z. Meanwhile, our proposed method can be extended to the multivariate X and Y scenario as well. Another challenge is the limited sample size compared to the dimensionality of Z. As result, many existing tests are ineffective, with either an inflated type-I error, or not having enough power to detect the alternatives. See Section for a detailed review.
We propose double generative adversarial networks (GANs, Goodfellow et al. (014)-based infer- ence procedure for the CI testing problem (T). Our proposal involves two key components, a double GANs framework to learn Z and Y given Z, and a maximum of generalized covariance measures of multiple combinations of the transformation functions of X and Y. We first establish that our test statistic is doubly-robust, which offers additional protections against potential misspecification of the conditional distributions (see Theorems
generators that approximate the conditional distribution of X given
[ and 2). Second, we show the resulting test achieves a valid control of the type-I er- ror asymptotically. and more importantly, under the conditions that are much weaker and practically more feasible (see Theorem 3) Finally, we prove the power of our test approaches one asymptoti- cally (see Theorem 4), and demonstrate it is more powerful than the competing tests empirically.
RELATED WORKS
There has been a growing literature on conditional independence testing in recent years; see [Li & Fan 2019) for review. Broadly speaking, the existing testing methods can be cast into four main categories, the metric- based tests, e.g., (Su & White 2007 | Wang et al. (015). the conditional randonization-basel tests (Candes et al. 2018 Bellot & der Schaar 2019), the kernel-based

---

Under review as a conference paper at ICLR 2021
tests (Fukumizu et al. 2008 Zhang et al.
2011). and the regression- based tests (Hoyer et al. 2009
Zhang et al. 2018; 8 Shah & Peters 2018). There are other types of tests, e.g., Bergsma (004)) ; Doran
et al. (2014); Sen al. (211 201
Berrett et al. (2099), to mention few.
The metric-based tests typically employ some kernel smoothers to estimate the conditional charac- teristic function or the distribution function of Y given X and Z. Kernel smoothers, however, known to suffer from the curse of dimensionality,, and as such, these tests are not suitable when the dimension of Z is high. The conditional randomization-basee tests require the knowledge of the conditional distribution of X|Z (Candes et al. 2018). If unknown, the type-I error rates of these tests rely critically on the quality the approximation of this conditional distribution. Kernel-based test is built upon the notion of maximum mean discrepancy (MMD, Gretton et al. (2012). and could have inflated type-I errors. The regression- -based tests have valid type-I error control, but may suffer from inadequate power. Next, we discuss in detail the conditional randomization-ban based tests, in particular, the work of Bellot & van der Schaar (2019), the regression- baasdd and the MMD-based tests, since our proposal is closely related to them.
2.1 CONDITIONAL RANDOMIZATION BASED TESTS
The family of conditional randomization- tests is built upon the following basis. If the condi- tional distribution Px 2x|2 of X given Z is known, then one can independently draw x(") Pxiz=z, for i = i.....,,, n, and these samples are independent of the observed samples X,'s and Y/'s. Write X = (X (X1,..., X,,,, (1) Here we use boldface letters to denote data matrices that consist of n samples. The joint distri- butions of (X, Y. Z) and (x("), Y, Z) are the same under Ho. Any large difference between the two distributions can be interpreted the evidence against Ho. Therefore, one can repeat the process M times, and generate X("") ~ Pxiz=z, i = I,..., n, m = I....,M. Write ((m) (x(mm) (m) ) Then, for any given test statistic p p(X, Y. Y.2 Z), its associated m=
(x(") ...... X(")),, "
Y (Yi..... Y,,", and Z = (Zi,..., ZZ,)..
p-value is p = [1+ [1+E T(p(X (m)) Y. Z) 2 p X, Y, (11]/(1 + M). where T(.) is the indi- cator function. Since the triplets (X,Y, 2), (X (1) Y. Z).
(X() Y, Z) are exchangeable under Ho, the p-value valid, and it satisfies that Pr(p S a Ho) S a + (11) for any 0 < a < 1.
In practice, however, Px z is rarely known, and Bellot & van der Schaar (2019) proposed to approx- imate it using GANs. Specifically, they learned generator G (., from the observed data, then took Z, and a noise variable (.(m) as input obtain a sample X(!)), which minimizes the divergence
between the distributions of (X;, Z,) and (X((m) 7,). The p-value is then computed by replacing X by X (m) = (X7m)) ..... {(m) T. They called this test GCIT, short for generative conditional independence test. By Theorem of Bellot & van der Schaar 2019)) the excess type-I error of this test is upper bounded by
Pr (p I a: (:Ho) a S Edrv z: Px(z) = Esup [Pr(X E A Pr( (m) E EA|Z)] = D, (2)
where dry is the total variation norm between two probability distributions, the supremum taken over all measurable sets, and the expectations in (2) are taken with respect to Z.
By definition, the quantity D on the right- of (2) measures the quality of the conditional distribution approximation. [Bellot& & van der Schaar (2019) argued that this error term is negligible due to the capacity of deep neural nets in estimating conditional distributions. To the contrary, we find this approximation error is usually not negligible, and consequently, it may inflate the type-I error and invalidate the test. We consider a simple example to further elaborate this.
Example 1. Suppose X is one-dimensional, and follows simple linear regression model, X Z Bp + E, where the error is independent of and 5 ~ ( of) for some oă > O.
Suppose we know priori that the linear regression model holds. We thus estimate 3g by ordinary least squares, and denote the resulting estimator by 3. For simplicity, suppose of known too. For this simple example, we have the following result regarding the approximation error term D.
Proposition 1 Suppose the linear regression model holds. The derived distribution P Pxiz is
N N(2,, where T, is the n identity matrix. Then D is not o(1).
2

---

Under review as a conference paper at ICLR 2021
To facilitate the understanding of the convergence behavior of D, we sketch few lines of the proof of Proposition A detailed proof is given in Appendix F.1 Let Pxiz=Z, denote the conditional
distribution of X (m) given Z₁ which is (Z, B, og) in this example. If D o(1), then,
D 1/2 V EdGy (Px1222 Px z=2.) o(1).
In other words, the validity GCIT requires the root mean squared total variation distance (3) to converge at a faster rate than n 11/2, However, this rate cannot be achieved in general. our simple Example 1, we have D 2 c for some universal constant not o1)). Proposition D shows that, even if we know a priori that the linear model holds, D is not to decay to zero as n grows to infinity. In practice, we do not have such prior model information. Then it would be even more difficult estimate the conditional distribution Px|2. Therefore, using GANs to approximate Pxiz does not guarantee a negligible approximation error, nor the validity of the test.
0. Consequently, D in 2 is
2.2 REGRESSION BASED BASED TESTS
The family of regression- tests is built upon a key quantity, the generalized covariance measure,
- GCM(X, = (x.- ( -xx-2)))))
where 2( and (Z) are the predicted condition mean 2( and B(Y respectively, by any supervised learner. When the prediction errors of É( ((X|2) and E(Y|Z) satisfy certain con- vergence rates, Shah & Peters| (2018) proved that GCM is asymptotically normal. Under Ho, the asymptotic mean of GCM is zero, and its asymptotic standard deviation can be consistently esti- mated by some standard error estimator, denoted by ((GCM). Therefore, at level a, we reject Hos if [GCM)/S( (GCM) exceeds the upper a/2th quantile of a standard normal distribution.
Such a test is valid. However, it may not have sufficient power to detect H.1 This is because the asymptotic mean of GCM equals GCM*(X,Y) based tests require |GCM"| to be nonzero under Hi to have power. However, there is no guarantee of this requirement. We again consider a simple example elaborate.
B{X E(X (e))(((-())) Z)}. The regression-
Example 2. Suppose X", Y and Z are independent random variables. Besides, X" has mean zero, and X X²)(YY) for some function g.
For this example, we have B(X = B(X), since both X* and Y are independent of Z, and so is X. Besides, E(X) E( B{g(Y)) O, since X is independent of Y and E(X*) 0. As such, GCM (x, Y) B(X B E(Y [ = 0 for any function On the other hand, X and are conditionally dependent given Z, as long as g not constant function. Therefore, for this example, the crression-basede tests would fail to discriminate between Ho and Hi.
2.3 MMD-BASED BASED TESTS
The family of kernel-basee tests often involves the notion of maximum mean discrepancy as mea- sure of independence. For any two probability measures P, Q and function space F, define
MMD(P, Q)F) sup (Ef(Wi) Ef(Wa)), W1 ~ P. W2 Q.
Let [H],, Hg be some function spaces of X and Y. Define éxY = MMD(Pxy: Qxy & H2), where & is the tensor product, Pxy is the joint distribution of (X, Y). and Qxy is the conditionally independent distribution with the same X and Y margins as Pxy. Then following the calculations in Appendix D we have, éxY eu . (X) (()))))(((()()))))))) (Y) [-)) We see that yxy measures the average conditional association between X and Y given Z. Under Ho. it equals zero, and hence estimator of this measure can be used as test statistic for Ho.
A NEW DOUBLE GANS-BASED TESTING PROCEDURE
We propose double GANs-based testing procedure for the conditional independence testing prob- lem (I). Conceptually, our test integrates GCIT, regression- baased and MMD-based tests. Mcan- while, new test addresses the limitations of the existing ones. Unlike GCTT that only learned the
3

---

Under review as a conference paper at ICLR 2021
AU
x
H
Gx
X
Dx
Vx
Deeaae T&T
7.
G7
Dy
7
Vy
hau
Figure I: Illustration of the conditional independence test with double GANs.
conditional distribution of X Z, we learn two generators Gx and Gy to approximate the conditional distributions of both X|Z and ZI.. We then integrate the two generators in an appropriate way to construct a doubly-robust test statistic, and we only require the root mean squared total variation norm to converge at a rate of n for some K 1/4. Such a requirement is much weaker and practically more feasible than the condition in (3) The notion of doubly-robustness property comes from the classical semiparametric theory in statistics (Tsiatis) (007). Specifically, a doubly robust procedure applies both types of models simultaneously, and produces a consistent estimateif either of the two models has been consistently estimated.
Moreover, improve the power of the test, we consider a set of the GCMs, (GCM(h) (X), hz2)))))))))))))) f1, J. for multiple combinations of transformation functions h4 (X) and (2()). We then take the maximum of all these GCMs our test statistic. This essentially yields éxY, which is connected with the notion of MMD. To see why the maximum-type statistic can enhance the power, we quickly revisit Example 2. When g is not a constant function, there exists some nonlinear function h such that (Y) B{h (X))Y) is not a constant function of Y. Set h2 = hj We have GCM" = Elh (Y - B(Y) = Var(hi(Y)) > 0. This enables us to discriminate Hi from Ho.
We next detail our testing procedure. A graphical overview is given in Figure
3.1 TEST STATISTIC
We begin with two function spaces, H, [hi.e, : 81 E Rd) and H2 {h2.0, 02 E R indexed by some parameters 82 and 02. respectively. In our implementation, we set H₁ and H2 to the classes of neural networks with single-hidden layer, finitely many hidden nodes, and the sigmoid activation function. We then randomly generate functions, h1.1,... hi,8 " H., fu2.1.... f2, E H22, where we independently generate i..d. multivariate normal variables .1,1,..., ,0,8 ~ (0, 21s, /di). and o2,1, 02,B ~ [0, 3d2) We then set 11,5 = h1 and h2g6 h2202,0 b b= 1,..., B. Consider the following maximum-type test statistic,
1 n
max b.,beel,..... B)
Ty ba
Σ n
M (X.) E(hi,, Z3)
122, (V:) - B( (Yg) (13)
(4)
where oh hh is a consistent estimator for VGGM((h) (X), h2 (Y)). See Section A for definition. To compute (4) however, we need to estimate the conditional means E( (2) (Y) for b,, b2 1,..., B. In theory, should diverge to infinity to guarantee the power property of the test. Separately applying supervised learning algorithms 2B times to compute these means is computationally very expensive. Instead, we propose to implement this step based on the generators Gx and Gy estimated using GANs, which is computationally much more efficient.
(m) JM Specifically, for i I,..., n, we randomly generate i.i.d. random vectors {u Jm=I fur J==1 and output the pseudo samples X(m) Gx (Z,, 1 ) f((m)
(m)
Gy (Z,, u (mm) for m
1,..., .,, to approximate the conditional distributions of X, and Y, given Z,. We then compute E(ha,, (X,) = M-1 >M 3m=1 1,,,, (x:())), and E(h,,,, (Y;) [ Z, = M- ΣM h2 (7)(m)) for b1,b2 b2 1,... B. Plugging those estimated means into produces our test statistic, T = maxb, bz
11/2 Ei_ Uhr b2.6. /. where
M
M Σ 122 (v:(0))) m=l
Ub, b2. = Tryios
11 (Xi) (X:)
Σ h1.
(x (x((0))
h2222 (Yi)
M
M
m=1
4

---

Under review as a conference paper at ICLR 2021
Input:
Number of functions B, number of pseudo samples M, and number of data splits
Step 1: Divide {1, ......... n into folds I _(!.....
Let I(( ,
..... II()),
(L.....,) n)
I(().
Step 2: For l = l..... L, train two generators G and G based on {(,, Z.) 2,zeoz and
{(Y,, Nez((n) to approximate the conditional distributions of X|2 and YIZ.
Step 3: For l L..... and i Ie, generate i..d. random noises
1 m=1"
(m)
) (m) M
m==1"
Set (m) Φ (Z,, ( and p/ GfP (
(m) m ....., M.
I,..,..... ....,,.......................... h1,B ...,,,.. E Hil and h........ 12,8 lill. Step 5: Compute the test statistic T.
Step 4: Randomly generate h1,1,
Algorithm 1: Algorithm for computing the test statistic.
To help reduce the type-I error of our test, we further employ a data splitting and cross-fitting strat- egy, which is commonly used in statistical testing (Romano & DiCiccio 2019). That is, we different subsets of data samples to learn GANs and to construct the test statistic. We summarize our procedure of computing the test statistic in Algorithm
3.2 BOOTSTRAPPING THE P-VALUE
Next, we propose a multiplier bootstrap method to approximate the distribution of viT under Ho to compute the corresponding p-value. The key observation is that th,, d2 = n 1/2 Σ 4/ .b.. asymptotically normal with zero mean under Hoi see the proof of Theorem 3 in Appendix F3 for details. As such, /nT = max6,,2 |n-1/2 E variables in absolute values.
.b. is
== bb is to converge to maximum of normal
To approximate this limiting distribution, we first estimate the covariance matrix of a B². dimensional vector formed by (1r,, 16, using the sample covariance matrix E. We then generate i..d, random vectors with the covariance matrix equal to Σ, and compute the maximum elements of each of these vectors in absolute values. Finally, we use these maximum absolute values to approximate the distribution of T under the null. We summarize this procedure in Algorithm
3.3 APPROXIMATING CONDITIONAL DISTRIBUTION VIA GANs
We adopt the proposal in Genevay et al. 2017) to learn the conditional distributions Px|z and Priz Recall that Pxiz is the distribution of pseudo outcome generated by the generator Gx given Z. We consider estimating Px|z by optimizing mingx max. De,e (Px z Px1z), where D., denotes the Sinkhorn loss function between two probability measures with respect to some cost function e and some regularization parameter >> 0. A detailed definition of D... is given in Appendix B Intu- itively, the closer the two probability measures, the smaller the Sinkhorn loss. As such, maximizing the loss with respect to the cost function learns a discriminator that can better discriminate the sam- ples generated between Pxiz and Px|z- On the other hand, minimizing the maximum cost with respect to the generator Gx makes closer to the true distribution Px|z- This yields the minimax
Input: Number of bootstrap samples J. and (1/b, do.. D.,,,,
62.5.
Step 1: Compute 2² x B² matrix Σ whose {b1 + B(b2 - 1), by + B(b, th entry given by (n 1) Σx= (vh, .z.. 1/d, br (177,,... 1/bs.h.). ).
J, Step 2: Generate i.i.d. standard normal variables ZG, for 1,, ,,..., B². Set
Z, (Z..,...., BT) and T, = S2/2 Zy 1lo.. where 51/2 is positive semi-definite matrix that satisfies 11/251/2 = E, and || I1 is the maximum element of a vector in absolute values.
Step 3: Compute the p-value, J-1 Σj= 1(T 2 Tz).
Algorithm 2: Algorithm for computing the v-luue
5

---

Under review as a conference paper at ICLR 2021
formulation mingx max; De.c(Px122PPr) that target. In practice, we approximate the cost and the generator based on neural networks. Integrations in the objective function De.. (Px PP Px(z) approximated by sample averages. A pseudocode detailing our learning procedure is given in Appendix B The conditional distribution Priz is estimated similarly.
We make remarks. First, our proposed framework is general, and any GANs learning procedure can be applied. In our implementation, we have chosen the Sinkhorn GANs because of its compet- itive performance. We did not use the original GANs due to its instability and the mode collapse issue. We did implement WGANs, but found the estimated distributions may suffer from the large bias resulting from the gradient penalty enforced in WGANs. Second, it is important to check the gfoodness-of-fi"" of the generator. In practice, this could be achieved by comparing the conditional histogram the generated samples to that of the true samples. See Appendix C for details.
4 ASYMPTOTIC THEORY
To derive the theoretical properties of the test statistic T, we first introduce a concept of the "oracle" test statistic T If Pxiz and Priz were known priori, then one can draw (x(m) }mm and (y-(m) from Pxiz=z, and Przzzz directly, and can compute the test statistic T by replacing {X, 25 (m)
Im
mm
and (Y((m)) Im with (x("),, Im and {Y, [y("") Imm- We call the resulting T an "oracle" test statistic.
We next establish the houble-robustneet property of T' which helps better understand why proposed test can relax the requirement in (3). Informally speaking, the double-robustness means that T is asymptotically equivalent to T when either the conditional distribution of X X|Z, or that of Z is well approximated by GANs. It guarantees T converges to T* at a faster rate than the estimated conditional distribution. In contrast, the convergence rate of the GCIT test statistic the same as the estimated conditional distribution. As such, our procedure requires a weaker condition.
Theorem (Double-robustness)) Suppose M is proportional to n, and B = Cgn for some con- 0. Then T T =1), when either (F(dGv (E) (-12), (x(1))))))
stants C, Co
o(log 1/2 n). or (E(Gvv (( (12).
o(log 1/2 n).
The conditions on M and B mild, as these user-specified parameters. As we have discussed, when both total variation distances converge to zero, the test statistic T converges at faster rate than those total variation distances. Therefore, we can greatly relax the condition in (3), and replace it with, for any l 1,..., L,
[E(cev (()) 2 Pxiz)]] 1/2 (0(n")) and [B(dGv Priz),Pyi))) Py(2))1 =(n
(5)
for some constant 0 < ń < 1/2, where p(0)
x]2 and p(() denote the conditional distributions approx- imated via GANs trained on the l-th subset. The next theorem summarizes this discussion.
Theorem Suppose the conditions in Theorem aand Then T T Op(n" -2x log n).
Since O, the convergence rate of (T T*) is faster than that in (). To ensure vn(T TT) (1), ), it suffices to require k > 1/4. In contrast to (3). this rate is achievable. We consider three examples illustrate, while the condition holds in wide range of settings.
Example 3 (Parametric setting). Suppose the parametric forms of Qx and Qy are correctly spec- ified. Then the requirement > 1/4 holds if k O(n')) for some to 1/4, where k is the dimension of the parameters defining the parametric model.
Example 4. (Nonparametric setting with binary data). Suppose X, Y are binary variables. Then suffices to estimate the conditional means X and Y given Z. The requirement k > 1/4 holds if the mean squared prediction errors of both nonparametric estimators are O(n))) for some Ko 1/4.
Example 5. (Nonparametric setting with general data). Suppose X, Y are continuous variables. We apply GANs learn the condition distributions. Chen et al. (2020) established the statistical properties of GANs, and one can apply their technical tools to check the convergence rates in (5) In general, the convergence rate depends on the smoothness of the conditional density function. The smoother the conditional density, the faster the rate.
Next, we establish the size and the power properties of our proposed test.

---

Under review as a conference paper at ICLR 2021
Theorem Suppose the conditions in Theorem 7 hold. Suppose 5 holds for some K >> 1/4. Then the p-value from Algorithm 2 satisfies that Pr(p I a(HH)) = (1).
Theorem 3 shows that our proposed test can control the type-I error. Next, to derive the power property, introduce the pair of hypotheses based on the notion of weak CI (Daudin 1980): * 0,
HG (Cov(f(()),,.. (()))) = 0, Vf E L'x. E LÝ versus Hi ECov(f(X),
where 2 and E denote the class of all squared integrable functions of X and Y, respectively. We note that conditional independence implies weak conditional independence, i.e., Hg implies Hij Consequently, Hi implies Hi. The next theorem shows that proposed test is consistent against the alternatives in Hi. but not against all alternatives in H1.
Theorem Suppose the conditions in Theorem hold. Then the p-value from Algorithm satisfies that Pr(p a {{{)) L,
00.
Finally, we remark that our test is constructed based on Φxy. Meanwhile, we may consider another test based on @xYZ = MMD(Pxyz. QXY |i| & H2 @ H3), where PXY is the joint distribution of (X, Y, Z), QXYZ Px 2 P2. and H3 is neural network class of functions of Z. This type of test is consistent against all alternatives H4. However, in our numerical experiments, we find it less powerful compared to test. This agrees with the observation by [i & Fan (2019) in that, even though the tests based on weak CI cannot fully characterize CI, they potentially benefit from an improved power.
NUMERICAL STUDIES
We give the implementation details of our testing procedure in Appendix A The time complexity of our test is dominated by Step of Algorithm I where distributions ix|z and Priz- The complexity each SGD iteration is ORRN"); see Appendix 16 NI CPUs Google Cloud Computing platform. The wall clock
GANs estimate the conditional
All experiments were time for computing a single test statistic was about 2.5 minutes.
5.1 SYNTHETIC DATA EXAMPLE
We generate synthetic data following the post non-linear noise model similarly as in Zhang et al. 2011 Doran al. (2014): Bellot & van der Schaar 2019))
= sin(aj +E) and Y = cos(a J Z bX + +
The entries of a 0g are randomly and uniformly sampled from [0, 1]. then normalized to unit norm. The noise variables are independently sampled from normal distribution with mean zero
8::005
6.4
KCT
K07
0.4
CST
085
040
KCT
6.5
E30
3.07
Dineriler
#-011
a-003
OCT
3007
D0ET
8007
04
Figure 2: Top panels: the empirical type-I error rate of various tests under Ho. From left to right: normal Z with a 0.1, normal Z with = 0.05, Laplacian Z with = 0.1, and Laplacian Z with a 0.05. Bottom panels: the empirical power of various tests under Hy, From left right: dz 100, = 0.1, dz 100, a 0.05, dz = 200, 0.1, and dz 200, a = 0.05.

---

Under review as a conference paper at ICLR 2021
Table 1: The variable importance measures of the elastic net and random forest models, versus the p-values of the GCIT and DGCIT tests for the anti-cancer drug example.
B60E
BRAEMC
HIPI
FTL3
CDC42BPA
THBS3
DNMTI
PRKDI
PIPSKIA
MAP3K5
EN
3
4
5
8
10
19
78
RF
2
3
14
34
28
18
GCIT DGCIT
<0.001
<0.001
0,008
0.521
0,050
0.013
0.020
0.002
0.001
<00001
0
0
0
0
0
0.794
and variance 0.25. In this model, the parameter b determines the degree of conditional dependence. When b 0, Ho holds, and otherwise Hi holds. The sample size is fixed at v = 1000.
We call our test DGCIT, short for double GANs-based conditional independence test. We compare it with the GCIT test of BBellot& & van der Schaar (2019), the regression- test (RCIT) of Shahh & Peters (2018) the kernel MMD-based test (KCIT) of Zhang et al. (2011). the classifier CI test (CCIT) of Sen et al. (2017) and the deep Haaming-basee CI test (DL-CIT) of Sen et al. (2018).
Type-I error under Hu- We vary the dimension of Z as d2 = 50, 100, 150, 200, 250, and consider two generation distributions. We first generate Z from a standard normal distribution, then from Laplace distribution. We set the significance level at cantareplcctoon report the empirical size of the tests aggregated over 500 data replications. We make the following observations. First, the type-I error rates of our test and RCTT are close to below the nominal level in nearly all cases. Second, KCIT and DL-CIT fail in that their type-I errors are considerably larger than the nominal level in all cases. Third, GCIT and CCIT have inflated type-I errors some cases. Take GCIT as an example. When Z is normal, dz = 250 and a = 0.1, its empirical size is close to 0.15. This is consistent with our discussion in Section 2.1 as GCIT requires a very strong condition to control the type-I error.
0.05 and 0.1. Figure 2 top panels
Powers under Hi. We generate Z from a standard normal distribution, with dz 100, 200, and vary the value of b = 0.3, 0.45, 0.6, 0.75, 0.9 that controls the magnitude of the alternative. Figure bottom panels report the empirical power of the tests over 500 data replications. We observe that our test is the most powerful, and the empirical power approaches I as b increases to 0.9, demonstrating the consistency of the test. Meanwhile, both GCIT and RCIT have no power in all cases. We did not report the power of KCIT and DL-CIT, because as we show earlier, they can not control the size, and thus they empirical powers are meaningless.
5.2 ANTI-CANCER DRUG DATA EXAMPLE
We illustrate our proposed test with an anti-cancer drug dataset from the Cancer Cell Line Ency- clopedia (Barretina et al. 2012). We concentrate on a subset, the CCLE data, that measures the treatment response of drug PLX4720. It is well known that the patient's 's cancer treatment response to drug can be strongly influenced by alterations in the genome (Garnett al.) 2012) This data mea- sures 1638 genetic mutations of 472 cell lines, and the goal of our analysis is to determine which genetic mutation is significantly correlated with the drug response after conditioning on all other mutations. The same data also analyzed in Tansey et al. 2018 and Bellot & van der Schaar (2019). We adopt the same screening procedure as theirs to screen out irrelevant mutations, which leaves total of 466 potential mutations for our conditional independence testing.
The ground truth is unknown for this data. Instead, we compare with the variable importance mea- sures obtained from fitting elastic net (EN) model and a random forest (RF) model reported in Barretina et al. (012)). In addition, we compare with the GCIT test of Bellot & van der Schaar (019)). Table D reports the corresponding variable importance measures and the p-values, for 10 mutations that were also reported by [Bllot & van der Schaar 2019). We see that, the p-values of the tests generally agree well with the variable important measures from the EN and RF models. Meanwhile, the two conditional independence tests agree relatively well, except for two genetic mu- tations, MAP3K5 and FTL.3. GCIT concluded that MAP3K5 is significant (p < 0.001) but FTL3 is not (p- 0.521), whereas our test leads to the opposite conclusion that MАP3K5 is insignificant (p = 0.794) but FTL3 is (p- = 0). Besides, both EN and RF place FTL3 as an important mutation. We then compare our findings with the cancer drug response literature. Actually, MAP3K5 has not been previously reported in the literature as being directly linked to the PLX4720 drug response. Mcanwhile, there is strong evidence showing the connections of the FLT3 mutation with cancer sponse (Tsai et al. [2088] Larrosa- Garnia & Baer 2017). Combining the existing literature with our theoretical and synthetic results, we have more confidence about the findings of our proposed test.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Jordi Barretina, Giordano Caponigro, Nicolas Stransky, Kavitha Venkatesan, Adam A Margolin, Sungjoon Kim, Christopher J Wilson, Joseph Lehár, Gregory V Kryukov, Dmitriy Sonkin, et al. The cancer cell line encyclopedia enables predictive modelling of anticancer drug sensitivity. Nature, 483((39)):603-60))) 607, 2012.
Andrew R Barron. Universal approximation bounds for superpositions of a sigmoidal function. IEEE Transactions Information theory, 39(3):330-94) 1993.
Alexis Bellot and Mihaela van der Schaar. Conditional independence testing using generative adver- sarial networks. In Advances in Neural Information Processing Systems, pp. 2199-2208, 2019.
Wicher Pieter Bergsma. Testing conditional independence for continuous random variables. Euran- dom, 2004.
Thomas B Berrett, Yi Wang. Rina Foygel Barber, and Richard Samworth. The conditional permu- tation test for independence while controlling for confounders. Journal of the Royal Statistical Society: Series B (Statistical Methodology), accepted, 2019.
Emmanuel Candes, Yingying Fan, Lucas Janson, and Jinchi Lv. Panning for bodd-moode-xxkooooool for high dimensional controlled variable selection. Journal of the Royal Statistical Society; Series B (Statistical Methodology), 80(3):551-577, 2018.
Minshuo Chen, Wenjing Liao, Hongyuan Zha, and Tuo Zhao. Statistical guarantees of generative adversarial networks for distribution estimation. arXiv preprint arXiv:2002.03933, 2020.
Victor Chernozhukov, Denis Chetverikov, and Kengo Kato. Gaussian approximation of suprema of empirical processes. The Annals of Statistics, 42(4):1564-1190 2014.
Victor Cheroozuukov, Denis Chetverikov, and Kengo Kato. Comparison and anti-concentration bounds for maxima of Gaussian random vectors. Probahility Theory and Related Fields, 162 (1-2):47-70, 2015.
2015.
Victor Chernozhukov. Denis Chetverikov, and Kengo Kato. Detailed proof of nazarov's inequality. arXiv preprint arXiv: 10696, 2017.
Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. In Advances in neural information processing systems, pp. 2292-2300, 2013.
JJ Daudin. Partial association measures and an application to qualitative regression. Biometrika, 67 (3):8,,, 1980.
Lue Devroye, Abbas Mehrabian, and Tommy Reddad, The total variation distance between high- dimensional gaussians. arXiv preprint 1xXvv:181000869, 2018
Gary Doran, Krikamol Muandet, Kun Zhang, and Bernhard Schölkopf. A permutation-based kernel conditional independence test. In UAI, pp. 132-141, 2014.
Kenji Fukumizu, Arthur Gretton, Xiaohai Sun, and Bernhard Schölkopf. Kernel measures of condi- tional dependence. In Advances in neural information processing systems, pp. 489-496, 2008
Mathew Garnett, Elena Edelman, Sonja Gill, Chris Greenman, Anahita Dastur, King Lau, Patricia Greninger, Richard Thompson, Xi Luo, Jorge Soares, Qingsong Liu, Francesco Iorio, Didier Surdez, Li Chen, Randy Milano, Graham Bignell, Ah Tam, Helen Davies, Jesse Stevenson, and Cyril Benes. Systematic identification of genomic markers of drug sensitivity in cancer cells. Nature, 483:570-5, 03 2012. doi: 10. 10.1038/nature11000
Aude Genevay, Gabriel Peyré, and Marco Cuturi. Learning generative models with sinkhorn diver- gences. arXiv preprint 1706.027700002090 2017.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farles Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural infor- mation processing systems, pp. 2672-2680, 2014.

---

Under review as a conference paper at ICLR 2021
Arthur Gretton, Karsten M Borgwardt, Malte Rasch, Bernhard Schölkopf, and Alexander Smola. A kernel two-sample test. Journal of Machine Learning Research, 13(Mar):723- 777-773, 2012.
Patrik O Hoyer, Dominik Janzing, Joris M Mooij, Jonas Peters, and Bernhard Schölkopf. Nonlin- ear causal discovery with additive noise models. In Advances in neural information processing systems, pp. 689-696, 2009.
D. Koller and N. Friedman. Probabilistic Graphical Models: Principles and Techniques. Adaptive computation and machine learning. MIT Press, 2009.
Maria Sarrosa-Garchs and Maria R. Baer. FIt3 inhibitors acute mycloid leukemia: Current status and future directions. Molecular Cancer Therapeutics, 16(6):9111101,, 2017.
Bing Li. Sufficient Dimension Reduction: Methods and Applications with R. CRC Press, 2018.
Chun Li and Xiaodan Fan. On nonparametric conditional independence tests for continuous vari-
ables. Wiley Interdisciplinary Reviews: Computational Statistics, pp. e1489, 2019.
Judea Pearl. Causality: Models, Reasoning and Inference. Cambridge: Cambridge University Press, 2nd Edition, 2009.
J Romano and Cyrus DiCiccio. Multiple data splitting for testing. Technical report, Technical report, 2019.
Rajat Sen, Ananda Theertha Suresh, Karthikeyan Shammugam, Alexandros Dimakis. and Sanjay Shakkottai. Model-powerts conditional independence test. In Advances in neural information processing systems, pp. 2951-2961, 2017.
Rajat Sen, Karthikeyan Shanmugam, Himanshu Asnani, Arman Rahimzamani, and Sreeram Kan- nan. Mimic and classify: A meta-algorithm for conditional independence testing. arXiv preprint
arXiv:1806.09708. 2018.
Rajen D Shah and Jonas Peters. The hardness of conditional independence testing and the gener- alised covariance measure. arXiv preprint 10K4v:8804.072000 2018.
Liangjun Su and Halbert White. A consistent characteristic function-based test for conditional in- dependence. Journal of Econometrics, 141(2):807883) 2007.
Liangjun Su and Halbert White. Testing conditional independence via empirical likelihood. Journal of Econometrics, 182(1):27-44, 2014.
Wesley Tansey, Victor Veitch, Haoran Zhang, Raul Rabadan, and David M Blei. The hold- out randomization test: Principled and easy black box feature selection. arXiv preprint arXiv:1811.00645. 2018.
James Tsai, John T. Lee, Weiru Wang, et al. Discovery of selective inhibitor of oncogenic b-raf kinase with potent antimelanoma activity. Proceedings of the National Academy of Sciences, 105 (8):3041-3046, 2008. doi: 10. (0.1073/pnas.0711744 171411105...
Anastasios Tsiatis. Semiparametric theory and missing data. Springer Science & Business Media, 2007.
Xueqin Wang, Wenliang Pan, Wenhao Hu, Yuan Tian, and Heping Zhang. Conditional distance correlation. Journal of the American Statistical Association, 1106512): 2015.
Hao Zhang, Shuigeng Zhou, and Jihong Guan. Measuring conditional independence by indepen- dent residuals: Theoretical results and application in causal discovery. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018
Zhang, Peters, D Janzing, and B Schölkopf. Kernel-based conditional independence test and application in causal discovery. In 27th Conference on Uncertainty in Artificial Intelligence (UAl 201 pp. 803. AUAI Press, 2011.
10

---

Under review as a conference paper at ICLR 2021
A ADDITIONAL DETAILS ABOUT THE PROPOSED TESTING PROCEDURE
We first discuss the computation time. In our synthetic data example, it took about 2.5 minutes to compute test, 2 minutes to compute CCIT, and 20 seconds compute GCIT and DL-CIT.
We next discuss some implementation details. For the number of functions in Algorithm D represents trade-off. By Theorem 4 B should be large possible guarantee good power. In practice, the computation complexity increases as B increases. Our numerical studies suggest that the value of B between 30 and 50 achieves good balance between the power and the computational cost, and we fix = 30. For the number of pseudo samples M, and the number of sample splittings not overly sensitive their choices, and thus fix M 100 and 7. = 3.
T., we find the results For the GANs, we use single-hidden layer neural network to approximate both the discriminator and generator. The number of nodes in the hidden layer is set 128. The dimension of the input noise U ((m) and u/,y set at 10. The performance of GANs is largely affected by the regularization parameter c and the number Sinkhorn iterations R. In our experiments, we set c = 0.8 and R 30. In practice, resulting generator. This could be achieved by comparing the conditional histogram of the generated samples to that of the samples. See Appendix for details. We use stochastic gradient descent (SGD) to update the parameters in GANs. The batch size N is set to 64. We find the resulting GANs work reasonably well in our experiments.
suggest tune these parameters by investigating the goodness-of-fi of the
Finally, we introduce the variance estimator, Define
[MLLs.X)) E(hi, (x)) [h2.0 (Y,) - E(h2,oz (Y)))
Th
max
n2
GGCM (EMM)h,,,XX)bba) (X), (x)))))))))))))
.60 .60
for any dy and b2, where denotes some sufficiently small constant. The constant is to guarantee that the denominator of 1h0,, .a, is strietly greater than zero, such that the proposed test has the desired size and power properties.
B ADDITIONAL DETAILS FOR CONDITIONAL DISTRIBUTION APPROXIMATION USING GANS
We first introduce the notion of optimal transport (OT), Let X' and y be some closed subsets of N and v be the probability measures on and y, respectively. The Kantorovich formulation of optimal transport is defined by,
/ c(I, ((x,8)T(da) dy).
Dc( inf =Ell(u.))
where D is the OT of i into with respect to cost function c(., ). and II(,,, 1/) is set containing all probability measures ı whose marginal distributions on X' and y correspond to and v.
Notably, when the cost function is the Fuclidean distance between the arguments, D is better known as the Wasserstein distance. To facilitate the computation of evaluating OT losses, Cuturr] (2013) and Genevay al. (2017) suggested to add an entropic regularization D.. This yields the objective function,
De,c(u.v)= inf xEll(p.r) / (c(x, - (Tl & (dx,
where H denotes the Kullback-Leine divergence, and μ & v is the product measure of μ and v. Such objective function can be efficiently evaluated by the Sinkhom algorithm (Cuturi 2013 Genevay et al. 2017). To alleviate the bias resulting from the entropic regularization term, Cuturi (2013) and Genevay al. (2017) further considered using the Sinkhorn loss function, defined as, ( (D. v) p),((u,)) (v,
Next, we present our algorithm for learning the generator G in Algorithm where, V and y denote the empirical distributions of {r,) and (1/) respecivelly.
11

---

Under review as a conference paper at ICLR 2021
Input: data (Xi)igzio, (eroo probability distribution Ç on the latent space V, initial values of On and yoo batch size N, regularization parameter e, number of Sinkhorn iterations R. and learning rate a.
Repeat:
Train discriminator:
Sample (xe) } from {(X, Nezoo
Sample {2²] from (Zi)ez
Sample {u' } from S;
V/ - {(Go(",, v)), for all j:
40 + & + +V D De. (x", ))
Train generator:
\N Sample {x² J3=1 from {(x,, Li)ii
Sample (- } from sero:
Sample {ur [ from Ç;
w/g (Ga(z ,)),))) 2) for all j:
- 0 a-a De, (x", 1)
until convergence.
Algorithm 3: Algorithm for training Sinkhorn GANs by stochastic gradient descent.
C ADDITIONAL DETAILS FOR GANS MODEL CHECKING
Let dz denote the dimension of Z, and Í the sample average n - E. Z, Let Gy v,,)
denote a simulated sample to approximate the distribution of Y = Z, obtained by the generator Gy. When Gy is accurate, expect the conditional distribution of Y, and Y, given Z; are similar.
As such, for any dz-dimensional vector a, the histograms (Ý; a T((Z, fizz) 0} and (Y; :
a' (Z, - p2) > 0} should be similar. We sample i.i.d. normal vectors {ag) from N Iaz ). For
each g. we plot the histogram (Y; : a (Z, ī > 0} and (Y.(m) ag (Z, 7Z) > 0}. See Figures 3 (a) and (b) for the conditional histograms with two choices of ag. respectively. It can be seen that the Sinkhorn GANs fit the conditional density reasonably well.
The fitted conditional distribution for Pxiz could be checked in similar fashion as well.
Samples
6.0
17B
6.00
1.44
0.5
10
152
(b)
Figure 3: Conditional Histograms. GANs are trained using data generated from the simulation study (see Section 5.I).
We next comment on the performance of the WGANs using Bellot & van der Schaar (0099)s 's code. Specifically, we apply their code to their simulation setting well as experiments train the WGAN. We then plot the histograms of the real samples and the WGAN samples scaled between 0 and 1 in Figure It can be seen that the WGANs perform poorly in both settings. For instance,
12

---

Under review as a conference paper at ICLR 2021
OCT1 IRDAN Samplies
Real Samppies
OCT WEJAN Serppies
Real Samplees
WCAR Samppies
94-
92
CCT Samplie
Real Samplies
GC1 WGAN Samplles
Red Samples
WGAN Samplaa
100
108
200
612
Figure 4: The line plots and histograms of real samples and GCIT WGAN samples (scaled between O and 1). The top two panels are the results Bellot & van der Schaar (2019)'s setup with three different simulations and the bottom top panels, are the results under our setup with three different simulations.
under Bellot & van der Schaar (2019'ss setup, the WGAN samples are very close zero and do not have much variability. This is very different from the distribution the real samples.
D ADDITIONAL DETAILS FOR MMD
Let X' and Y' be independent copies of X and Y such that they are conditionally independent given Z. Note that,
E(b)(X)))((Y') (E) (E
Henceforth,
éx MMD(Pxv: QXY H1 122) he (())(()))))))) (XX))))))))
sup
sup Ag
[B(hi(X)ae (Y)) - (112311((1)223
sup (X)((()))))))) E(M (X) h2 [2)] (((x)(()))((
(x) - F() [12(]) - -((2(()22)] + (IE(( = sup
E
AgE
E ADDITIONAL NUMERICAL RESULTS
In our simulation settings, we find that DL-CIT, CCIT and KCIT cannot control the type-I error in finite samples. In particular, the type-I errors of DL-CIT are larger than 0.2 in almost all cases. We first use DI-CIT an example to investigate its type-I error with larger sample size. We fix dz 150, and generate Z from a standard normal distribution. can be seen that DL-CIT requires very large sample size, e.g., 5000, in order control the type-I error.
We next conduct additional experiments to investigate the performance of the proposed test with small sample size, i.e., when 500. We fix the dimension d = 100, generate Z from standard normal distribution, and set 5 to 0.9 under H1. We did not implement DL-CIT and KCIT, because they no longer valid test is consistent under this small sample size setting too. Meanwhile, GCIT and RCIT do not have powers under Hi. CCIT has inflated type-I error under Ho. Specifically, its type-I emors under
when 1000. Table report the results. It is that the proposed
13

---

By Theorem 1.2 of Devroye al. 2018), have dTv Px 2) is proportional to Under review as a conference paper at ICLR 2021
Type error comparison for DL-CIT with various sample sizes dimension 150
040
e 0.1
030
Type eeror
0.5
4000
Figure Type-I errors of DL-CIT with larger sample sizes.
Ho
H1
DGCIT
GCIT
RCIT
CCIT
DGCIT
GCIT
RCTT
0.05
0.04
0.06
0.05
0.08
0.22
0.07
0.07
0.1
0.06
0.13
0.11
0.13
0.35
0.10
0.10
Table Empirical type-I error rate and power of various tests under Ho and H1. n 500, d 100,
normal Z and 8 0.9 under Hi,
0.05 and 0.1 0.08 and 0.13, respectively. The differences 0.08 0.05 and 0.13 0.1 statistically significant they exceed the Monte Carlo error 1.96 /0.05 0.95/500 0.019 and 1.96 /0.6 = 0.9/500 0.026. Thus its power becomes meaningless.
F
PROOFS
We provide the proofs of Proposition I. and Theorems since is similar as that of Theorem classes Hy, and HH],, which are set to the classes of neural networks with a single-hidden layer, finitely many hidden nodes, and the sigmoid activation function, used in implementation. Meanwhile, the results can be extended to more general cases.
3 and We omit the proof Theorem I Cecoorems[[aar established under our choice of the function
F.1 PROOF OF PROPOSITION 1
Note that the total variation distance bounded by 1. Suppose Edrv(Px Then have drv (Px 2x (1). By the dominated convergence theorem, have ( Pxiz) o(1).
2, o(1).
min
1, 1,
Σ - B2) -
i=1
It follows that
[2] -0-3))- o(1)
14

---

Under review as a conference paper at ICLR 2021
Applying Theorem 1.2 of Devroye et al. (018 again, obtain drv (Px 22, Pxiz=z)) is pro-
portional to
min 1.00 | (3 B00]
Thus, we obtain
72
Σ Edii (x1=2=) o(1).
Since the data is exchangeable, we have
Edfy [xxzz= Px|z=z) o(n
(6)
This shows that when RHS of is (1). 6 automatically holds.
Next, we show
(6 is violated in the linear regression example. By the data exchangability, it suffices
to show Σ 1= Ediv Px Px|z= 2 } is not o(1). With some calculations, we obtain
Emin 1,00 (3 Bo) = Σ Bo 21, - (6) (3 30)1 1
ET (o 1 (3 B0)1 1) Σ Eg 7 ( Bo)1
(7)
Elog (8 - B01 - 11 (3 B0) > 17.
By the definition of 3, we have
Σ EBT (3 f0) _ 12
1 3) 2 2(3 3)=
E(3-3)
1 Ee Z - E,
0
where E (E1,... ,n)) consist of i..d copies of defined in Example follows that
"
1 Bo 2zz (3 - 30)1 Ee z(Z 2) T z E
1 T
trace { (EEE' T Z(z T 2) -0'7 ' }z
(8)
trace (EZ(Z) -)-1z
z dz,
where dz is the dimension of Z. In the following, we show
Σ
-1) -
B0)1 2 = o(1.
(9)
Combining this together with 7 and 8 yields
Σ Emin 1, o
217 (3 [0]1²] 2 2zz o(1) 2 o(1),,
and hence Σ Edfy PXizz=,, Φ (-1Z)) 2 1 - (1). The proof is hence completed.
remains show or equivalently,
Eno, 1 (8 10)1 27 (3 30)1 = =(1)
We have already shown that Eno, 212 (3 30) = =dz. By dominated convergence theorem, suffices to show
no 21 (3 36) 11100110000000000000 21 (3 30)12 N 1) (1.)
By definition, it suffices to show
Pr o 2 | (3 30) N 1) + 0.
However, this is immediate see by Markov's 's inequality as
Ea 22, (@ 30) = dz + 0.
n
This completes the proof of Proposition 1
15

---

Under review as a conference paper at ICLR 2021
F.2 PROOF OF THEOREM 2
We begin by providing an upper bound for the function classes Hi and H2.. Recall that 01,6 and 02.6 are generated according to (0, 2Ta, /d1, and (0,, 2T2./d2) respectively. Since IH1 and Hi2 are classes of neural networks with single hidden layer and finitely many hidden nodes. Both dj and d2 are finite. All the elements {vd101.6, 1..... and (vd2026.1 are ii.d. standard normal. For any standard normal variable Z, we have for any t 2 that
Pr( > 1) 2 <(z)dz S S. =0(2)d2 = 2exp
f (
(-
where (-) denotes the standard normal density function. Setting /2 log n for some constant
O, we obtain Pr[[Z] t) S n="". It follows from Bonferroni' inequality that
Pr (f b.j
+n max Vd222.001 > ( h.
BdBBddgn "2
(10)
Since d1 and dg are finite and = (n)) for some constant e > 0, the right-hand-side converges zero with proper as long as > Consequently, all the weights can be uniformly bounded by O(vlog Since the sigmoid function is set to the activation function, the function classes are
bounded by O(vlog nz) with probability tending to Without loss of generality, we assume this event holds throughout the proofs of Theorems
Define test statistic
T max b1,
1
Σ
n
M Σ m=1
hi (x((m)))
1 M
Σ h2 (X, (m())))) m=1
b2.b2
M
where the 0b, d2 constructed based on (}((m) Im and (8((m)) Im instead of (x(m))) (Y,(T) (m)) Jm. Thus, it suffices to show - = Op(n (2n) and - T 2n)).
mm and
Consider the difference T For any sequences {an {bnJas have
max Ja,l max [b,.1| I max la, bnl-
(11)
Consequently, the difference [ T is upper bounded by 1T + 12 + 13 where
T max T b,,b2
n
(h1,0, (x/m)) - 11, (X (m)))
h2,b,
1
h2 (m(1))
by
n
M
M
M M
M
1 = max T bb by
11, (X+)
11.m (m!)
(12 (X((m))
122 (p((m)))
M
miml
VIX
M
13 max by b,
(1i
h his (X(m)))
(12.00 (Y,(!m)
122 (m)))
M
M
m=1
m=m
By definition, we have min o bb
2 Ve for some constant
> 0. It suffices to show I5
Op(n 2x l) for j 1,2,3 where
H max
(1r,0 (x.(((((()) h10, (x,,,(( (X(m)))
(o,,,(Y)) M
Σ he (X:(m))
b1.
T1
M
m=1
m=1
M
I max
h1 (X₂)
M Σ
h1, (x((m))
(he, (X(1m))
h2 (8(
b1
M
M
M Σ
(hr (x(m)) - h1 (8(m)))
The (V (x((m)) - 12. (Y, (V((m))))
T5 max
b, by
n
M
M
m=
m=l
16

---

Under review as a conference paper at ICLR 2021
The number of folds L is finite, as such, it suffices show 14 Op(mi 2* log for j 1,2,3 and l 1, L. where
M
1 max bj b2
Σ (hi
(x(mi) h1 (8.
Σ
h2 (X/m)))
M
(EI
M
1
(x.)
Σ
hi (x((m))
1
(22, (X, (m((
122 (X,(m)))
max
by bg
M
iEZU
M
3
10)
Σ (11 (X!m))
11 (X(m)))
Σ
t22 (V/(1))
h22 (8 }}
b1
M
M
The rest of the proof is divided into four steps. the first three steps, we show 7!0)
Op(n 2x l) for = 1, the last step, we show T T = Op(n 22x logn).
Step Without loss of generality, suppose functions in H2, and H2 are bounded by logn n in absolute values. By Bernstein's 's inequality, we have
M h \xx -M0(_MMEE)))))212 2t S 2exp 2(M log n + Iy
{-
12
}
Pr
/ log n/3)
m==1
for any b and i. Set I. = 3 2)M log where the constant e is defined in the statement of Theorem For sufficiently large n, we have tvlog n/3 s M log n/2. It follows that
M
Pr
2 /3(c+ M)M logn S no+22
m==1
By Bonferroni's inequality, we obtain
M
Pr
max bE( B) (E(1, m)
max
Σ
hx. - (X,) 2 233(c+2) log
m=1
M hx/(X//m)) - MM((h,, 2 /3(c 2)M logn
2Bn
<
Bn max
61 .B (E .n}
max
Pr
n
m=1
Under the condition = O(n),, we obtain with probability 0(n that
M
max be (1,m B) .n)
max
Σ
hx.(( MF((h1,, [4) S 50(1)n 1/2 log n,
(12)
as M is proportional to and (11) denotes some positive constant.
Similarly, we can show
max bE( B} (E2
Σ
(X(())) M / 1110(1) (dr)
(11))/logn,,
with probability (n as well. Combining this together with 12
we obtain with probability
O(n that
Σ - (1) M / h1o0))((112=2
max be (e, (EZ(4)
((0)
x| (dz)}
(13)
O(L))
Conditional on Z, the expectation of h2 (Y;) - -1 SM -M h2 (X:(!)) equals zero. Under
the null hypothesis, the expectation of M-1 EM (X h h,,,(x (X(m))) (12,,,,
17

---

Under review as a conference paper at ICLR 2021
M- M-'Em= b2 (3) equals zero well. Applying Bernstein's 's inequality again, we sim- ilarly show with probability tending to that 1/ can be upper bounded by O(1)(on 1og n n log" n),
(14)
where ((1) denotes some positive constant and
M
2
0² max E b1.
Σ
(h1,6, (X,,,((
h1,b, (X/ (h2,,,
1
h2 (y.(m)))
M
M
m=1
< max E
1 M
Σ (x,
h1,0 bi (X, (m))) logn.
m121
Let A denote the event in
13
The last term on the second line can be bounded from above by
M
max E
(1 (x.( h1,0, (X, (x((m) }}
I(AA) log
(15)
b
M
m=1
M
Σ (hr,b, (x(m)) - h1 (x(m))
+ max E
I(AA9) log
(16)
bin
M
m=1
Since M is proportional to n, by
[T). the term on the first line of [15 upper bounded by
2
O(1)
log"
max be (1..r. igz(t)
E
b1 (dx) Pxiz=2 (dx)
log n.
By the boundedness of the function class HI,, can be further bounded from above by
0(1) {" n log" + E X|2: Px(z) log" n
(17)
Under the current conditions, the above quantity is of the order O(n 2x log n). Consequently, [5) the order O(n -2n log" n)).
Note that the event A occurs with probability at least 1 - ((n By the boundedness of the function class HH, 16 is of the order On log" n)).
To summarize, have further that a² is of the order O(n log n). This implies z(") can be bounded from above by O0n-1/(2-0 10///~/og55/// log n). This yields Z(") = Op 2x log n).
Step 2. Step 2 can be proven in similar manner Step 1 and is thus omitted.
Step 3. Under Hg. the expectation of
M
11,0, (h1,,, (X(m))) 121.)-11100000 11,0, (X((m))
M Σ
(12,,2 (Y, (Y/(m)) 122,22 (P((m))
I iEZO)
equals
Ef h1, (P1) (dz) Pxiz(dx)) /
E
x2z(pxz220 (dz)
hr2 (u) - Priz(dy)).
Similar to 17) its absolute value can be upper bounded by
ple) Edrv x|2=2, Pxiz)dTv' rIz=2, Priz logn.
((t)
Under the given conditions, it follows from Cauchy-Schwarz Schwarz inequality that
Edrv (p(e) (P Pxiz]drv
((0) 1izzzz, Priz]
Ed ( x| Pxiz} Edfv(Pi! r Prizi 1010=0(m -2n).
< 2
18

---

Under review as a conference paper at ICLR 2021
This yields that
/ h1. (x)
(dx) Pxiz(de)) / (v) (w) riz(ddy) Priz(du)3 O(n 2x log n).
max B
by bb2
Using similar arguments in Step 1, we can show that
(0 max B h1.,, (iz (dx) - Px]z(dr)) / h2. (w) - Priz(du)) = Op(n 2x log n).
Thus, obtain 15 (n log n).. This completes the proof of Step
Step 4. Denote by ob, the variance estimator with (X((m) Im and { (m)) Im replaced by {X; {x!mm) Fm
and {X, (m)) 1m. Using 11 the difference between 7 and T is upper bounded by
M
max 09,02 b1.b2
h1,6 (X.)
1 M
Σ h1, (x/m))
h2202 (Y:)
Σ
12 (y/m))
Under Ho, similar to 13 we can show
M Σ
M
max ,22
n
h1,,
hid
(x. (ma))
h2 hoao0(Y))) (Yi) - M m=1
1
Σ
h2 (X, (ma))
M m=1
Op(n
1/2 3/2 n).
10 Tbr = Op(n for some constant 0. Since both o by and ōb bz are well-bounded away from zero. suffices to show maxs, by 10 bb2 01.02 = Op(n F).
To show 1T - T Op 2x), it suffices to show maxb, d2
Using similar arguments in Steps and 3, we can show
max 22 b1
Var [111,6, -X) (X)2 - B(h2,a (X))2) = OO(nn
b1.b2
and
max bi.bg
b1.
Var [h1,0, (X) (x)2,, - -,02,,,, (Y)))))
=Op(n
n
This completes the proof of Theorem 2
F.3
PROOF OF THEBRRMM 3
In the proof of Theorem we have shown T = Op(n" 22x log n). Using similar arguments in
Step 4 of the proof of Theorem 2 we can show T T
Op(n 2x llggn where
M
M
T
max b1.ba
b2
h1r (Xi)
M
Σ
hLd (xf(m)))
(X)
M
1
Σ
h2 (Y (m))
where
Th max
Var
[he (X) - E(hi, (X)[2) (h2.022)))
E(h2,02 (2)2H)
n1
By 12 using similar arguments in in proof of Theorem we can show T T
Op(n 2x logn) where
1 M5
Σ h2 (x.(m))
T
max
of by
n
h1, (X,) - E(h1,, (x,)
h22 (Yi) (2,) M
by
m=l
19

---

Under review as a conference paper at ICLR 2021
Similarly, show T -To Op(n 2x log where
To max b2.02
by h2
1M1,,, (X+) E(hi1,b, (X, ZzH (22 (YG) B{h2,02 (Yi)
To summarize, we have shown T - To = -p(ri 2x log ni). Since we require x > 1/4, we obtain
Vn(T To) Op (log 1/2 n).
(18)
Define B x B matrix Eg whose {bi + B(b² 1), bg B(bs 1) entry is given by
Cov bbi 111,0, (X+) B( (x.) 122 (Y;) B2 (Y:) [[
by [ (X2) E( (x,) 21 (2.) B(hg, (Y2)
In the following, show sup (Pr(VTTo S ((Ho) - (O, Ea) - = o
(19)
When is finite, this is implied by the classical weak convergence results. When B diverges with n, we require B (n²) for some constant > 0. By the definition of ob by. the variance of
0b [h1,b, (X,) B(h,,6, b, (Z.) [h2,,, (Y:) E(h2, (Y:)
is bounded from above by (n 1)/n. Moreover, combining the boundedness assumption on the function spaces HII and H2 together with the definition of Thi.b2 yields that
{ oo [ (X,) B(h1,,, (x)) [b,,,, (Y:) B(h2,6, (Y.) [z)] b1,b2 (1..... B)
uniformly bounded away from infinity. Similar Corollary 4.1 of Chernozhukov et (2014). we can show that 19 holds. Suppose B = 1. Then N (0, Eo) is a single normal variable. This implies that
.b n
1/2 1h (X,) - Efhi,
(X.) (X,1,, (X,) - B(h2, (Y))
is asymptotically normal with zero mean.
Combining [99 together with [8] yields
Pr(vnT S t/7o)) 2 Pr(I)N(O,E I - 6 log 1/2 n) o(1),
(20)
Pr( < H/74)) < (r En) St + en log 1/2 n) + o(1),
for any sufficiently small <o 0, where the little-o terms are uniform in t.
Using similar arguments in Step 4 and Step 5 of the proof of Theorem 2 we can show that -| Eolle Solloo.ce = (n ") for some constant ē > Using similar arguments for (20) and also Lemma 3.1 of Chemoozuukov al. (2015), we have that Pr(vnT < Ho)) 2 (O, E) -t-26ple log 1/2 n|(E - o(1),
Pr(vni' < {(H)) < (O, 1) St+ 2ep log 1/2 n [ + o(),
for any sufficiently small 60 > 0. Since the littl--o terms are uniform in f E R, we obtain
sup |Pr(v HoH) (r( 10 - +[$3] S o(1)
sup (T )e S + 2e 1/2 n|L) Pr([I E) St 2elog 10/lggg1/// 1/2 n ΕΣ}-
The term on the second line can be bounded by 0(1)c 1/2 B log - 1/2 n where O(1) denotes some positive constant, by Theorem of [Chernozuukov et al. (0117). Since B = O(n"),
1/2 B log 1/22, n = 0(1). As e grows to zero, this term becomes negligible. Consequently. obtain
sup PT S (1H) Pr Eto < イ S o(1).
As such, the distribution of our test statistic can be well-approximated by that of the bootstrap samples. This completes the proof of Theorem 3
20

---

Under review as a conference paper at ICLR 2021
F.4 PROOF OF THEOREM 4
According to the universal approximation theorem (Barron 1993). neural networks with a single hidden layer and the sigmoid activation function are universal approximators. Under Hi. there exist Two neural networks functions f(X) and g(Y), with single hidden layer and the sigmoid activation function, such that
B[f(x) ( ZHI)(Y) E(g(Y) +
Note that (g) can be represented by linear combinations of functions in НH] (Hll). It implies that there exists some constant e* > such that GCM' (X), hY)) h2 E Hl.. Let 0r and 0ī be the corresponding parameters such that h = h1,,; h2 h2205
for some h IHI and
We next show that GCM" (h1,0, (X), hz (Y)) is a Lipschitz continuous function of (01, 02). Note that h1, (X) and h2 (Y) are Lipschitz continuous functions of 0, and 02. respectively, For any 01,1, 01, E TRd,, 02,1, 0, E Rdd, we have
GGCM (hi, (X), h,, (Y)) - GCM (h1, (X), /2 (Y))
S [ 1(h (X))2) (x) + E(h2, (X)) ( [2,, B(h2, (V)
(21)
+(E(),,((X 2 (x)) B(h2,1 (V)] h2,2(Y) B( (Y)
(22)
Since H2 is bounded function class, RHS of 21 is bounded from above by
0 (x)) - (X)]z) h1,2(X) E(h22 logn,
where O(1) denotes some positive constant. further bounded from above by
By Jensen's inequality, the above quantity can be
O(1)BI(I,(()) (X))(2 logn S LI|O1, - -1,2112 /lallvlogn,
for some constant L > O. Using similar arguments, can show RHS of 22 is bounded from above by LI(02,1 - 02.2112. To summarize, we have shown that
|GCM' (h1,e, (X), h2,8, (Y)) GCM GCM h2,,,
S I((10,,1 - 01,2112 + |02,1 02.2 22) log n.
As such, for any sufficiently small > 0, there exists neighborhood. N = ((),00): |10 - 0j1l2 S 5 log 1/2 11/2 n for constant 8 around (0T,03 such that GCM" (X), h2 (Y)) for any (0,, 02) that belongs to this neighborhood.
Since
multivariate normal generate (01,,, 02,1) and the dimensions di and d2 are finite, the probability that (01,6, 02.6) belongs this neighborhood is strictly greater than Olog n) for some constant C₁ > 0. Since B = conz", the probability that at least one pair of parameters (01,s, 02,b2) belongs this neighborhood approaches one. Consequently, we have
max GCM (h1,0, (X), (Y)) e, b1,02
with probability tending to 1.
show that IT maxs, GCM* (h1,,, (X), (Y)) = (1) and T, = (1(. Consequently, both probabili- ties Pr(T < e/2) and Pr(T; 2 e/2) converge zero. As such, the probability that the p-value is greater than a is bounded by the probability that Pr(T < <//2), and hence converges to zero. This completes the proof of Theorem
Using similar arguments in the proof of Theorems and
21


Reviews:
[{"review_id": "r__JZ4zYnEZ", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "- Overview\n\nThis paper develops a test for conditional independence using GAN.\nConditional independence is one of the well-known problems, and calculating the conditional distribution of a random variable is a challenge.\nThe authors pointed out in Proposition 1 that the existing method, named GCIT, cannot avoid a non-negligible approximation bias.\nThe authors newly developed test combines the conditional distribution with GAN and regression-based and MMD-based tests to construct a valid test under weaker conditional requirements than the previous method.\n\n- Comments.\n\nThe paper points out some important issues with existing research.\nHowever, there are a few things I don't understand.\n\nThe experiments show that the GCIT appear to have no power at all, but the experiments of the original paper (Bellot and van der Schaar (2019)) report that GCIT has sufficient power.\nWhere does this discrepancy come from?\nThe submitted paper says that it uses a similar setting to Bellot and van der Schaar (2019), so I would like to know why the results are so different.\n\nThe theoretical advantages and their relationship to the experiment are less clear.\nThe paper states that the conditions required for GCIT are unsatisfied in general, but does this show up in the experimental results?\nThe Type I errors appear to be a bit more or less dominant, but not by much.\nCould that be the reason for the poor performance of GCIT in the analysis for power? \nIf so, then it should be clear why there is a significant difference with Bellot and van der Schaar (2019), as discussed above.\n\nI didn't understand why the conditions by the submitted paper are weaker.\nI don't doubt the accuracy, but if the technical points are not properly explained, it is not kind for readers.\nI want a clear explanation that can weaken the conditions.\n\nHow is the computational time of the proposed method?\nTo be a practical method, the computation time should be short, but if we use the GAN, it would be a big cost.\nOf course, this is true for all studies using GANs, not only this paper, but also for all studies using GANs, but it is important for practical purposes, so please let me know.", "extra_scores": {}, "extra_text": {}}, {"review_id": "gcRKQSxAcqp", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nThe paper proposed a novel simulation based testing procedure for conditional independence: X\\perp Y | Z. The testing procedure incorporate the techniques of GAN, which is especially useful for dealing with high-dimensional data. The testing procedure first learn the generative adversarial network that is able to simulate the conditional distribution of X|Z and Y|Z and check a particular kernel-based independence criteria presented in Eq.(4). Instead of kernel-based method that takes the supreme over a class of RKHS functions, the proposed testing procedure searches the maximum \"discrepancy\" over a class of neural network function by simulation. Empirical results show a well controlled type-I error and better test power performances compare to existing methods and a cancer data application is discussed.\n\nThis is an interesting paper combining kernel testing and GAN setting which can potentially broaden the scope of kernel-based tests. In addition to the contributions, there are some unclear parts in presentation and concerns on the proposed schemes.\n\n1. The kernel-based test, e.g. MMD relies on the positive definiteness of the RKHS. The notion of characteristic kernel ensures that the null hypothesis hold iff the test statistics is 0. In this work, do you or how do you ensure the positive definiteness? By taking the max over an absolute value in Eq.(5) does ensure non-negativity of the test statistics, but not yet the characteristic notion. Instead of RKHS kernel, it may end up being a Krein space kernel, i.e. RKKS. This may reduce the test power of the statistics in particular scenario, but it is unclear.\n\n2. From the flow chart in Fig.2, the two generators and discriminators are trained separately. How does the notion of \"doubly\" comes in. If the training is interactive between Y|Z and X|Z network, how is it done? This part is not entirely clear.\n\n3. Is the notion of robustness come from the learned conditional density (or generator) is robust against noisy samples or some form of adversarial attack, or some other notion? Not entirely clear just from Thereom 1, instead, it sound more like asymptotic property. \n\n4. The number of \"bootstrap\" functions B need to go to infinity, maybe it will be better to emphasize that.\n\n\n5. Building up from the GCM type of statistic, instead of MMD, it may be better to refer Hilbert Space Independence Criterion (HSIC) instead of MMD.\nAppendix D derivation, despite mathematically correct,  need more steps: the add-in and subtract h_1(X) times conditional expectation of h_2(Y)|Z term, otherwise it does not supply additional information to the main text.\n\n6. Experiment findings: in top panel of Fig.1, there is a spike on dim=150 for DL-CIT, is there any intuition why this happens? And KCIT and CCIT papers both claimed their controlled type-I error, what is the reason for this scenario that they fail?\n\nThanks for the presentation.", "extra_scores": {}, "extra_text": {}}, {"review_id": "fjbRTyURCE0", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose a non-parametric conditional independence test that approximates distances in a Hilbert space of functions using a generative approach, both evaluating conditional expectations using samples from GANs and evaluating a supremum over a set of functions previously generated at random. The test incorporates benefits from different lines of research and is demonstrated to outperform alternatives in well-known benchmarks. I think this study is interesting and serves as a good example of the synergies that may be achieved by combining powerful function approximators and strong statistical arguments.\n\nMy biggest concern is that conditional independence testing is an unsupervised problem, there is therefore little scope to test for the goodness of fit of hyperparameter choices or the accuracy of approximations. The proposed approach has plenty of user-defined parameters yet the sensitivity of performance to different choices is not investigated nor is there a discussion of sensible values to be recommended in practice. \n\nSome specific questions on this thread are as follows. \n- Appendix A says: \"The performance of GANs is largely affected by the regularization parameter and the number of Sinkhorn iterations R.\", should we expect performance to vary a lot in practice? \n- Appendix C tests the goodness of fit of the GAN approximation by comparing observed and estimated distributions p(y|x). Why should this be a good indication that GANs approximate well p(y|z) or p(x|z)?\n\nMost consistency guarantees are asymptotic in nature. With increasing conditioning set size, the fact that complex conditional distributions need to be approximated and that separate sets of samples are needed to train and compute the test statistic, I worry that a very large number of samples will be needed to achieve good performance. Would experiments as a function of sample size be possible to include in the paper?\n\n\nMinor comments:\n- The test itself is quite involved, with many moving parts that are described over various pages in the paper. I would recommend to have a higher-level summary of the approach perhaps using Figure 2 in the Appendix earlier in the paper.\n- A single data generating mechanism is used for performance comparisons. I think different multiple different choices here would be needed to test different aspects of the model.", "extra_scores": {}, "extra_text": {}}, {"review_id": "ACAqZkstpCr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper considers the problem of conditional independence testing, especially when the variables are high-dimensional. The authors proposed a double GAN based algorithm. Two GANs are designed to learn the conditional probability distributions P_{X|Z} and P_{Y|Z}, then used to generate samples to compute the test statistic. It is proved that the error of the test statistic is O_p(n^{-2k} \\log n) when the total variation error of the GANS is O(n^{-k}). So to ensure that the test statistic converges, only o(\\log^{-1/2} n) rate is required for the total variation error of the GANs.\n\nThe result of this paper is quite strong. Compared to the paper of Bellot & van der Schaar (2019)  which requires the TV error of GAN to be o(n^{-1/2}), this paper significantly reduce the requirement to o(\\log^{-1/2} n).  I did not check the full proof.\n\nQuestions to the authors: In the description of Bellot & van der Schaar(2019), it seems that the data used for training the GAN and the data used for computing testing statistic are shared. However, in the proposed algorithm (Algorithm 1), the data are split into L blocks where GANs are trained by L-1 blocks of data and test statistics are computed by the other. Is this a critical reason for the improvement of convergence rate? If so, how will the convergence rate be if we apply data splitting to Bellot & van der Schaar (2019)? If not, what is the key reason for the improvement of convergence rate, double-GAN or the randomly generated h functions? ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper discusses the conditional independence test using GAN.   In the same way as GCIT (Bellot & van der Schaar, 2019), they realize sampling under the null hypothesis by generating sample from P(X|Z) approximately with GAN.  They propose to use a test statistic defined by the maximum of generalized covariance measures (GCM) over random neural networks.  They theoretically discuss the advantage of GCM and show the asymptotic results of the proposed test statistic, which demonstrates improved justification over GCIT.  Experimental results show favorable performances over existing conditional independence tests. 

The proposed method gives an advance in the methodology of conditional independence tests for continuous domain, which is an important but difficult problem because of the difficulty of obtaining the null distribution.  In the line of Bellot & van der Shcaar (2019), they solve it using the strong conditional sampling ability of GAN, which is an important research area. The theoretical analysis and experimental results are also making good contributions.  

However, there are some weakness in the proposed method and comparison with existing methods. First, as R4 points out, there are many hyperparameters in the proposed method, and their choice is not easy.  While the authors addressed some aspects of this issue in their rebuttal and revision, it is still unclear how to justify the choice of B, the functions h_j, and the neural networks of GAN, which should potentially have significant influence on the test performance.  Second, the comparison with Bellot and van der Schaar (2019) is not very clear.  In the paper, the GCIT has been used with the distance correlation, which is known to be an instance of HSIC (MMD) with a specific choice of positive definite kernel (Sejdinovic et al 2013).  The HSIC can be formulated as the maximum of generalized covariance measures over the unit ball of the RKHS.  Thus, the difference of GCIT with distance correlation and the proposed methods are essentially the difference of the function classes for the maximum.  On the other hand, the experimental results show significant difference in the test performance.  I think more elaborate and careful comparison is needed for these two methods. 

Overall, the paper is a good contribution on the topic.  However, the evaluation of the reviewers is not high enough to justify the acceptance in the high competition of ICLR.  I encourage the authors complete their work by reflecting reviewers’ comments and submit this work to another conference or journal. 

Reference:
Sejdinovic, D., Sriperumbudur, B., Gretton, A., & Fukumizu, K. (2013). Equivalence of distance-based and RKHS-based statistics in hypothesis testing. Annals of Statistics, 41(5), 2263–2291.

Decision: Reject