Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: - Overview

This paper develops a test for conditional independence using GAN.
Conditional independence is one of the well-known problems, and calculating the conditional distribution of a random variable is a challenge.
The authors pointed out in Proposition 1 that the existing method, named GCIT, cannot avoid a non-negligible approximation bias.
The authors newly developed test combines the conditional distribution with GAN and regression-based and MMD-based tests to construct a valid test under weaker conditional requirements than the previous method.

- Comments.

The paper points out some important issues with existing research.
However, there are a few things I don't understand.

The experiments show that the GCIT appear to have no power at all, but the experiments of the original paper (Bellot and van der Schaar (2019)) report that GCIT has sufficient power.
Where does this discrepancy come from?
The submitted paper says that it uses a similar setting to Bellot and van der Schaar (2019), so I would like to know why the results are so different.

The theoretical advantages and their relationship to the experiment are less clear.
The paper states that the conditions required for GCIT are unsatisfied in general, but does this show up in the experimental results?
The Type I errors appear to be a bit more or less dominant, but not by much.
Could that be the reason for the poor performance of GCIT in the analysis for power? 
If so, then it should be clear why there is a significant difference with Bellot and van der Schaar (2019), as discussed above.

I didn't understand why the conditions by the submitted paper are weaker.
I don't doubt the accuracy, but if the technical points are not properly explained, it is not kind for readers.
I want a clear explanation that can weaken the conditions.

How is the computational time of the proposed method?
To be a practical method, the computation time should be short, but if we use the GAN, it would be a big cost.
Of course, this is true for all studies using GANs, not only this paper, but also for all studies using GANs, but it is important for practical purposes, so please let me know.
review_title: Seems sound, but some unclear points.

final_score: rating: 5: Marginally below acceptance threshold
review_content: 
The paper proposed a novel simulation based testing procedure for conditional independence: X\perp Y | Z. The testing procedure incorporate the techniques of GAN, which is especially useful for dealing with high-dimensional data. The testing procedure first learn the generative adversarial network that is able to simulate the conditional distribution of X|Z and Y|Z and check a particular kernel-based independence criteria presented in Eq.(4). Instead of kernel-based method that takes the supreme over a class of RKHS functions, the proposed testing procedure searches the maximum "discrepancy" over a class of neural network function by simulation. Empirical results show a well controlled type-I error and better test power performances compare to existing methods and a cancer data application is discussed.

This is an interesting paper combining kernel testing and GAN setting which can potentially broaden the scope of kernel-based tests. In addition to the contributions, there are some unclear parts in presentation and concerns on the proposed schemes.

1. The kernel-based test, e.g. MMD relies on the positive definiteness of the RKHS. The notion of characteristic kernel ensures that the null hypothesis hold iff the test statistics is 0. In this work, do you or how do you ensure the positive definiteness? By taking the max over an absolute value in Eq.(5) does ensure non-negativity of the test statistics, but not yet the characteristic notion. Instead of RKHS kernel, it may end up being a Krein space kernel, i.e. RKKS. This may reduce the test power of the statistics in particular scenario, but it is unclear.

2. From the flow chart in Fig.2, the two generators and discriminators are trained separately. How does the notion of "doubly" comes in. If the training is interactive between Y|Z and X|Z network, how is it done? This part is not entirely clear.

3. Is the notion of robustness come from the learned conditional density (or generator) is robust against noisy samples or some form of adversarial attack, or some other notion? Not entirely clear just from Thereom 1, instead, it sound more like asymptotic property. 

4. The number of "bootstrap" functions B need to go to infinity, maybe it will be better to emphasize that.


5. Building up from the GCM type of statistic, instead of MMD, it may be better to refer Hilbert Space Independence Criterion (HSIC) instead of MMD.
Appendix D derivation, despite mathematically correct,  need more steps: the add-in and subtract h_1(X) times conditional expectation of h_2(Y)|Z term, otherwise it does not supply additional information to the main text.

6. Experiment findings: in top panel of Fig.1, there is a spike on dim=150 for DL-CIT, is there any intuition why this happens? And KCIT and CCIT papers both claimed their controlled type-I error, what is the reason for this scenario that they fail?

Thanks for the presentation.
review_title: Novel approach combine GAN technique with Conditional Independence Testing; still room to improve

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper considers the problem of conditional independence testing, especially when the variables are high-dimensional. The authors proposed a double GAN based algorithm. Two GANs are designed to learn the conditional probability distributions P_{X|Z} and P_{Y|Z}, then used to generate samples to compute the test statistic. It is proved that the error of the test statistic is O_p(n^{-2k} \log n) when the total variation error of the GANS is O(n^{-k}). So to ensure that the test statistic converges, only o(\log^{-1/2} n) rate is required for the total variation error of the GANs.

The result of this paper is quite strong. Compared to the paper of Bellot & van der Schaar (2019)  which requires the TV error of GAN to be o(n^{-1/2}), this paper significantly reduce the requirement to o(\log^{-1/2} n).  I did not check the full proof.

Questions to the authors: In the description of Bellot & van der Schaar(2019), it seems that the data used for training the GAN and the data used for computing testing statistic are shared. However, in the proposed algorithm (Algorithm 1), the data are split into L blocks where GANs are trained by L-1 blocks of data and test statistics are computed by the other. Is this a critical reason for the improvement of convergence rate? If so, how will the convergence rate be if we apply data splitting to Bellot & van der Schaar (2019)? If not, what is the key reason for the improvement of convergence rate, double-GAN or the randomly generated h functions? 
review_title: Official Blind Review #1

final_score: rating: 6: Marginally above acceptance threshold
review_content: The authors propose a non-parametric conditional independence test that approximates distances in a Hilbert space of functions using a generative approach, both evaluating conditional expectations using samples from GANs and evaluating a supremum over a set of functions previously generated at random. The test incorporates benefits from different lines of research and is demonstrated to outperform alternatives in well-known benchmarks. I think this study is interesting and serves as a good example of the synergies that may be achieved by combining powerful function approximators and strong statistical arguments.

My biggest concern is that conditional independence testing is an unsupervised problem, there is therefore little scope to test for the goodness of fit of hyperparameter choices or the accuracy of approximations. The proposed approach has plenty of user-defined parameters yet the sensitivity of performance to different choices is not investigated nor is there a discussion of sensible values to be recommended in practice. 

Some specific questions on this thread are as follows. 
- Appendix A says: "The performance of GANs is largely affected by the regularization parameter and the number of Sinkhorn iterations R.", should we expect performance to vary a lot in practice? 
- Appendix C tests the goodness of fit of the GAN approximation by comparing observed and estimated distributions p(y|x). Why should this be a good indication that GANs approximate well p(y|z) or p(x|z)?

Most consistency guarantees are asymptotic in nature. With increasing conditioning set size, the fact that complex conditional distributions need to be approximated and that separate sets of samples are needed to train and compute the test statistic, I worry that a very large number of samples will be needed to achieve good performance. Would experiments as a function of sample size be possible to include in the paper?


Minor comments:
- The test itself is quite involved, with many moving parts that are described over various pages in the paper. I would recommend to have a higher-level summary of the approach perhaps using Figure 2 in the Appendix earlier in the paper.
- A single data generating mechanism is used for performance comparisons. I think different multiple different choices here would be needed to test different aspects of the model.
review_title: Sophisticated conditional independence test with well-studied theoretical guarantees.

Meta-review: This paper discusses the conditional independence test using GAN.   In the same way as GCIT (Bellot & van der Schaar, 2019), they realize sampling under the null hypothesis by generating sample from P(X|Z) approximately with GAN.  They propose to use a test statistic defined by the maximum of generalized covariance measures (GCM) over random neural networks.  They theoretically discuss the advantage of GCM and show the asymptotic results of the proposed test statistic, which demonstrates improved justification over GCIT.  Experimental results show favorable performances over existing conditional independence tests. 

The proposed method gives an advance in the methodology of conditional independence tests for continuous domain, which is an important but difficult problem because of the difficulty of obtaining the null distribution.  In the line of Bellot & van der Shcaar (2019), they solve it using the strong conditional sampling ability of GAN, which is an important research area. The theoretical analysis and experimental results are also making good contributions.  

However, there are some weakness in the proposed method and comparison with existing methods. First, as R4 points out, there are many hyperparameters in the proposed method, and their choice is not easy.  While the authors addressed some aspects of this issue in their rebuttal and revision, it is still unclear how to justify the choice of B, the functions h_j, and the neural networks of GAN, which should potentially have significant influence on the test performance.  Second, the comparison with Bellot and van der Schaar (2019) is not very clear.  In the paper, the GCIT has been used with the distance correlation, which is known to be an instance of HSIC (MMD) with a specific choice of positive definite kernel (Sejdinovic et al 2013).  The HSIC can be formulated as the maximum of generalized covariance measures over the unit ball of the RKHS.  Thus, the difference of GCIT with distance correlation and the proposed methods are essentially the difference of the function classes for the maximum.  On the other hand, the experimental results show significant difference in the test performance.  I think more elaborate and careful comparison is needed for these two methods. 

Overall, the paper is a good contribution on the topic.  However, the evaluation of the reviewers is not high enough to justify the acceptance in the high competition of ICLR.  I encourage the authors complete their work by reflecting reviewers’ comments and submit this work to another conference or journal. 

Reference:
Sejdinovic, D., Sriperumbudur, B., Gretton, A., & Fukumizu, K. (2013). Equivalence of distance-based and RKHS-based statistics in hypothesis testing. Annals of Statistics, 41(5), 2263–2291.

Decision: Reject