Title: On the Hardness of Conditional Independence Testing In Practice
Venue: NeurIPS 2025
Accepted: yes
Keywords: conditional independence

statistical testing

kernel methods
Abstract: Tests of conditional independence (CI) underpin a number of important problems in machine learning and statistics, from causal discovery to evaluation of predictor fairness and out-of-distribution robustness. Shah and Peters (2020) showed that, contrary to the unconditional case, no universally finite-sample valid test can ever achieve nontrivial power. While informative, this result (based on “hiding” dependence) does not seem to explain the frequent practical failures observed with popular CI tests. We investigate the Kernel-based Conditional Independence (KCI) test – of which we show the Generalized Covariance Measure underlying many recent tests is _nearly_ a special case – and identify the major factors underlying its practical behavior. We highlight the key role of errors in the conditional mean embedding estimate for the Type I error, while pointing out the importance of selecting an appropriate conditioning kernel (not recognized in previous work) as being necessary for good test power but also tending to inflate Type I error.


Peer reviews:
comments: SUMMARY:
This paper examines issues about conditional independence testing. It gives a good review of how this has been done over the ages and investigates kernel-based conditional independence as well as how to choose the best kernels and the associated pros and cons.

STRENGTHS AND WEAKNESSES:
This paper presents a detailed background for all the key methods used in rich statistical formulations. However the paper could benefit from performing more experiments to prove the points they are establishing.

QUESTIONS:
In line 170, where you define $t_n$, what is $q$?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 4
IMPACT: 4
ORIGINALITY: 3
DATE: 2025-06-27

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper provides a comprehensive theoretical analysis of kernel-based conditional independence (KCI) testing. The authors first introduce the KCI framework, and re-interpret it from the perspective of generalized covariance measures (GCM), in a distinct way from [1]. Then, the paper presents an analytical example to illustrate the behavior of the proposed KCI statistic both with and without regression errors. Several simulations were conducted to emphasize the importance of selecting an appropriate kernel of conditioning variables. The theoretical contributions of the paper mainly focus on establishing an error bound of SNR estimator (the KCI statistic divided by its variance), as well as the bounds on the type I error of the test in the presence of regression error. Overall, the authors provide a theoretically sound and carefully argued analysis of KCI, supported by a few simulations that validate their theoretical insights.\n\n[1] Wenjie Wang et al. (2025), Practical Kernel Learning for Kernel-based Conditional Independent Test, Openreview.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The analyses of the example in Section 4 are detailed and valuable.\n2. The work improves understanding of KCI testing, and provides a solid foundation for further theoretical improvements.\n3. The authors gave the bound of type one error which is missing from [1].\n\nWeaknesses:\n1. It would be beneficial if the authors could provide more extensive simulations to further examine the the proposed CIT and theories. It would be better if the results could be supported by real data.\n2. I suggest that the authors consider moving some of the important theoretical results in the appendix into the main text, to better highlight the contributions of the paper. Conversely, certain parts of the main text could be condensed, or even moved to the appendix, such as some part of Section 2, 3.\n\nQUESTIONS:\n1. I strongly recommend to fix the y-axis when plotting Figure 3 (a), (b). In addition, it is quite surprising that the type II errors of blue and green lines increase with a larger sample size, when $l_C^2 = 10^{-5}$. Moreover, the standard deviation of the type I error becomes noticeably larger with larger sample size. Could the authors provide further explanation for these phenomena?\n2. Could you explain where the $\\lambda = \\ell n^{-1/3}$ in line 874 comes from? This is crucial for the first term in the convergence rate of the SNR. And I noticed Liu et al. also chose a similar $\\lambda$ in their Th 6 [2].\n3. Could the authors extend their discussion of KCI to also RCIT[3] and SCIT [4]? Given that these methods are highly correlated to GCM and KCI.\n\n[2] Feng Liu et al. (2020), Learning Deep Kernels for Non-Parametric Two-Sample Tests, ICML.\n\n[3] Hao Zhang et al. (2017), Causal Discovery Using Regression-Based Conditional Independence Tests, AAAI.\n\n[4] Hao Zhang et al. (2022), Residual Similarity Based Conditional Independence Test and Its Application in Causal Discovery, AAAI.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nAiming to improve the understanding and failure modes of conditional independence (CI) tests, this work provides several novel insights into the hardness of CI testing in practice.\n The authors mainly analyse the Kernel-based conditional independence test (KCI), noting that results also relate closely to the (weighted) Generalized Covariance Measure (GCM) under their formulation of the test in Thm. 2.2. The main insight is that the estimation of conditional mean embeddings presents the main practical limitation of KCI. Specifically, the well-known hardness result by Shah and Peters (2020) does not hold for KCI when using true mean embeddings (Prop. 3.1) but its practical issues arise from the finite-sample estimates of the embeddings. The authors show that this can be somewhat alleviated through the kernel choice (Section 4) but also acknowledge that this comes at the expense of inflating Type I errors, i.e., false positive rejections of independence (Figure 2).\n\nSTRENGTHS AND WEAKNESSES:\nThe work contributes several novel insights that are likely of interest to the literature. This includes the reframing of KCI to more closely relate it to GCM (Thm. 2); the insight on practical failures of KCI being related to conditional mean embedding estimation which, according to the authors, is not yet acknowledged in existing work; the discussion of appropriate kernel choices (Section 4); and the exploration and cautionary remarks regarding Type I errors (Section 5). Especially in light of the importance of CI testing and widespread use of KCI, this makes this work of solid significance, even though the practical contribution is perhaps limited.\nThere are no apparent issues regarding quality, all claims have thorough justifications. While some results are inspired from previous insights, such as the strategy motivated in Prop. 4.1. following related settings, the contributions can all in all judged to be of sufficient originality. \nRegarding clarity, the writing is dense  but overall well structured around the main claims, and all insights are placed in the context of previous work. \n\n Overall, this work advances the current understanding of the behavior of KCI and can be judged an important contribution.\n\nQUESTIONS:\nMinor suggestions and questions include \n\n- If space allows, the density of the text could be reduced. For example, an explanation of Fig. 1 (besides ln. 271) seems to be missing in the text, and some paragraphs could be expanded such as lines 327-335 to explain the insights given in Appendix G.2, G.3.  \n- a clearer separation of previous insights compared to those that are novel contributions could be helpful, e.g. some of the existing works could be postponed to the Appendix or a related work section.\n\nLIMITATIONS:\nNo major unadressed limitations or societal impacts.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper examines the practical hardness of conditional independence testing with kernels. Specifically, Shah and Peters give a result that conditional independence testing is statistically hard, however their constructive proof gives little insight as to why common kernel based tests fail. In this work, the authors examine the failure of these tests, demonstrating that if we had exact knowledge of the conditional mean embedding the testing problem becomes feasible, constructing a test that is consistent against any fixed alternative. They then study how the choice of kernel impacts the estimation of the conditional mean embedding, due to its centrality to the testing problem.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- I find the work to be interesting and novel, it plugs in my opinion a needed gap between the hardness result of Shah and Peters and practical conditional independence testing.\n\n- The observation that access to the true conditional mean embedding leads to a consistent test is a good one and is in line with the cited model X work. \n\nWeaknesses:\n\n- Theorem 2.2 is not new or an extension of Daudin 1980. In fact (without the weighting term) this is the same as condition i) in Lemma 2.2 of [1]. Seeing this classic result described as novel leads me to be concerned about the validity of the rest of the work and the authors familiarity with the area. \n\n-  Whilst I do agree that the observation that the true CME are enough for a valid test is an interesting one, it does seem to me that this result is an example of (or very closely related to) the idea of goodness of fit tests with kernels [2]. Specifically, having access to both $\\mu_{A|C}$ and $\\mu_{B|C}$ gives us access to $\\mu_{A,B|C}$ under the null distribution. Given this, conditional independence testing could be framed as a goodness of fit test for if samples $(a_i,b_i,c_i)$ are consistent with $\\mu_{A,B|C}$. In this light a consistent test feels unsurprising due to results of consistency of goodness of fit tests. \n\nOverall whilst this paper seems interesting and is a valid contribution to the literature, my current feeling is that it needs a bit more work before publication as evidenced by the claims of novelty of theorem 2.2 and the incomplete limitations section. Having said this if the authors provide strong arguments I would consider changing my score.\n\nSmall note: I would consider changing the phrase \"It has a reputation, however, of doing a poor job at controlling Type I error (that is, it falsely identifies conditional dependence too often).\" and ideally fine a citation or include an experimental result justifying this as without a strong familiarity with the area this statement is hard to verify.\n\n[1] - Kernel-based Conditional Independence Test and Application in Causal Discovery. Kun Zhang, Jonas Peters, Dominik Janzing, Bernhard Schoelkopf.\n[2] - A Kernel Test of Goodness of Fit. A Kernel Test of Goodness of Fit.\n\nQUESTIONS:\n- Is Theorem 2.2 he same as condition i) in Lemma 2.2 of [1]?\n\n- Can the authors comment on similarity to goodness of fit tests?\n\n- What would the authors feel are the key limitations of the paper?\n\nLIMITATIONS:\nThe limitations are not discussed despite the fact that this box is checked in the checklist.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}