Title: Conditional independence testing under misspecified inductive biases
Venue: NeurIPS 2023
Accepted: yes
Keywords: conditional independence

hypothesis testing

misspecification
Abstract: Conditional independence (CI) testing is a fundamental and challenging task in modern statistics and machine learning. Many modern methods for CI testing rely on powerful supervised learning methods to learn regression functions or Bayes predictors as an intermediate step; we refer to this class of tests as regression-based tests. Although these methods are guaranteed to control Type-I error when the supervised learning methods accurately estimate the regression functions or Bayes predictors of interest, their behavior is less understood when they fail due to misspecified inductive biases; in other words, when the employed models are not flexible enough or when the training algorithm does not induce the desired predictors. Then, we study the performance of regression-based CI tests under misspecified inductive biases. Namely, we propose new approximations or upper bounds for the testing errors of three regression-based tests that depend on misspecification errors. Moreover, we introduce the Rao-Blackwellized Predictor Test (RBPT), a regression-based CI test robust against misspecified inductive biases. Finally, we conduct experiments with artificial and real data, showcasing the usefulness of our theory and methods.

Conditional independence testing under misspecified inductive biases

Abstract

Conditional independence (CI) testing is a fundamental and challenging task in modern statistics and machine learning. Many modern methods for CI testing rely on powerful supervised learning methods to learn regression functions or Bayes predictors as an intermediate step; we refer to this class of tests as regressionbased tests. Although these methods are guaranteed to control Type-I error when the supervised learning methods accurately estimate the regression functions or Bayes predictors of interest, their behavior is less understood when they fail due to misspecified inductive biases; in other words, when the employed models are not flexible enough or when the training algorithm does not induce the desired predictors. Then, we study the performance of regression-based CI tests under misspecified inductive biases. Namely, we propose new approximations or upper bounds for the testing errors of three regression-based tests that depend on misspecification errors. Moreover, we introduce the Rao-Blackwellized Predictor Test (RBPT), a regression-based CI test robust against misspecified inductive biases. Finally, we conduct experiments with artificial and real data, showcasing the usefulness of our theory and methods.

1 Introduction

Conditional independence (CI) testing is fundamental in modern statistics and machine learning (ML). Its use has become widespread in several different areas, from (i) causal discovery [12, 24, 34, 11] and (ii) algorithmic fairness [28], to (iii) feature selection/importance [5, 37] and (iv) transfer learning [25]. Due to its growing relevance across different sub-fields of statistics and ML, new testing methods with different natures, from regression to simulation-based tests, are often introduced.

Regression-based CI tests, i.e. , tests based on supervised learning methods, have become especially attractive in the past years due to (i) significant advances in supervised learning techniques, (ii) their suitability for high-dimensional problems, and (iii) their simplicity and easy application. However, regression-based tests usually depend on the assumption that we can accurately approximate the regression functions or Bayes predictors of interest, which is hardly true if (i) either the model classes are misspecified or if (ii) the training algorithms do not induce the desired predictors, i.e. , if we have misspecified inductive biases. Misspecified inductive biases typically lead to inflated Type-I error rates but also can cause tests to be powerless. Even though these problems can frequently arise in practical situations, more attention should be given to theoretically understanding the effects of misspecification on CI hypothesis testing. Moreover, current regression-based methods are usually not designed to be robust against misspecification errors, making CI testing less reliable. In this work, we study the performance of three major regression-based conditional independence tests under misspecified inductive biases and propose the Rao-Blackwellized Predictor Test (RBPT), which is more robust against misspecification.

With more details, our main contributions are:

We present new robustness results for three relevant regression-based conditional independence tests: (i) Significance Test of Feature Relevance (STFR) [7], (ii) Generalized Covariance Measure (GCM) test [31], and (iii) REgression with Subsequent Independence Test (RESIT) [42, 24, 12]. Namely, we derive approximations or upper bounds for the testing errors that explicitly depend on the level of misspecification.

We introduce the Rao-Blackwellized Predictor Test (RBPT), a modification of the Significance Test of Feature Relevance (STFR) [7] test that is robust against misspecified inductive biases. In contrast with STFR and previous regression/simulation-based 1 methods, the RBPT does not require models to be correctly specified to guarantee Type-I error control. We develop theoretical results about the RBPT, and experiments show that RBPT is robust when controlling Type-I error while maintaining non-trivial power.

2 Preliminaries

Conditional independence testing. Let ( X,Y,Z ) be a random vector taking values in X × Y × Z ⊆ R d X × d Y × d Z and P be a fixed family of distributions on the measurable space ( X × Y × Z , B ) , where B = B ( X ×Y ×Z ) is the Borel σ -algebra. Let ( X,Y,Z ) ∼ P and assume P ∈ P . If P 0 ⊂ P is the set of distributions in P such that X ⊥ ⊥ Y | Z , the problem of conditional independence testing can be expressed in the following way:

In this work, we also write H 0 : X ⊥ ⊥ Y | Z and H 1 : X ̸⊥ ⊥ Y | Z . We assume throughout that we have access to a dataset D ( n + m ) = { ( X i , Y i , Z i ) } n + m i =1 independent and identically distributed (i.i.d.) as ( X,Y,Z ) , where D ( n + m ) splits into a test set D ( n ) te = { ( X i , Y i , Z i ) } n i =1 and a training set D ( m ) tr = { ( X i , Y i , Z i ) } n + m i = n +1 . For convenience, we use the training set to fit models and the test set to conduct hypothesis tests, even though other approaches are possible.

Misspecified inductive biases in modern statistics and machine learning. Traditionally, misspecified inductive biases in statistics have been linked to the concept of model misspecification and then strictly related to the chosen model classes. For instance, if the best (Bayes) predictor for Y given X , f ∗ , is a non-linear function of X , but we use a linear function to predict Y , then we say our model is misspecified because f ∗ is not in the class of linear functions. In modern machine learning and statistics, however, it is known that the training algorithm also plays a crucial role in determining the trained model. For example, it is known that training overparameterized neural networks using stochastic gradient descent bias the models towards functions with good generalization [14, 33]. In addition, D'Amour et al. [9] showed that varying hyperparameter values during training could result in significant differences in the patterns learned by the neural network. The researchers found, for instance, that models with different random initializations exhibit varying levels of out-of-distribution accuracy in predicting skin health conditions for different skin types, indicating that each model learned distinct features from the images. The sensitivity of the trained model concerning different training settings suggests that even models capable of universal approximation may not accurately estimate the target predictor if the training biases do not induce the functions we want to learn.

We present a toy experiment to empirically demonstrate how the training algorithm can prevent us from accurately estimating the target predictor even when the model class is correctly specified, leading to invalid significance tests. We work in the context of a high-dimensional (overparameterized) regression with a training set of 250 observations and 500 covariates. We use the Generalized Covariance Model (GCM) test 2 [31] to conduct the CI test. The data are generated as

where the first five entries of β X are set to 20 , and the remaining entries are zero, while the last five entries of β Y are set to 20 , and the remaining entries are zero. This results in X and Y being conditionally independent given Z and depending on Z only through a small number of entries. Additionally, E [ X | Z ] = β ⊤ X Z and E [ Y | Z ] = β ⊤ Y Z , indicating that the linear model class is correctly specified. To

perform the GCM test, we use LASSO ( ∥·∥ 1 penalization term added to empirical squared error) and the minimum-norm least-squares solution to fit linear models that predict X and Y given Z . In this problem, the LASSO fitting approach provides the correct inductive bias since β X and β Y are sparse.

We set the significance level to α = 10% and estimate the Type-I error rate for 100 different training sets. Figure 1 provides the Type-I error rate empirical distribution and illustrates that, despite using the same model class for both fitting methods, the training algorithm induces an undesired predictor in the minimum-norm case, implying an invalid test most of the time. On the other hand, the LASSO approach has better Type-I error control. In Appendix A, we give a similar example but using the Significance Test of Feature Relevance (STFR) [7].

In this work, we formalize the idea of misspecified inductive biases in the following way. Assume that a training algorithm A is used to choose a model ˆ g ( m ) = A ( D ( m ) tr ) from the class G ( m ) . We further assume that the sequence (ˆ g ( m ) ) m ∈ N converges to a limiting model g ∗ in a relevant context-dependent sense. We use different notions of convergence depending on the specific problem under consideration, which will be clear in the following sections. We say that g ∗ "carries misspecified inductive biases" if it does not equal the target Bayes predictor or regression function f ∗ . There are two possible reasons for g ∗ carrying misspecified

biases: either the limiting model class is small and does not include f ∗ , or the training algorithm A cannot find the best possible predictor, even asymptotically.

Notation. We write E P and Var P for the expectation and variance of statistics computed using i.i.d. copies of ( X,Y,Z ) ∼ P . Consequently, P P ( A ) = E P 1 A , where 1 A is the indicator of an event A . If E P and Var P are conditioned on some other statistics, we assume those statistics are also computed using i.i.d. samples from P . As usual, Φ is the N (0 , 1) distribution function. If ( a m ) m ∈ N and ( b m ) m ∈ N are sequences of scalars, then a m = o ( b m ) is equivalent to a m /b m → 0 as m →∞ and a m = b m + o (1) means a m -b m = o (1) . If ( V ( m ) ) m ∈ N is a sequence of random variables, where V ( m ) as constructed using i.i.d. samples of P ( m ) ∈ P for each m , then (i) V ( m ) = o p (1) means that for every ε > 0 we have P P ( m ) ( | V ( m ) | > ε ) → 0 as m → ∞ , (ii) V ( m ) = O p (1) means that for every ε > 0 there exists a M > 0 such that sup m ∈ N P P ( m ) ( | V ( m ) | > M ) < ε , (iii) V ( m ) = a m + o p (1) means V ( m ) -a m = o p (1) , (iv) V ( m ) = o p ( a m ) means V ( m ) /a m = o p (1) , and (v) V ( m ) = O p ( a m ) means V ( m ) /a m = O p (1) . Finally, let ( V ( m ) P ) m ∈ N ,P ∈P be a family of random variables that distributions explicitly depend on m ∈ N and P ∈ P . Wegive an example to clarify what we mean by "explicitly" depending on a specific distribution. Let V ( m ) P = 1 m ∑ m i =1 ( X i -µ P ) , where µ P = E P [ X ] . Here, V ( m ) P explicitly depends on P because of the quantity µ P . In this example, X i 's outside the expectation can have an arbitrary distribution (unless stated), i.e. , could be determined by P or any other distribution. With this context, (i) V ( m ) P = o P (1) means that for every ε > 0 we have sup P ∈P P P ( | V ( m ) P | > ε ) → 0 as m →∞ , (ii) V ( m ) P = O P (1) means that for every ε > 0 there exists a M > 0 such that sup m ∈ N ,P ∈P P P ( | V ( m ) P | > M ) < ε , (iii) V ( m ) P = o P ( a m ) means V ( m ) P /a m = o P (1) , and (iv) V ( m ) P = O P ( a m ) means V ( m ) P /a m = O P (1) .

Related work. There is a growing literature on the problem of conditional independence testing regarding both theoretical and methodological aspects 3 . From the methodological point of view, there is a great variety of tests with different natures. Perhaps, the most important groups of tests are (i) simulation-based tests [5, 3, 4, 32, 35, 20], (ii) regression-based tests [40, 24, 42, 38, 31, 7], (iii) kernel-based tests [10, 8, 34, 30], and (iv) information-theoretic based tests [29, 16, 39]. Due to the advance of supervised and generative models in recent years, regression and simulation-based tests have become particularly appealing, especially when Z is not low-dimensional or discrete. A related but different line of research is constructing a lower confidence bound for conditional dependence of X and Y given Z [41]. In that work, the authors propose a method that relies on computing the conditional expectation of a possibly misspecified regression model, which can be related to

our method presented in Section 4. Despite the relationship between methods, their motivations, assumptions, and contexts are different.

Simulation-based tests depend on the fact that we can, implicitly or explicitly, approximate the conditional distributions P X | Z or P Y | Z . Two relevant simulation-based methods are the conditional randomization and conditional permutation tests (CRT/CPT) [5, 3, 4, 35]. For these tests, Berrett et al. [4] presents robustness results showing that we can approximately control Type I error even if our estimates for the conditional distributions are not perfect and we are under a finite-sample regime. However, it is also clear from their results that CRT and CPT might not control Type I error asymptotically when models for conditional distributions are misspecified. On the other hand, regression-based tests work under the assumption that we can accurately approximate the conditional expectations E [ X | Z ] and E [ Y | Z ] or other Bayes predictors, which is hardly true if the modeling and training inductive biases are misspecified. To the best of our knowledge, there are no published robustness results for regression-based CI tests like those presented by Berrett et al. [4]. We explore this literature gap.

3 Regression-based conditional independence tests under misspecified inductive biases

This section provides results for the Significance Test of Feature Relevance (STFR) [7]. Due to limited space, the results for the Generalized Covariance Measure (GCM) test [31] and the REgression with Subsequent Independence Test (RESIT) [42, 24, 12] are presented in Appendix A. From the results in Appendix A, one can easily derive a double robustness property for both GCM and RESIT, implying that not all models need to be correctly specified or trained with the correct inductive biases for Type-I error control.

3.1 Significance Test of Feature Relevance (STFR)

The STFR method studied by Dai et al. [7] offers a scalable approach for conducting conditional independence testing by comparing the performance of two predictors. To apply this method, we first train two predictors ˆ g ( m ) 1 : X × Z → Y and ˆ g ( m ) 2 : Z → Y on the training set D ( m ) tr to predict Y given ( X,Z ) and Z , respectively. We assume that candidates for ˆ g ( m ) 2 are models in the same class as ˆ g ( m ) 1 but replacing X with null entries. Using samples from the test set D ( n ) te , we conduct the test rejecting H 0 : X ⊥ ⊥ Y | Z if the statistic Λ ( n,m ) ≜ √ n ¯ T ( n,m ) / ˆ σ ( n,m ) exceeds τ α ≜ Φ -1 (1 -α ) , depending on the significance level α ∈ (0 , 1) . We define ¯ T ( n,m ) and ˆ σ ( n,m ) as

with T ( m ) i ≜ ℓ (ˆ g ( m ) 2 ( Z i ) , Y i ) -ℓ (ˆ g ( m ) 1 ( X i , Z i ) , Y i ) + ε i . Here, ℓ is a loss function, typically used during the training phase, and { ε i } n i =1 iid ∼ N (0 , ρ 2 ) are small artificial random noises that do not let ˆ σ ( n,m ) vanish with a growing training set, thus allowing the asymptotic distribution of Λ ( n,m ) to be standard normal under H 0 : X ⊥ ⊥ Y | Z . If the p -value is defined as p ( D ( n ) te , D ( m ) tr ) = 1 -Φ(Λ ( n,m ) ) , the test is equivalently given by

The rationale behind STFR is that if H 0 : X ⊥ ⊥ Y | Z holds, then ˆ g ( m ) 1 and ˆ g ( m ) 2 should have similar performance in the test set. On the other hand, if H 0 does not hold, we expect ˆ g ( m ) 1 to have significantly better performance, and then we would reject the null hypothesis. Said that, to control STFR's Type-I error, it is necessary that the risk gap between ˆ g ( m ) 1 and ˆ g ( m ) 2 , E P [ ℓ (ˆ g ( m ) 2 ( Z ) , Y ) | D ( m ) tr ] -E P [ ℓ (ˆ g ( m ) 1 ( X,Z ) , Y ) | D ( m ) tr ] , under H 0 vanishes as the training set size increases. Moreover, we need the risk gap to be positive for the test to have non-trivial power. These conditions can be met if the risk gap of g ∗ 1 ,P and g ∗ 2 ,P , the limiting models of ˆ g ( m ) 1 and ˆ g ( m ) 2 , is the same as the risk gap of the Bayes' predictors

where the minimization is done over the set of all measurable functions 4 . However, the risk gap between ˆ g ( m ) 1 and ˆ g ( m ) 2 will typically not vanish if g ∗ 1 ,P and g ∗ 2 ,P are not the Bayes' predictors even under H 0 . In general, we should expect g ∗ 1 ,P to perform better than g ∗ 2 ,P because the second predictor does not depend on X . Furthermore, their risk gap can be non-positive even if f ∗ 1 ,P performs better than f ∗ 2 ,P . In Appendix A.2, we present two examples in which model misspecification plays an important role when conducting STFR. The examples show that Type-I error control and/or power can be compromised due to model misspecification.

To derive theoretical results, we adapt the assumptions from Dai et al. [7]:

Assumption 3.1. There are functions g ∗ 1 ,P , g ∗ 2 ,P , and a constant γ > 0 such that

Assumption 3.2. There exists a constant k > 0 such that

Assumption 3.3. For every P ∈ P , there exists a constant σ 2 P > 0 such that

Finally, we present the results for this section. We start with an extension of Theorem 2 presented by Dai et al. [7] in the case of misspecified inductive biases.

Theorem 3.4. Suppose that Assumptions 3.1, 3.2, and 3.3 hold. If n is a function of m such that n →∞ and n = o ( m 2 γ ) as m →∞ , then

where o (1) denotes uniform convergence over all P ∈ P as m →∞ and

Theorem 3.4 demonstrates that the performance of STFR depends on the limiting models g ∗ 1 ,P and g ∗ 2 ,P . Specifically, if Ω STFR P > 0 , then E P [ φ STFR α ( D ( n ) te , D ( m ) tr )] → 1 even if H 0 : X ⊥ ⊥ Y | Z holds. In practice, we should expect Ω STFR P > 0 because of how we set the class for ˆ g ( m ) 2 . In contrast, we could have Ω STFR P ≤ 0 , and then E P [ φ STFR α ( D ( n ) te , D ( m ) tr )] ≤ α + o (1) , even if the gap between Bayes' predictors is positive. See examples in Appendix A.2 for both scenarios. Next, we provide Corollary 3.6 to clarify the relationship between testing and misspecification errors. This corollary formalizes the intuition that controlling Type-I error is directly related to misspecification of g ∗ 2 ,P , while minimizing Type-II error is directly related to misspecification of g ∗ 1 ,P .

Definition 3.5. For a distribution P and a loss function ℓ , define the misspecification gaps:

The misspecification gaps defined in Definition 3.5 quantify the difference between the limiting predictors g ∗ 1 ,P and g ∗ 2 ,P and the Bayes predictors f ∗ 1 ,P and f ∗ 2 ,P , i.e. , give a misspecification measure for g ∗ 1 ,P and g ∗ 2 ,P . Corollary 3.6 implies that the STFR controls Type-I error asymptotically if ∆ 2 ,P = 0 , and guarantees non-trivial power if the degree of misspecification of g ∗ 1 ,P is not large compared to the performance difference of the Bayes predictors ∆ P , that is, when ∆ P -∆ 1 ,P > 0 .

Corollary 3.6 (Bounding testing errors) . Suppose we are under the conditions of Theorem 3.4.

(Type-I error) If H 0 : X ⊥ ⊥ Y | Z holds, then

where o (1) denotes uniform convergence over all P ∈ P 0 as m →∞ .

(Type-II error) In general, we have

where o (1) denotes uniform convergence over all P ∈ P as m →∞ and ∆ P ≜ E P [ ℓ ( f ∗ 2 ,P ( Z ) , Y )] -E P [ ℓ ( f ∗ 1 ,P ( X,Z ) , Y )] .

4 A robust regression-based conditional independence test

This section introduces the Rao-Blackwellized Predictor Test (RBPT), a misspecification robust conditional independence test based on ideas from both regression and simulation-based CI tests. The RBPT assumes that we can implicitly or explicitly approximate the conditional distribution of X | Z and does not require inductive biases to be correctly specified. Because RBPT involves comparing the performance of two predictors and requires an approximation of the distribution of X | Z , we can directly compare it with the STFR [7] and the conditional randomization/permutation tests (CRT/CPT) [5, 4]. The RBPT can control Type-I error under relatively weaker assumptions compared to other tests, allowing some misspecified inductive biases.

The RBPT can be summarized as follows: (i) we train ˆ g ( m ) that predicts Y given ( X,Z ) using D ( m ) tr ; (ii) we obtain the Rao-Blackwellized predictor h ( m ) by smoothing ˆ g ( m ) , i.e. ,

then (iii) compare its performance with ˆ g ( m ) 's using the test set D ( n ) te and a convex loss 5 function ℓ (not necessarily used to train ˆ g ( m ) ), and (iv) if the performance of ˆ g ( m ) is statistically better than h ( m ) 's,

1 2 3 4 5 we reject H 0 : X ⊥ ⊥ Y | Z . The procedure described here bears a resemblance to the Rao-Blackwellization of estimators. In classical statistics, the Rao-Blackwell theorem [17] states that by taking the conditional expectation of an estimator with respect to a sufficient statistic, we can obtain a better estimator if the loss function is convex. In our case, the variable Z can be seen as a "sufficient statistic" for Y under the assumption of conditional independence H 0 : X ⊥ ⊥ Y | Z . If H 0 holds and the loss ℓ (ˆ y, y ) is convex in its first argument, we can show using Jensen's inequality that the resulting model h ( m ) has a lower risk relative to the initial model ˆ g ( m ) , i.e. , E P [ ℓ ( h ( m ) ( Z ) , Y ) | D ( m ) tr ] -E P [ ℓ (ˆ g ( m ) ( X,Z ) , Y ) | D ( m ) tr ] ≤ 0 . Then, the risk gap in RBPT is non-positive under H 0 in contrast with STFR's risk gap,

Algorithm 1: Obtaining p -value for the RBPT

Input: (i) Test set D ( n ) te = { ( X i , Y i , Z i ) } n i =1 , (ii) initial predictor ˆ g ( m ) , (iii) conditional distribution estimate ˆ Q ( m ) X | Z , (iv) convex loss function ℓ ;

Output: p-value p ;

and ˆ σ ( n,m ) being { T i } 's sample std dev (Eq. 3.1).

which we should expect to be always non-negative given the definition of ˆ g ( m ) 2 in that case. That fact negatively biases the RBPT test statistic, enabling better Type-I error control.

In practice, we cannot compute h ( m ) exactly because P X | Z is usually unknown. Then, we use an approximation ˆ Q ( m ) X | Z , which can be given explicitly, e.g. , using probabilistic classifiers or conditional density estimators [13], or implicitly, e.g. , using generative adversarial networks (GANs) [22, 3]. We assume that ˆ Q ( m ) X | Z is obtained using the training set. The approximated h ( m ) is

where the integral can be solved numerically in case ˆ Q ( m ) X | Z has a known probability mass function or Lebesgue density ( e.g. , via trapezoidal rule) or via Monte Carlo integration in case we can only sample from ˆ Q ( m ) X | Z . Finally, for a fixed significance level α ∈ (0 , 1) , the test φ RBPT α is given by Equation 3.2 where the p -value is obtained via Algorithm 1.

Before presenting RBPT results, we introduce some assumptions. Let Q ∗ X | Z represent the limiting model for ˆ Q ( m ) X | Z . The conditional distribution Q ∗ X | Z depends on the underlying distribution P , but we omit additional subscripts for ease of notation. Assumption 4.1 defines the limiting models and fixes a convergence rate.

Assumption 4.1. There is a function g ∗ P , a conditional distribution Q ∗ X | Z , and a constant γ > 0 s.t.

where d TV denotes the total variation (TV) distance. Additionally, assume that both ˆ Q ( m ) X | Z and Q ∗ X | Z are dominated by a common σ -finite measure which does not depend on Z or m .

The common dominating measure in Assumption 4.1 could be, for example, the Lebesgue measure in R d X . Next, Assumption 4.2 imposes additional constraints on the limiting model Q ∗ X | Z . Under that assumption, the limiting misspecification level must be uniformly bounded over all P ∈ P .

Assumption 4.2. For all P ∈ P , the chi-square divergence

is a well-defined integrable random variable and sup P ∈P E P [ χ 2 ( Q ∗ X | Z || P X | Z )] < ∞ .

Now, assume ˆ g ( m ) is chosen from a model class G ( m ) . Assumption 4.3 imposes constraints on the model classes {G ( m ) } and loss function ℓ .

Assumption 4.3. Assume (i) sup g ∈G ( m ) sup ( x,z ) ∈X×Z ∥ g ( x, z ) ∥ 1 ≤ M < ∞ , for some real and positive M > 0 , uniformly for all m , and (ii) that ℓ is a L -Lipschitz loss function (with respect to its first argument) for a certain L > 0 , i.e., for any ˆ y, ˆ y ′ , y ∈ Y , we have that | ℓ (ˆ y, y ) -ℓ (ˆ y ′ , y ) | ≤ L ∥ ˆ y -ˆ y ′ ∥ 2 .

Assumption 4.3 is valid by construction since we choose G ( m ) and the loss function ℓ . That assumption is satisfied when, for example, (a) models in ∪ m G ( m ) are uniformly bounded, (b) ℓ (ˆ y, y ) = ∥ ˆ y -y ∥ p p with p ≥ 1 , and (c) Y is a bounded subset of R d Y , i.e. , in classification problems and most of the practical regression problems. The loss ℓ (ˆ y, y ) = ∥ ˆ y -y ∥ p p , with p ≥ 1 , is also convex with respect to its first entry and then a suitable loss for RBPT. It is important to emphasize that ℓ does not need to be the same loss function used during the training phase. For example, we could use ℓ (ˆ y, y ) = ∥ ˆ y -y ∥ 2 2 in classification problems, where y is a one-hot encoded class label and ˆ y is a vector of predicted probabilities given by a model trained using the cross-entropy loss.

Theorem 4.4. Suppose that Assumptions 3.2, 3.3, 4.1, 4.2, and 4.3 hold. If n is a function of m such that n →∞ and n = o ( m γ ) as m →∞ , then

where o (1) denotes uniform convergence over all P ∈ P as m →∞ and Ω RBPT P = Ω RBPT P, 1 -Ω RBPT P, 2 with

and

When H 0 : X ⊥ ⊥ Y | Z holds and ℓ is a strictly convex loss function (w.r.t. its first entry), we have that Ω RBPT P, 2 > 0 , allowing 6 some room for the "incorrectness" of Q ∗ X | Z . That is, from Theorem 4.4, as long as Ω RBPT P ≤ 0 , i.e. , if Q ∗ X | Z 's incorrectness (measured by Ω RBPT P, 1 ) is not as big as Jensen's gap Ω P, 2 , RBPT has asymptotic Type-I error control. Uniform asymptotic Type-I error control is possible if sup P ∈P 0 Ω RBPT P ≤ 0 . This is a great improvement of previous work ( e.g. , STFR, GCM, RESIT, CRT, CPT) since there is no need for any model to converge to the ground truth if Ω RBPT P, 1 ≤ Ω RBPT P, 2 , which is a weaker condition. See however that a small Ω RBPT P, 2 reduces the room for Q ∗ X | Z incorrectness. In the extreme case, when g ∗ P is the Bayes predictor, and therefore does not depend on X under H 0 , we need 7 Q ∗ X | Z = P X | Z almost surely. On the other hand, if g ∗ P is close to the Bayes predictor, RBPT has better power. That imposes an expected trade-off between Type-I error control and

rat

1.0

0.8

dz =30

1.0

0.8

dz = 40

ra

1.0

0.8

y=0

1.0

0.8

y=0.02

error

error

0.6

0.6

0.6

0.6

0.4

0.4

e

0.4

0.4

be-l

Type

0.2

0.2

M

0.2

0.2

0.0

0.0

0.0

0.0

0.00

0.01

0.02

0.00

0.01

0.02

0

2

4

0

2

4

Y

θ

θ

RBPT

RBPT2

STFR

RESIT

GCM

RBPT

CRT

CPT

power. To make a comparison with Berrett et al. [4] 's results in the case of CRT and CPT, we can express our remark in terms of the TV distance between Q ∗ X | Z and P X | Z . It can be shown that if E P [ d TV ( Q ∗ X | Z , P X | Z )] ≤ Ω RBPT P, 2 / (2 ML ) , then Type-I error control is guaranteed (see Appendix A.5). This contrasts with Berrett et al. [4] 's results because E P [ d TV ( Q ∗ X | Z , P X | Z )] = 0 is not needed.

We conclude this section with some relevant observations related to the RBPT.

On RBPT's power. Like STFR, non-trivial power is guaranteed if the predictor g ∗ P is good enough . Indeed, the second part of Corollary 3.6 can be applied for an upper bound on RBPT's Type-II error.

Semi-supervised learning. Let Y denote a label variable. Situations in which unlabeled samples ( X i , Z i ) are abundant while labeled samples ( X i , Y i , Z i ) are scarce happen in real applications of conditional independence testing [5, 4]. RBPT is well suited for those cases because the practitioner can use the abundant data to estimate P X | Z flexibly. The semi-supervised learning scenario also applies to RBPT2, which we describe next.

Running RBPT when it is hard to estimate P X | Z : the RBPT2. There might be situations in which it is hard to estimate the full conditional distribution P X | Z . An alternative approach would be estimating the Rao-Blackwellized predictor directly using a second regressor. After training ˆ g ( m ) , we could use the training set, organizing it in pairs { ( Z i , ˆ g ( m ) ( Z i , X i )) } , to train a second predictor ˆ h ( m ) to predict ˆ g ( m ) ( Z, X ) given Z . That predictor could be trained to minimize the mean-squared error. The model ˆ h ( m ) should be more complex than ˆ g ( m ) , in the sense that we should hope that the first model performs better than the second under H 0 in predicting Y . Consequently, this approach is effective when unlabeled samples are abundant, and we can train ˆ h using both unlabeled data and the given training set. After obtaining ˆ h ( m ) , the test is conducted normally. We name this version of RBPT as "RBPT2". We include a note on how to adapt Theorem 4.4 for RBPT2 in Appendix A.6.

5 Experiments

We empirically 8 analyze RBPT/RBPT2 in the following experiments and compare them with relevant benchmarks, especially when the used models are misspecified. We assume α = 10% and ℓ (ˆ y, y ) = (ˆ y -y ) 2 . The benchmarks encompass STFR [7], GCM [31], and RESIT [42], which represent regression-based CI tests. Furthermore, we examine the conditional randomization/permutation tests (CRT/CPT) [5, 4] that necessitate the estimation of P X | Z .

Artificial data experiments. Our setup takes inspiration from Berrett et al. [4], and the data is generated as

Here, d Z denotes the dimensionality of Z , a and b are sampled from N (0 , I d Z ) , the constant c determines the conditional dependence of X and Y on Z , and the parameter γ dictates the hardness of conditional independence testing: a non-zero γ implies potential challenges in Type-I error control as there might be a pronounced marginal dependence between X and Y under H 0 . Moreover, the training (resp. test) dataset consists of 800 (resp. 200) entries, and every predictor we employ operates on linear regression. RESIT employs Spearman's correlation between residuals as a test statistic while

CRT and CPT 9 deploy STFR's test statistic, all of them with p -values determined by conditional sampling/permutations executed 100 times ( B = 100 ), considering ˆ Q X | Z = N ( ( b ⊤ Z ) 2 + θ, 1 ) . The value of θ gives the error level in approximating P X | Z . To get ˆ h in the two variations of RBPT, we either use ˆ Q X | Z (RBPT) or kernel ridge regression (KRR) equipped with a polynomial kernel to predict ˆ g 1 ( X,Z ) from Z (RBPT2). We sample generative parameters ( a, b ) five times using different random seeds, and for each iteration, we conduct 480 Monte Carlo simulations to estimate Type-I error and power. The presented results are the average ( ± standard deviation) estimated Type-I error/power across iterations.

dz = 40

1.0

1000

3PT2)

Unla

B

0.8

800

b

D

rat

0.6

600

p

S

error

0.4

400

De-l

e

0.2

200

0.0

0

0.000

0.005

0.010

0.015

0.020

Y

In Figure 2 (resp. 4) we compare our methods' TypeI error rates (with c = 0 ) (resp. power) against benchmarks. Regarding Figure 2, we focus on regressionbased tests (STFR, GCM, and RESIT) in the first two plots and on simulation-based tests (CRT and CPT) in the last two plots. Regarding the first two plots, it is not straightforward to compare the level of misspecification between our methods and the benchmarks, so we use this as an opportunity to illustrate Theorem 4.4 and results from Section 3 and Appendix A. Fixing θ = 0 for RBPT, the Rao-Blackwellized predictor h is perfectly obtained, permitting Type-I error control regardless of the chosen d Z . Using KRR for RBPT2 makes ˆ h close to h when d Z is not big and permits Type-I error control. When d Z is big, more data is needed to fit ˆ h , which can be accomplished using unlabeled data, as demonstrated in Figure 3 and commented in Section 4. On the other hand, Type-I error control is always violated for STFR, GCM, and RESIT when γ grows. Regarding the final two plots, we can more readily assess the robustness of the methods when discrepancies arise between ˆ Q X | Z and P X | Z as influenced by varying θ . Figure 2 illustrates that CRT is the least robust test in this context, succeeded by RBPT and CPT. In Figure 4, we investigate how powerful RBPT and RBPT2 can be in practice when d Z = 30 . We compare our methods with CPT (when θ = 0 ), which seems to have practical robustness against misspecified inductive biases. Figure 4 shows that RBPT2 and CPT have similar power while RBPT is slightly more conservative.

Some concluding remarks are needed. First, RBPT and RBPT2 have shown to be practical and robust alternatives to conditional independence testing, exhibiting reasonable Type-I error control, mainly when employed in conjunction with a large unlabeled dataset, and power. Second, while CPT demonstrates notable robustness and relatively good power, its practicality falls short compared to RBPT (or RBPT2). This is because CPT needs a known density functional form for ˆ Q X | Z (plus the execution of MCMC chains) whereas RBPT (resp. RBPT2) can rely on conventional Monte Carlo integration using samples from ˆ Q X | Z (resp. supervised learning).

Real data experiments. For our subsequent experiments, we employ the car insurance dataset examined by Angwin et al. [2]. This dataset encompasses four US states (California, Illinois, Missouri, and Texas) and includes information from numerous insurance providers compiled at the ZIP code granularity. The data offers a risk metric and the insurance price levied on a hypothetical customer with consistent attributes from every ZIP

1.0

二

RBPT

二

RBPT2

0.8

CPT

ler

0.6

M

0.4

0.2

0.0

0.0

0.1

0.2

0.3

0.4

code. ZIP codes are categorized as either minority or non-minority, contingent on the percentage of non-white residents. The variables in consideration are Z , denoting the driving risk; X , an indicator for minority ZIP codes; and Y , signifying the insurance price. A pertinent question revolves around

0.30

0.30

RBPT

STFR

CRT

RBPT

STFR

CRT

0.25

RBPT2

GCM

CPT

0.25

RBPT2

GCM

CPT

0.20

0.20

error

0.15

0.15

p-va

M

0.10

0.10

0.05

0.05

0.00

0.00

California

Illinois

Missouri

Texas

California

Illinois

Missouri

Texas

the validity of the null hypothesis H 0 : X ⊥ ⊥ Y | Z , essentially questioning if demographic biases influence pricing.

We split our experiments into two parts. In the initial part, our primary goal is to compare the Type-I error rate across various tests. To ensure that H 0 is valid, we discretize Z into twenty distinct values and shuffle the Y values corresponding to each discrete Z value. If a test maintains Type-I error control, we expect it to reject H 0 for at most α = 10% of the companies in each state. In the second part, we focus on assessing the power of our methods. Given our lack of ground truth, we qualitatively compare RBPT and RBPT2 findings with those obtained by baseline methods and delineated by Angwin et al. [2], utilizing a detailed and multifaceted approach. In this last experiment, we aggregate the analysis for each state without conditioning on the firm. We resort to logistic regression for estimating the distribution of X | Z used by RBPT, GCM, CRT, and CPT. For RBPT2, we use a CatBoost regressor [26] to yield the Rao-Blackwellized predictor. We omit RESIT in this experiment as the additive model assumption is inappropriate. Both CRT and CPT methods utilize the same test metrics as STFR. The first plot 10 of Figure 5 shows that RBPT and RBPT2 methods have better control over Type-I errors compared to all other methods, including CPT. The second plot reveals that all methods give the same qualitative result that discrimination against minorities in ZIP codes is most evident in Illinois, followed by Texas, Missouri, and California. These findings corroborate with those of Angwin et al. [2], indicating that our methodology has satisfactory power while maintaining a robust Type-I error control.

6 Conclusion

In this work, we theoretically and empirically showed that widely used regression-based conditional independence tests are sensitive to the specification of inductive biases. Furthermore, we introduced the Rao-Blackwellized Predictor Test (RBPT), a misspecification-robust conditional independence test. RBPT is theoretically grounded and has been shown to perform well in practical situations compared to benchmarks.

Limitations and future work. Two limitations of RBPT are that (i) the robustness of RBPT can lead to a more conservative test, as we have seen in the simulations; moreover, (ii) it requires the estimation of the conditional distribution P X | Z , which can be challenging. To overcome the second problem, we introduced a variation of RBPT, named RBPT2, in which the Rao-Blackwellized predictor is obtained in a supervised fashion by fitting a second model ˆ h : Z → Y that predicts the outputs of the first model ˆ g : X × Z → Y . However, this solution only works if ˆ h is better than ˆ g in predicting Y under H 0 , which ultimately depends on the model class for ˆ h and how that model is trained. Future research directions may include (i) theoretically studying the power of RBPT in more detail and (ii) better understanding RBPT2 from a theoretical or methodological point of view, e.g. , answering questions on how to choose and train the second model.

References

Chunrong Ai, Li-Hsien Sun, Zheng Zhang, and Liping Zhu. Testing unconditional and conditional independence via mutual information. Journal of Econometrics , 2022.

Julia Angwin, Jeff Larson, Lauren Kirchner, and Surya Mattu. Minority neighborhoods pay higher car insurance premiums than white areas with the same risk, April 2017. URL https://www.propublica.org/article/ minority-neighborhoods-higher-car-insurance-premiums-white-areas-same-risk . [Online; last accessed in 09-September-2022].

Alexis Bellot and Mihaela van der Schaar. Conditional independence testing using generative adversarial networks. Advances in Neural Information Processing Systems , 32, 2019.

Thomas B Berrett, Yi Wang, Rina Foygel Barber, and Richard J Samworth. The conditional permutation test for independence while controlling for confounders. Journal of the Royal Statistical Society: Series B (Statistical Methodology) , 82(1):175-197, 2020.

Emmanuel Candes, Yingying Fan, Lucas Janson, and Jinchi Lv. Panning for gold:'modelx'knockoffs for high dimensional controlled variable selection. Journal of the Royal Statistical Society: Series B (Statistical Methodology) , 80(3):551-577, 2018.

Olivier Cappé, Eric Moulines, and Tobias Rydén. Inference in hidden markov models. In Proceedings of EUSFLAT conference , pages 14-16, 2009.

Ben Dai, Xiaotong Shen, and Wei Pan. Significance tests of feature relevance for a black-box learner. IEEE Transactions on Neural Networks and Learning Systems , 2022.

Gary Doran, Krikamol Muandet, Kun Zhang, and Bernhard Schölkopf. A permutation-based kernel conditional independence test. In UAI , pages 132-141, 2014.

Alexander D'Amour, Katherine Heller, Dan Moldovan, Ben Adlam, Babak Alipanahi, Alex Beutel, Christina Chen, Jonathan Deaton, Jacob Eisenstein, Matthew D Hoffman, et al. Underspecification presents challenges for credibility in modern machine learning. Journal of Machine Learning Research , 2020.

Kenji Fukumizu, Arthur Gretton, Xiaohai Sun, and Bernhard Schölkopf. Kernel measures of conditional dependence. Advances in neural information processing systems , 20, 2007.

Clark Glymour, Kun Zhang, and Peter Spirtes. Review of causal discovery methods based on graphical models. Frontiers in genetics , 10:524, 2019.

Patrik Hoyer, Dominik Janzing, Joris M Mooij, Jonas Peters, and Bernhard Schölkopf. Nonlinear causal discovery with additive noise models. Advances in neural information processing systems , 21, 2008.

Rafael Izbicki and Ann B. Lee. Converting high-dimensional regression to high-dimensional conditional density estimation. 2017.

Dimitris Kalimeris, Gal Kaplun, Preetum Nakkiran, Benjamin Edelman, Tristan Yang, Boaz Barak, and Haofeng Zhang. Sgd on neural networks learns functions of increasing complexity. Advances in neural information processing systems , 32, 2019.

Ilmun Kim, Matey Neykov, Sivaraman Balakrishnan, and Larry Wasserman. Local permutation tests for conditional independence. arXiv preprint arXiv:2112.11666 , 2021.

Mariusz Kubkowski, Jan Mielniczuk, and Pawel Teisseyre. How to gain on power: Novel conditional independence tests based on short expansion of conditional mutual information. J. Mach. Learn. Res. , 22:62-1, 2021.

Erich L Lehmann and George Casella. Theory of point estimation . Springer Science & Business Media, 2006.

Erich Leo Lehmann, Joseph P Romano, and George Casella. Testing statistical hypotheses , volume 3. Springer, 2005.

Chun Li and Xiaodan Fan. On nonparametric conditional independence tests for continuous variables. Wiley Interdisciplinary Reviews: Computational Statistics , 12(3):e1489, 2020.

Molei Liu, Eugene Katsevich, Lucas Janson, and Aaditya Ramdas. Fast and powerful conditional randomization testing via distillation. Biometrika , 109(2):277-293, 2022.

Alexander Marx and Jilles Vreeken. Testing conditional independence on discrete data using stochastic complexity. In The 22nd International Conference on Artificial Intelligence and Statistics , pages 496-505. PMLR, 2019.

Mehdi Mirza and Simon Osindero. Conditional generative adversarial nets. arXiv preprint arXiv:1411.1784 , 2014.

Matey Neykov, Sivaraman Balakrishnan, and Larry Wasserman. Minimax optimal conditional independence testing. The Annals of Statistics , 49(4):2151-2177, 2021.

Jonas Peters, Joris M Mooij, Dominik Janzing, and Bernhard Schölkopf. Causal discovery with continuous additive noise models. Journal of Machine Learning Research , 15:2009-2053, 2014.

Felipe Maia Polo, Rafael Izbicki, Evanildo Gomes Lacerda Jr, Juan Pablo Ibieta-Jimenez, and Renato Vicente. A unified framework for dataset shift diagnostics. Information Sciences , 649: 119612, 2023.

Liudmila Prokhorenkova, Gleb Gusev, Aleksandr Vorobev, Anna Veronika Dorogush, and Andrey Gulin. Catboost: unbiased boosting with categorical features. Advances in neural information processing systems , 31, 2018.

Sidney Resnick. A probability path . Springer, 2019.

Ya'acov Ritov, Yuekai Sun, and Ruofei Zhao. On conditional parity as a notion of nondiscrimination in machine learning. arXiv preprint arXiv:1706.08519 , 2017.

Jakob Runge. Conditional independence testing based on a nearest-neighbor estimator of conditional mutual information. In International Conference on Artificial Intelligence and Statistics , pages 938-947. PMLR, 2018.

Meyer Scetbon, Laurent Meunier, and Yaniv Romano. An asymptotic test for conditional independence using analytic kernel embeddings. arXiv preprint arXiv:2110.14868 , 2021.

Rajen D Shah and Jonas Peters. The hardness of conditional independence testing and the generalised covariance measure. The Annals of Statistics , 48(3):1514-1538, 2020.

Chengchun Shi, Tianlin Xu, Wicher Bergsma, and Lexin Li. Double generative adversarial networks for conditional independence testing. J. Mach. Learn. Res. , 22:285-1, 2021.

Samuel Smith, Erich Elsen, and Soham De. On the generalization benefit of noise in stochastic gradient descent. In International Conference on Machine Learning , pages 9058-9067. PMLR, 2020.

Eric V Strobl, Kun Zhang, and Shyam Visweswaran. Approximate kernel-based conditional independence tests for fast non-parametric causal discovery. Journal of Causal Inference , 7(1), 2019.

Wesley Tansey, Victor Veitch, Haoran Zhang, Raul Rabadan, and David M Blei. The holdout randomization test for feature selection in black box models. Journal of Computational and Graphical Statistics , 31(1):151-162, 2022.

Alexandre B Tsybakov. Introduction to nonparametric estimation, 2009. URL https://doi.org/10.1007/b13794. Revised and extended from the , 9(10), 2004.

David S Watson and Marvin N Wright. Testing conditional independence in supervised learning algorithms. Machine Learning , 110(8):2107-2129, 2021.

Hao Zhang, Shuigeng Zhou, Jihong Guan, and Jun Huan. Measuring conditional independence by independent residuals for causal discovery. ACM Transactions on Intelligent Systems and Technology (TIST) , 10(5):1-19, 2019.

Hao Zhang, Shuigeng Zhou, Kun Zhang, and Jihong Guan. Residual similarity based conditional independence test and its application in causal discovery. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 36, pages 5942-5949, 2022.

Kun Zhang, Jonas Peters, Dominik Janzing, and Bernhard Schölkopf. Kernel-based conditional independence test and application in causal discovery. arXiv preprint arXiv:1202.3775 , 2012.

Lu Zhang and Lucas Janson. Floodgate: inference for model-free variable importance. arXiv preprint arXiv:2007.01283 , 2020.

Qinyi Zhang, Sarah Filippi, Seth Flaxman, and Dino Sejdinovic. Feature-to-feature regression for a two-step conditional independence test. In 33rd Conference on Uncertainty in Artificial Intelligence, UAI 2017 . Association for Uncertainty in Artificial Intelligence (AUAI), 2017.

A Extra content

A.1 Misspecified inductive biases in modern statistics and machine learning

We present a toy experiment to empirically demonstrate how the training algorithm can prevent us from accurately estimating the Bayes predictor even when the model class is correctly specified, leading to invalid significance tests. We work in the context of a high-dimensional (overparameterized) regression with a training set of 250 observations and ≥ 300 covariates. We use the Significance Test of Feature Relevance test 11 (STFR) [7] to conduct the CI test. The data are generated as

where the first thirty entries of β are set to 1, and the remaining entries are zero. See that X ⊥ ⊥ Y | Z and that Y is linearly related to Z and ( X,Z ) , and then the class of linear predictors is correctly specified when predicting Y from Z or ( X,Z ) . To perform the STFR test, we use LASSO ( ∥·∥ 1 penalization term added to empirical squared error) and principal components regression (PCR) to train the linear predictors. Since β

is sparse, the LASSO fit provides the correct inductive bias while PCR leads to misspecification . We set the significance level to α = 1% and estimate the Type-I error rate for 100 different training sets. Figure 6 provides the Type-I error rate empirical distribution and illustrates that, despite using the same model class for both fitting methods, the training algorithm induces the wrong model in the PCR case, implying an invalid test most of the time.

A.2 Examples on when STFR fails

Examples A.1 and A.2 show simple situations in which Type-I error control is compromised or the conditional independence test has no power due to model misspecification. As we see in the next examples, Type-I error control is directly related to G 2 misspecification, while Type-II error minimization directly relates to G 1 misspecification.

Example A.1 (No Type-I error control) . Suppose Y = Z + Z 2 + ε y and X = Z 2 + ε x , where ε y , ε x ∼ N (0 , 1) are independent noise variables and Z has finite variance. Consequently, X ⊥ ⊥ Y | Z . Let G 1 and G 2 be the classes of linear regressors with no intercept, i.e.,

If ℓ denotes the mean squared error, we have that E P [ ℓ ( g ∗ 2 ,P ( Z ) , Y )] -E P [ ℓ ( g ∗ 1 ,P ( X,Z ) , Y )] > 0 because the model class G 2 is misspecified, that is, it does not contain the Bayes predictor given by the conditional expectation E [ Y | Z ] .

Example A.2 (Powerless test) . Suppose Y = Z +sin( X ) + ε y , where Z, X, ε y iid ∼ N (0 , 1) . Consequently, X ̸⊥ ⊥ Y | Z . Define G 1 and G 2 as in Example A.1. If ℓ denotes the mean squared error, we have that 12 E P [ ℓ ( g ∗ 2 ,P ( Z ) , Y )] -E P [ ℓ ( g ∗ 1 ,P ( X,Z ) , Y )] = 0 even though the different Bayes' predictors have a difference in performance. This happens because the model g ∗ 1 ,P is not close enough to the Bayes predictor E P [ Y | X,Z ] .

A.3 Generalized Covariance Measure (GCM) test

In the GCM test proposed by Shah and Peters [31], the expected value of the conditional covariance between X and Y given Z is estimated and then tested to determine if it equals zero. To simplify the exposition, we consider X and Y univariate and work in a setup similar to the STFR's. If ( X,Y,Z ) ∼ P , the GCM test relies on the observation that we can always write

where f ∗ 1 ,P ( Z ) = E P [ X | Z ] and f ∗ 2 ,P ( Z ) = E P [ Y | Z ] while the error terms { ϵ, η } have zero mean when conditioned on Z . Consequently, we can write E P [ Cov P ( X,Y | Z )] = E P [ ϵη ] . To estimate E P [ Cov P ( X,Y | Z )] , we can first fit two models ˆ g ( m ) 1 : Z → X and ˆ g ( m ) 2 : Z → Y , that approximate f ∗ 1 ,P and f ∗ 2 ,P , using the training set D ( m ) tr , and then compute an empirical version of E P [ ϵη ] = E P [( X -f ∗ 1 ,P ( Z ))( Y -f ∗ 2 ,P ( Z ))] using ˆ g ( m ) 1 , ˆ g ( m ) 2 , and D ( n ) te .

In the GCM test, we reject H 0 : X ⊥ ⊥ Y | Z if the statistic Γ ( n,m ) ≜ | √ n ¯ T ( n,m ) / ˆ σ ( n,m ) | exceeds τ α/ 2 ≜ Φ -1 (1 -α/ 2) , depending on the test significance level α ∈ (0 , 1) . Here, ¯ T ( n,m ) and ˆ σ ( n,m ) are defined as in 3.1 with T ( m ) i ≜ ( X i -ˆ g ( m ) 1 ( Z i ))( Y i -ˆ g ( m ) 2 ( Z i )) . If the p -value is defined as p ( D ( n ) te , D ( m ) tr ) = 2(1 -Φ(Γ ( n,m ) )) , the test φ GCM α ( D ( n ) te , D ( m ) tr ) is analogously given by Equation 3.2. Like the STFR, the GCM test depends on the models' classes and implicitly on the training algorithm. If the limiting models g ∗ 1 ,P and g ∗ 2 ,P are not f ∗ 1 ,P and f ∗ 2 ,P , then Type-I error control is not guaranteed.

We introduce definitions and assumptions. Assumption A.3 gives a rate of convergence for the models ˆ g ( m ) j in the mean squared error sense. Definition A.4 gives a definition for the misspecification gaps.

Assumption A.3. There are functions g ∗ 1 ,P , g ∗ 2 ,P , and a constant γ > 0 such that

Definition A.4. For each j ∈ { 1 , 2 } , define the misspecification gap as δ j,P ≜ g ∗ j,P -f ∗ j,P .

In the next result, we approximate GCM test Type-I error rate and power using the gaps in Definition A.4 and Assumptions A.3, 3.2, and 3.3 applied to this context.

Theorem A.5. Suppose that Assumptions 3.2, 3.3, and A.3 hold. If n is a function of m such that n →∞ and n = o ( m γ ) as m →∞ , then

where o (1) denotes uniform convergence over all P ∈ P as m →∞ and

̸

From Theorem A.5, it is possible to verify that if δ j,P ( Z ) is zero for at least one j ∈ { 1 , 2 } , i.e. , if at least one model converges to the conditional expectation, the GCM test asymptotically controls Type-I error. This can be seen as a double-robustness property of the GCM, which is not present 13 in Shah and Peters [31]. If E P [ δ 1 ,P ( Z ) δ 2 ,P ( Z )] = 0 , then E P [ φ GCM α ( D ( n ) te , D ( m ) tr )] → 1 as m →∞ even when H 0 : X ⊥ ⊥ Y | Z . Under the alternative, if Ω GCM P = 0 , Type-II error approaches 0 asymptotically.

̸

A.4 REgression with Subsequent Independence Test (RESIT)

As revisited by Zhang et al. [42], the idea behind RESIT is to first residualize Y and X given Z and then test dependence between the residuals. It is similar to GCM, but requires the error terms and Z to be independent. When that assumption is reasonable, one advantage of RESIT over GCM is that it has power against a broader set of alternatives. In this section, we use a permutation test [18, Example 15.2.3] to assess the independence of residuals. We analyse RESIT's Type-I error control.

If ( X,Y,Z ) ∼ P and ( X,Y ) can be modeled as an additive noise model (ANM), that is, we can write

where f ∗ 1 ,P ( Z ) = E P [ X | Z ] , f ∗ 2 ,P ( Z ) = E P [ Y | Z ] , and the error terms ( ϵ, η ) are independent of Z , it is possible to show that X ⊥ ⊥ Y | Z ⇔ ϵ ⊥ ⊥ η. To facilitate our analysis 14 , we consider first fitting two models ˆ g ( m ) 1 and ˆ g ( m ) 2 that approximate f ∗ 1 ,P and f ∗ 2 ,P using the training set D ( m ) tr and then test the independence of the residuals 15 ˆ ϵ i = X i -ˆ g ( m ) 1 ( Z i ) and ˆ η i = Y i -ˆ g ( m ) 2 ( Z i ) using the test set D ( n ) te . Define (i) ( ˆ ϵ , ˆ η ) ≜ { (ˆ ϵ i , ˆ η i ) } n i =1 (test set residuals vertically stacked in matrix form) and (ii) ( ˆ ϵ , ˆ η ) ( b )

as one of the B permutations, i.e. , consider that we fix ˆ ϵ and permute ˆ η row-wise. Let Ψ be a test statistic and Ψ(( ˆ ϵ , ˆ η )) and Ψ(( ˆ ϵ , ˆ η ) ( b ) ) its evaluation on the original residuals and the b -the permuted set. If the permutation p -value is given by

a test φ RESIT α aiming level α ∈ (0 , 1) is given by Equation 3.2.

Similarly to STFR and GCM, we consider g ∗ 1 ,P and g ∗ 2 ,P to be the limiting models for ˆ g ( m ) 1 and ˆ g ( m ) 2 . Different from GCM, both models g ∗ 1 ,P and g ∗ 2 ,P are multi-output since X and Y are not necessarily univariate.

Now, we introduce some assumptions before we present our result for RESIT. Assumption A.6 gives a rate of convergence for the models ˆ g ( m ) j in the mean squared error sense.

Assumption A.6. There are models g ∗ 1 ,P , g ∗ 2 ,P , and a constant γ > 0 such that

Assumption A.7 puts more structure on the distributions of the error terms ( ϵ, η ) and is a mild assumption.

Assumption A.7. Assume that for all P ∈ P 0 , the distribution of ( ϵ, η ) , P ϵ,η , is absolutely continuous with respect to the Lebesgue measure in R d X × R d Y with L -Lipschitz density p ϵ,η for a certain L > 0 . That is, for any e 1 , e 2 ∈ R d X and h 1 , h 2 ∈ R d Y , we have

We assume that L does not depend on P .

Assumption A.8 states that some of the variables we work with are uniformly almost surely bounded over all P ∈ P 0 . This assumption is realistic in most practical cases.

Assumption A.8. There is bounded Borel set A ∈ B ( R d X × d Y ) such that

Here, inf g 1 ,g 2 is taken over the model classes we consider (if the model classes vary with m , consider the union of model classes). In the following, we present the result for RESIT. For that result, let: (i) ϵ ∗ ≜ ϵ -δ 1 ,P ( Z i ) and η ∗ ≜ η -δ 2 ,P ( Z i ) , where the misspecification gaps are given as in Definition A.4; (ii) d TV represent the total variation (TV) distance between two probability distributions [36]; and (iii) the superscript n , e.g. , in P n ϵ ∗ ,η ∗ , represent a product measure.

Theorem A.9. Under Assumptions A.6, A.7, and A.8, if H 0 : X ⊥ ⊥ Y | Z holds and n is a function of m such that n →∞ and n = o ( m γ 2 ) as m →∞ , then

where o (1) denotes uniform convergence over all P ∈ P 0 as m →∞ .

From Theorem A.9, we can see that if at least one of the misspecification gaps δ 1 ,P or δ 2 ,P is null, then E P [ φ RESIT α ( D ( n ) te , D ( m ) tr )] ≤ α under H 0 : X ⊥ ⊥ Y | Z . This can be seen as a double-robustness property of the RESIT. If none of the misspecification gaps are null, we do not have any guarantees on RESIT's Type-I error control. It can be shown that the proposed upper bound converges to 1 .

and

A.5 RBPT extra derivation

Let µ X be a dominating measure of Q ∗ X | Z and P X | Z that does not depend on Z . Let q ∗ X | Z (resp. p X | Z ) be Q ∗ X | Z (resp. P X | Z ) density with respect to µ X .

If then

A.6 How to obtain a result for RBPT2?

We informally give some ideas on extending Theorem 4.4. Theorem 4.4 states that

(1) where Ω RBPT P = Ω RBPT P, 1 -Ω RBPT P, 2 with Ω RBPT P, 1 ≜ E P [ ℓ (∫ g ∗ P ( x, Z ) dQ ∗ X | Z ( x ) , Y ) ] -E P [ ℓ (∫ g ∗ P ( x, Z ) dP X | Z ( x ) , Y ) ] and ) ]

If we wanted to adapt that result for RBPT2, we would have to redefine Ω RBPT P, 1 . The analogue of Ω RBPT P, 1 for RBPT2 would be

where ˜ E P [ g ∗ P ( X,Z ) | Z = z ] denotes the limiting regression model to predict g ∗ P ( X,Z ) given Z . If we assume the existence of a big unlabeled dataset, deriving a result might be easier as we can avoid the asymptotic details on the convergence of the second regression model by assuming that the limiting Rao-Blackwellization model, for a fixed initial predictor, is known. The only challenge is proving the convergence of ˜ E P [ˆ g ( X,Z ) | Z ] to ˜ E P [ g ∗ P ( X,Z ) | Z ] .

B Technical proofs

B.1 STFR

Lemma B.1. Assume we are under the conditions of Theorem 3.4. Then:

Proof. First, see that for an arbitrary ε > 0 , there must be 16 a sequence of probability measures in P , ( P ( m ) ) m ∈ N , such that

Pick one of such sequences. Then, to prove that (ˆ σ ( n,m ) ) 2 -Var P [ T ( m ) 1 | D ( m ) tr ] = o P (1) as m →∞ , it suffices to show that

Now, expanding (ˆ σ ( n,m ) ) 2 we get

Then,

Using a law of large numbers for triangular arrays [6, Corollary 9.5.6] (we comment on needed conditions to use this result below) and the continuous mapping theorem, we have that

and then

as m →∞ , i.e.,

Conditions to use the law of large numbers. Let ( P ( m ) ) m ∈ N be an arbitrary sequence of probability measures in P . Define our triangular arrays as { V ( m ) i, 1 } 1 ≤ i ≤ n and { V ( m ) i, 2 } 1 ≤ i ≤ n , where V ( m ) i, 1 ≜ T ( m ) i -E P ( m ) [ T ( m ) 1 | D ( m ) tr ] and V ( m ) i, 2 ≜ ( T ( m ) i ) 2 -E P ( m ) [( T ( m ) 1 ) 2 | D ( m ) tr ] . Now, we comment on the conditions for the law of large numbers [6, Corollary 9.5.6]:

This condition naturally applies by definition and because of Assumption 3.2.

From Assumption 3.2 and Resnick [27, Example 6.5.2],

and

Fix any ϵ > 0 and let k be defined as in Assumption 3.2. See that

where the third inequality is obtained via Minkowski Inequality [27] and the fifth step is an application of Assumption 3.2 and Resnick [27, Example 6.5.2]. Analogously, define k ′ = k/ 2 and see that

Theorem 3.4. Suppose that Assumptions 3.1, 3.2, and 3.3 hold. If n is a function of m such that n →∞ and n = o ( m 2 γ ) as m →∞ , then

where o (1) denotes uniform convergence over all P ∈ P as m →∞ and

Proof. First, note that there must be 17 a sequence of probability measures in P , ( P ( m ) ) m ∈ N , such that

Then, it suffices to show that the RHS vanishes when we consider such a sequence ( P ( m ) ) m ∈ N . Now, let us first decompose the test statistic Λ ( n,m ) in the following way:

Λ

(

n,m

)

≜

Given that n is a function of m , we omit it when writing the W ( m ) j,P ( m ) 's. Define σ ( m ) P ( m ) ≜ √ Var P ( m ) [ T ( m ) 1 | D ( m ) tr ] and see that

Implying that

Justifying step B.1. First, from a central limit theorem for triangular arrays [6, Corollary 9.5.11], we have that

we comment on the conditions to use this theorem below.

Second, we have that

To see why the random quantity above converges to zero in probability, see that because of Assumption 3.3, Lemma B.1, and continuous mapping theorem, we have that

Additionally, because of Assumptions 3.1, 3.3 and condition n = o ( m 2 γ ) , we have that

Finally,

by Slutsky's theorem. Because N (0 , 1) is a continuous distribution, we have uniform convergence of the distribution function [27][Chapter 8, Exercise 5] and we do not have to worry about the fact that τ α -√ n Ω STFR P ( m ) σ ( m ) depends on m .

Conditions to apply the central limit theorem. Now, we comment on the conditions for the central limit theorem [6, Corollary 9.5.11]. Define our triangular array as { V ( m ) i } 1 ≤ i ≤ n , where V ( m ) i ≜

This condition naturally applies by definition and because of Assumption 3.2.

See that, for any m , we have that

Fix any ϵ > 0 and let k be defined as in Assumption 3.2. See that

where the third inequality is obtained via Minkowski Inequality [27] and the last inequality is an application of Assumption 3.2 and Resnick [27, Example 6.5.2].

Corollary 3.6 . Suppose we are under the conditions of Theorem 3.4.

(Type-I error) If H 0 : X ⊥ ⊥ Y | Z holds, then

where o (1) denotes uniform convergence over all P ∈ P 0 as m →∞ .

(Type-II error) In general, we have

where o (1) denotes uniform convergence over all P ∈ P as m →∞ and ∆ P ≜ E P [ ℓ ( f ∗ 2 ,P ( Z ) , Y )] -E P [ ℓ ( f ∗ 1 ,P ( X,Z ) , Y )] .

Proof. Under the conditions of Theorem 3.4, we start proving that

∆ P -∆ 1 ,P ≤ Ω STFR P holds;

Under H 0 , Ω STFR P ≤ ∆ 2 ,P holds.

For (1), see that,

Finally, see that and

Combining these observations with Theorem 3.4, we get the result.

B.2 GCM

Theorem A.5 . Suppose that Assumptions A.3, 3.2, and 3.3 hold. If n is a function of m such that n →∞ and n = o ( m γ ) as m →∞ , then

where o (1) denotes uniform convergence over all P ∈ P as m →∞ and

18 ( m )

Then, it suffices to show that the RHS vanishes when we consider such a sequence ( P ( m ) ) m ∈ N . Now, let us first decompose the test statistic Γ ( n,m ) in the following way:

The terms involving one of the ϵ and η were all zero and were omitted. Given that n is a function of m , we omit it when writing the W ( m ) j,P ( m ) 's. Define σ ( m ) P ( m ) ≜ √ Var P ( m ) [ T ( m ) 1 | D ( m ) tr ] and see that

Implying that

Justifying step B.2. First, from a central limit theorem for triangular arrays [6, Corollary 9.5.11], we have that

The conditions for the central limit theorem [6, Corollary 9.5.11] can be proven to hold like in Theorem 3.4's proof.

Second, we have that

and

To see why the random quantities above converge to zero in probability, see that because of Assumption 3.3, Lemma 19 B.1, and continuous mapping theorem, we have that

Additionally, because of Assumptions A.3, 3.3, Cauchy-Schwarz inequality, and condition n = o ( m γ ) , we have that

Finally,

and

by Slutsky's theorem. Because N (0 , 1) is a continuous distribution, we have uniform convergence of the distribution function [27][Chapter 8, Exercise 5] and we do not have to worry about the fact that τ α/ 2 -√ n σ 2 P ( m ) Ω GCM P ( m ) or -τ α/ 2 -√ n σ 2 P ( m ) Ω GCM P ( m ) depends on m .

B.3 RESIT

Lemma B.2. Let P U,V and P ′ U,V be two distributions on ( U × V , B ( U × V ) ) , U × V ⊆ R d U × d V , with d U , d V ≥ 1 . Assume P U and P ′ U are dominated by a common σ -finite measure µ and that P V | U = u = P ′ V | U = u is dominated by a σ -finite measure ν u , for all u ∈ R d U . Then,

where d TV denotes the total variation distance between two probability measures.

Proof. Let p U and p ′ U denote the densities of P U and P ′ U w.r.t. µ , and let p V | U ( · | u ) denote the density of P V | U = u = P ′ V | U = u w.r.t. ν u . From Scheffe's theorem [36][Lemma 2.1], we have that:

Lemma B.3. For all i ∈ [ n ] , consider

Under Assumption A.7 and H 0 : X ⊥ ⊥ Y | Z , we have that

where d TV denotes the total variation distance between two probability measures.

Proof. Let us represent the stacked residuals (in matrix form) as ( ˆ ϵ , ˆ η ) = { (ˆ ϵ i , ˆ η i ) } n i =1 . See that

Then, because the random quantities above are conditionally discrete, their distribution is dominated by a counting measure depending on ( ¯ ϵ , ¯ η ) . Because the distribution of ( ϵ, η ) is absolutely continuous with respect to the Lebesgue measure, (ˆ ϵ, ˆ η ) | D ( m ) tr and ( ϵ, ˆ η ) | D ( m ) tr are also absolutely continuous 20 for every training set configuration, and then we can apply Lemma B.2 to get that

In the last step, we abuse TV distance notation: by the TV distance of two random variables, we mean the TV distance of their distributions.

Now, define the event

By the definition of the TV distance, we have that (under H 0 ):

where the last equality holds from the fact that, given the training set, rows of ( ˆ ϵ , ˆ η ) and ( ϵ , ˆ η ) are i.i.d.. See that P P ( (( ϵ , ˆ η ) , ( ϵ , ˆ η ) (1) , · · · , ( ϵ , ˆ η ) ( B ) ) ∈ A α | D ( m ) tr ) ≤ α because H 0 holds and therefore ϵ i ⊥ ⊥ ˆ η i | D ( m ) tr (making the permuted samples exchangeable).

Using symmetry, we have that

Lemma B.4. For any i ∈ [ n ] , consider that

where ˆ ϵ i = ϵ i -δ 1 ,P ( Z i ) -( ˆ g ( m ) 1 ( Z i ) -g ∗ 1 ,P ( Z i ) ) , ϵ ∗ i = ϵ i -δ 1 ,P ( Z i ) , ˆ η i = η i -δ 2 ,P ( Z i ) -( ˆ g ( m ) 2 ( Z i ) -g ∗ 2 ,P ( Z i ) ) , and η ∗ i = η i -δ 2 ,P ( Z i ) . Then, under H 0 : X ⊥ ⊥ Y | Z and Assumptions

A.6 and A.7, all six distributions are absolutely continuous with respect to the Lebesgue measure and their densities are given by

Additionally, we have that

Proof. Assume we are under H 0 . In order to show that P ˆ ϵ, ˆ η |D ( m ) tr is absolutely continuous w.r.t. Lebesgue measure (for each training set configuration) and that its density is given by

it suffices to show that

for any measurable set A .

Using Fubini's theorem, we get

The proof is analogous to the other distributions.

Now, we proceed with the second part of the lemma. Using Assumptions A.6 and A.7, we get

where the last inequality is obtained via Jensen's inequality. Because the upper bound obtained through Lipschitzness does not depend on ( e, h ) , we get

The results for the other converging quantities are obtained in the same manner.

Theorem A.9. Under Assumptions A.6, A.7, and A.8, if H 0 : X ⊥ ⊥ Y | Z holds and n is a function of m such that n →∞ and n = o ( m γ 2 ) as m →∞ , then

where o (1) denotes uniform convergence over all P ∈ P 0 as m →∞ .

Proof. We are trying to prove that

First, see that using Lemma B.3 we get

and

where

It suffices to show that sup P ∈P 0 ∆ ( m ) P = o (1) as m →∞ . Next step is to show that and

as m →∞ . Given the symmetry, we focus on the first problem.

By the triangle inequality, we obtain

and consequently

We treat these terms separately.

(I) Consider a sequence of probability distributions ( P ( m ) ) m ∈ N in P 0 such that

Here, the distributions P and P ( m ) determine not only the distribution associated with D ( m ) tr but also the distribution of (ˆ ϵ, ˆ η, ϵ ∗ , η ∗ ) . Because we have that

where V is the volume of a ball containing the support of the RVs (existence of that ball is due to Assumption A.8), we also have that d TV ( P ( m ) n ˆ ϵ, ˆ η |D ( m ) tr , P ( m ) n ϵ ∗ ,η ∗ ) = o p (1) , when D ( m ) tr samples come from the sequence ( P ( m ) ) m ∈ N . By the Dominated Convergence Theorem (DCT) [27, Corollary 6.3.2], we have

To see why we can use the DCT here, realize that when samples come from P ( m ) , W ( m ) = d TV ( P ( m ) n ˆ ϵ, ˆ η |D ( m ) tr , P ( m ) n ϵ ∗ ,η ∗ ) can be seen as a measurable function going from an original probability space to some other space. Different distributions { P ( m ) } are due to different random variables while the original probability measure is fixed. Because of that,

for the bounded random variable W ( m ) , where the last expectation is taken in the original probability space. We apply the DCT in E [ W ( m ) ] .

(II) Following the same steps as in part (I), we obtain

Going back to step B.3, consider another sequence of probability distributions ( Q ( m ) ) m ∈ N in P 0 such that where

Because of continuity of min and and

we have that

Finally implying, from step B.3,

B.4 RBPT

Theorem 4.4. Suppose that Assumptions 4.1, 4.2, 4.3, 3.2, and 3.3 hold. If n is a function of m such that n →∞ and n = o ( m γ ) as m →∞ , then

where o (1) denotes uniform convergence over all P ∈ P as m →∞ and

with and

Proof. First, note that there must be 21 a sequence of probability measures in P , ( P ( m ) ) m ∈ N , such that

Then, it suffices to show that the RHS vanishes when we consider such a sequence ( P ( m ) ) m ∈ N . Now, let us first decompose the test statistic Ξ ( n,m ) in the following way:

( m ) ( m )

Implying that

Justifying step B.5. First, from a central limit theorem for triangular arrays [6, Corollary 9.5.11], we have that

The conditions for the central limit theorem [6, Corollary 9.5.11] can be proven to hold like in Theorem 3.4's proof.

Second, we have that

To see why the random quantity above converges to zero in probability, see that because of Assumption 3.3, Lemma 22 B.1, and continuous mapping theorem, we have that

Additionally, because of Assumptions 4.1, 4.2, and 4.3 and condition n = o ( m γ ) , we have that

where M is an upper bound for the sequence (∥ ∥ ∥ ˆ g ( m ) ( x, z ) ∥ ∥ ∥ 1 ) m ∈ N (Assumption 4.3) and R is an upper bound for the sequence ( ( E P ( m ) [ χ 2 ( Q ∗ X | Z || P X | Z ) +1 ] ) 1 / 2 ) m ∈ N (Assumption 4.2). Finally,

by Slutsky's theorem. Because N (0 , 1) is a continuous distribution, we have uniform convergence of the distribution function [27][Chapter 8, Exercise 5], and we do not have to worry about the fact that τ α -√ n Ω RBPT P ( m ) σ P ( m ) depends on m .

C Experiments

C.1 Running times

Artificial-data experiments. Regarding running times (average per iteration), RBPT took 5 · 10 -4 s to run, RBPT2 took 1 . 9 s , STFR took 7 . 5 -4 s , RESIT took 1 . 3 · 10 -1 s , GCM took 6 · 10 -4 s , CRT took 1 . 7 · 10 -2 s , and CPT took 6 . 2 · 10 -1 s , all in a MacBook Air 2020 M1.

Real-data experiments. Regarding running times (average per iteration), RBPT took 1 . 1 · 10 -1 s to run, RBPT2 took 2 . 4 · 10 -1 s , STFR took 10 -3 s , GCM took 10 -3 s , CRT took 2 . 5 · 10 -2 s , and CPT took 7 . 2 · 10 -1 s , all in a MacBook Air 2020 M1.

C.2 Extra results

We include extra experiments in which Y | X,Z has skewed normal distributions with location µ = cX + a ⊤ Z + γ ( b ⊤ Z ) 2 , scale σ = 1 , and shape s = 3 (shape s = 0 lead to the normal distribution). From the following plots, we can learn that the skewness often impacts negatively in Type-I error control.

1.0

dz=30

1.0

dz = 40

rate

1.0

y=0

1.0

y=0.02

北

0.8

0.8

0.8

0.8

error

0.6

0.6

error

0.6

0.6

oe-l

0.4

0.4

e

0.4

0.4

Type

0.2

0.2

p

e

0.2

0.2

0.0

0.0

0.0

0.0

0.00

0.01

0.02

0.00

0.01

0.02

0

2

4

0

2

4

Y

θ

θ

RBPT

RBPT2

STFR

RESIT

GCM

RBPT

CRT

CPT

dz = 40

1.0

1000

3PT2)

Unla

B

0.8

800

b

D

rat

0.6

600

p

S

error

0.4

400

be-l

0.2

200

size

D

Type

0.0

0.000

0.005

0.010

0.015

0.020

Q

Y

1.0

RBPT

二

RBPT2

0.8

CPT

Power

0.6

0.4

0.2

0.0

0.0

0.1

0.2

0.3

0.4

Peer reviews:
comments: SUMMARY:
The paper proposes a new Rao-Blackwellized Predictor Test which is a regression-based conditional independence test. It was shown to be robust in misspecification settings.

STRENGTHS:
The paper is in general well-written and considers an important scenario which is not covered by standard conditional independence tests. The new approach guarantees Type-1 error control (asymptotically).

WEAKNESSES:
The paper is quite technical (what is good), however, a number of important details are in the Appendix what is not always practical.

QUESTIONS:
I would like a clarification for the phrase "even models capable of universal approximation may not accurately estimate the Bayes predictor...". What is exactly meant? Do you mean that if some hyper-parameters are not well-chosen then even a well-specified model does not approach the Bayes error? Even asymptotically? However, in an asymptotic case hyper-parameters do not play any role?

On page 3, Figure 4 is mentioned. Where is it? 

In Section 5, "real data experiments": X represents the driving risk and X is a minority indicator, but what is Y? Is it the price? It would be helpful to write it explicitly. 

"For RBPT2, we use a CatBoost regressor...": could you detail how you apply the CatBoost to get the Rao-Blackwellized predictor? What kind of modifications are made to the original CatBoost?

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-06-26

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors study model-based tests of conditional independence under conditions where the models may be biased. They propose new approximations and bounds on testing errors resulting from bias in model-based CI tests. They also introduce RBPT, a new model-based CI test and evaluate it against several alternatives.\n\nSTRENGTHS:\nThe paper is well-written and provides substantial theoretical derivation of the Type I error rate and power of the proposed test.\n\nThe authors address a very important problem that is under-appreciated. Many researchers would assume (mistakenly) that non-parametric conditional independence testing is a solved problem. In reality, there exists no generally accepted solution, despite substantial prior work.\n\nFurthermore, the authors work on a very important aspect of the problem. Model misspecification, as they term it, appears to be both ubiquitous and difficult to detect and diagnose. Making model-based CI tests that are robust to this source of bias would represent a major step forward. \n\nThe authors correctly point out that, despite a variety of deep models being *capable* of being universal function approximators, that does not mean that unbiased models are *actually learned in practice*.\n\nEmpirical evaluation of CI tests is difficult, at best. However, the authors provide both simulation and real-data experiments that provide some evidence that their approach delivers on its promises.\n\nWEAKNESSES:\nAs the authors note, the term “model misspecification” has a long history in statistics (and some parts of machine learning), and it seems a stretch to expand that term to include non-optimal parameter settings due to the limitations of a given training algorithm. There is certainly an issue here, but calling it “model misspecification” is likely to cause some initial confusion for readers. The authors are quite clear that they are redefining the term, but seeking an alternative term may be more helpful to readers. Some options include “non-optimal training”, “underspecification”, and “non-identifiability”, though I suspect there is an even better term.\n\nIn the caption of Figure 3, the authors state that “RBPT and RBPT2 have better Type-I error control compared to other methods.” However, STFR fairly clearly outperforms RBPT2 in terms of absolute error with respect to the 0.10 gold standard and STFR usually outperforms RBPT.\n\nOne minor correction: The text describing Figure 1 says that the Type I error rate was set to 0.01, but the figure shows that it was set to 0.10. Furthermore, that text refers to Figure 4, which does not appear to exist in the paper.\n\nQUESTIONS:\n(none)\n\nLIMITATIONS:\nThere appear to be relatively few negative impacts to the authors' work. Indeed, it facilitates *avoiding* important classes of errors that could have substantial negative impacts.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

comments: SUMMARY:
This paper presents new robustness results for three regression-based conditional independence (CI) tests, namely, Significance Test of Feature Relevance (STFR), Generalized Covariance Measure (GCM) test, and REgression with Subsequent Independence Test (RESIT). It introduces a new CI test, the Rao-Blackwellized Predictor Test (RBPT), designed to be more robust to model misspecification. Theoretical findings and experimental results show that the RBPT is able to control Type-I error effectively even when models are misspecified.

STRENGTHS:
- Well written and structured. Thanks
- Great intro to related work.
- Addresses a significant problem
- Provides theoretical and empirical insights.

WEAKNESSES:
- The authors make a long and good argument, that misspecification needs to account for the training algorithm. It seems that this is a natural result of the model posing a non-convex or underspecified optimization problem for training. Therefore, I'm not sure this needs to be demonstrated in such depth. (Please correct me, if I'm misunderstanding this)
- Could have more empirical validation.

QUESTIONS:
- In the eq. between lines 76, and 77. Why do you write " Y | X, Z ~ ... " and not just " Y | Z " ?

LIMITATIONS:
n.a.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper discusses the challenges of conditional independence (CI) testing in modern statistics and machine learning. The authors note that many modern methods for CI testing rely on supervised learning methods to learn regression functions or Bayes predictors, but these methods can be unreliable when the models are misspecified. The paper proposes new approximations or upper bounds for the testing errors of three regression-based tests that depend on misspecification errors. The authors present new robustness results for three relevant regression-based CI tests and derive approximations or upper bounds for the testing errors that explicitly depend on the misspecification of the prediction models. The paper concludes that more attention should be given to theoretically understanding the effects of misspecification on CI hypothesis testing and that current regression-based methods are usually not designed to be robust against misspecification errors, making CI testing less reliable.\n\nSTRENGTHS:\nThe writing is good.\n\nThe introduction on model misspecification is good.\n\nWEAKNESSES:\nLack of introducing conditional independence testing.\n\nRBPT2 is powerful but it cost much more time compared with other methods, such as STFR, GCM. Lack of analysis the time complexity of each designed method.\n\nOnly one real dataset. This method should be applied on more real application and dataset, like STFR which is applied on MNIST handwritten digits, Mechanisms of Action (MoA) prediction for new drugs and Chest X-rays for pneumonia diagnosis.\n\nQUESTIONS:\nWhy are regression-based and simulation-based tests more appealing than the other two tests?\n\nWhat is type-I error?\n\nWhy there are no result on Illinois and Texas in p-value of Figure 3?\n\nIn Figure 2, why you did not have results of RBPT2 on \\theta VS Type-I error rate?\n\nSince you compared your designed method with STFP, why not choose the same dataset as STFP to show the efficiency of your method?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 1, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper under review offers a thoughtful exploration of how model misspecification influences conditional independence testing. The significance and practical applications of such testing are aptly underscored in the introduction. The use of a toy example to demonstrate the impact of model misspecification on Type I error is a compelling approach that effectively propels their study. However, the paper's structure could be improved for better readability. While the work is technically exhaustive and appears sound, the layout requires refinement. For instance, the discussion and conclusion section seems truncated due to an overflow of technical details, presumably in an attempt to conform to the page limit.\n\nAdditionally, the paper presents an insightful study on Type I error bounds and illuminates how factors in model misspecification might influence the performance of our model. This is examined across three leading methods of regression-based conditional independence testing, STFR, GCM, and RESIT; a commendable breadth of investigation. On the other hand, there lies a lot of  ingenuity in the application of sufficient statistics and the Rao-Blackwell theorem to devise a conditional independence test - an approach that stands out as particularly clever.\n\nDespite the paper's notable theoretical contribution and illumination of model misspecification implications, its current state warrants a rejection recommendation. The primary rationale lies in the sparse experimental validation, which leaves room for further expansion. While a more comprehensive set of experimental results could potentially sway my assessment towards endorsement, the manuscript, as it stands, along with the limited set of experiments, doesn't quite meet the high standards necessary for acceptance. Thus, my score, on balance, falls on the lower side.\n\nSTRENGTHS:\n1. The paper exhibits a robust theoretical foundation, ingeniously leveraging the principles of sufficient statistics and the Rao-Blackwell theorem to address conditional independence testing. This innovative application, to the best of my understanding, introduces a novel concept and forms a significant contribution.\n2. The paper provides extensive misspecification bounds that comprehensively span across various regression-based methods. Moreover, the experiments also benchmark against simulation-based approaches.\n3. The exploration of examples in Appendix A.2 is highly instructive, where model misspecification can influence and Type I error of our test significantly. In addition, the choice of synthetic experiments adapted from Berrett et al. serves as a fitting means of demonstrating the concept in this study.\n4. In an innovative deviation from the traditional setting, this paper presents proofs across a series of models that converge to a pre-established class of models, thereby aligning well with contemporary machine learning paradigms\n\nWEAKNESSES:\n1. Acknowledging that the paper's paramount contribution resides in its theoretical development, I must express my reservations regarding the experimental element, which appears somewhat insufficient. To bolster the validation process, I've proposed additional experimental situations in the \"Questions\" section. As it stands, the validation primarily leans on a single toy experimental setup, and a comprehensive benchmarking across different methods in a real-world setup is lacking.\n2. In terms of presentation, the paper could benefit from some refinement. Notably, a significant portion of the proof concepts across different Conditional Independence Testing methods appears to overlap. A majority of these proofs could have been concisely presented by adopting lemmas that generalize across multiple methods, thus increasing reader engagement. Also, the main body is currently dense with scattered technical terms that could be better consolidated.\n3. Just a minor point of clarification, I'd like to delve into the details of the conducted toy experiments. Developing a unified framework that frames functions as a sequential path towards a distinct model group is truly commendable. Nonetheless, there's scope for refining its presentation. Specifically, the introduction gave the impression that this was a broad generalization of model misspecification, which it doesn't appear to be.\nUpon incorporating regularization terms during training, it's important to note that while the overarching model class remains constant, the subset of models that surface must adhere to the regularization-imposed limits. Consequently, the model's prime focus may deviate from merely minimizing the Bayes predictor loss.\nThis deviation could result from either integrating regularization terms into the optimization formula, thus modifying the loss characteristics, or from imposing these terms as constraints, effectively limiting the model class.\nConsidering these factors, I'd suggest positioning your model misspecifications as a tool to tailor the concept to real-world machine learning environments, rather than representing it as a broad generalization of the concept of model misspecification itself.\n4. One of the method's most significant practical implications, in my view, is the potential for semi-supervised learning on unlabeled data - a scenario where RBPT is likely to outshine RBPT2. Given its importance in practical real-world settings, this deserves an additional set of experiments. An easy experiment to consider would be to add a large set of unlabeled data to the toy experiment.\n\nQUESTIONS:\n1. Regarding eq. (3.1), it seems to me that $\\hat{\\sigma}$ mismatches the sample standard deviation of $T_i^{(m)}$; it is incorrectly expressed as a biased estimator. For simplicity, you might want to just denote $\\bar{T}^{(n,m)}$ and $\\hat{\\sigma}^{(n, m)}$ as the sample mean and variance estimators and not include any formulas, referencing Dei et al. for looking up the exact derivations.\n2. In the experiments section, especially concerning figure 2, I have several points of interest:\n* It would be insightful to observe how RBPT behaves in terms of Type I error when $Q$ is misspecified. For instance, a scenario where $c=0$, $\\theta=1$, and $\\gamma$ is progressively increased could be worth examining. Alternatively, to evaluate how sensitive RBPT is to such misspecification, setting $\\gamma$ to a larger value (say 0.02), and sweeping through different $\\theta$ values to assess Type I error might be valuable. I believe this experiment is of extreme importance since, in most real-world situations, the total variation between $Q^*(X|Z)$ and $P(X|Z)$ is significant and may not be bounded by the Jensen gap. Assessing this sensitivity is pivotal to demonstrating the robustness of the setting. The second plot, where $\\gamma = 0$, may not adequately reflect this robustness.\n* It would be beneficial to include the values of other hyperparameters set in the caption of Figure 2. This would prevent the need to navigate back and forth through the text to understand the setup; also, I assumed that for the third plot $\\gamma = \\theta = 0$, but it was not mentioned in the main text.\n3. Why hasn't the power of RBPT been directly compared with the baselines for the real-world examples? The reasons for not contrasting RBPT are somewhat elusive to me. While I concur that both RBPT and RBPT2 theoretically establish bounds for Type I error control under model misspecification, a quantitative comparison with baselines is still essential due to the trade-off between Type I and Type II errors. Although we don't expect RBPT to outperform other models in terms of power, it would be interesting to see if it possesses comparable power, as suggested by the third plot in Figure 2.\n4. I am perplexed by the stark disparity between the performance of RBPT2 in low versus high-dimensional settings. It appears that an increase in the dimension of Z invariably leads to rejection of the null hypothesis, resulting in a rise in both power and Type I error rate. Could this be attributed to the need for a dimension normalization term in $b$? How are $b$ and $a$ generated per experiment, and how does $b^T Z$ scale with the dimension? Why doesn't this impact RBPT or CPT? After all, one of the main applications of regression-based conditional independence tests are high-dimensional settings, and it is important to address such concerns.\n\nLIMITATIONS:\n1. In the RBPT methodology, model misspecification is conceptualized as the difference between $Q^*(X|Z)$ and $P(X|Z)$, and between $g^*(x, y)$ and $f^*(x, y)$. This deviates from prior methods where the misspecification is solely defined between $g_1^*$ and $f_1^*$, and $g_2^*$ and $f_2^*$. While the latter defines misspecification between functions, the former is defined on an entire distribution.\nNotably, even if we posit that the disparity between the predicted $\\hat{Q}(X|Z)$ and $Q^*(X|Z)$ is infinitesimal, the divergence between $Q^*(X|Z)$ and $P(X|Z)$ remains far from negligible. Furthermore, the effectiveness of the entire method hinges on this misspecification gap ($\\Omega_{P,1}^{\\text{RBPT}}$) being smaller than the Jensen's gap, which places stringent restrictions on its applicability. While theoretically, this method relaxes the bound on the expected total variation between $Q^*$ and $P$ and sets it to $\\frac{\\Omega_{P,2}^{\\text{RBPT}}}{M \\cdot L}$, the $M$ and $L$ coefficients will be pretty large in practice. On the other hand, while RBPT2 appears to circumvent this issue, its theoretical grounding isn't as robust as that of RBPT, thereby reducing its perceived solidity.\nI believe this issue will become more prominent in real-world settings and the toy experiments do not reveal this core flaw of the method. Therefore, the authors should have mentioned this limitation or run extra experiments to show the effect of this gap.\n2. A prominent constraint associated with RBPT2 appears to be its performance when scaling $Z$. Unfortunately, this aspect is not addressed within the main paper text. It would be beneficial to incorporate some commentary on this issue, as it pertains to the most plausible plug-and-play application. Furthermore, I harbor concerns regarding the potential for error from the initial regression to percolate into the subsequent one, which could potentially escalate the overall margin of error. This is pointed out in Appendix A.6, and with the empirical results acting strangely with scaling (I also mentioned in the “Questions” section), I am concerned as to how much this can hinder the performance of RBPT2.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-25"}