SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The misspecification gaps defined in Definition 3.5 quantify the difference between the limiting predictors  $g_{1,P}^{*}$  and  $g_{2,P}^{*}$  and the Bayes predictors  $f_{1,P}^{*}$  and  $f_{2,P}^{*}$ , i.e., give a misspecification measure for  $g_{1,P}^{*}$  and  $g_{2,P}^{*}$ . Corollary 3.6 implies that the STFR controls Type-I error asymptotically if  $\Delta_{2,P} = 0$ , and guarantees non-trivial power if the degree of misspecification of  $g_{1,P}^{*}$  is not large compared to the performance difference of the Bayes predictors  $\Delta_P$ , that is, when  $\Delta_P - \Delta_{1,P} > 0$ .
Corollary 3.6 (Bounding testing errors). Suppose we are under the conditions of Theorem 3.4.
(Type-I error) If  $H_0: X \perp Y \mid Z$  holds, then
$$
\mathbb {E} _ {P} \left[ \varphi_ {\alpha} ^ {\text {S T F R}} \left(\mathcal {D} _ {t e} ^ {(n)}, \mathcal {D} _ {t r} ^ {(m)}\right) \right] \leq 1 - \Phi \left(\tau_ {\alpha} - \sqrt {\frac {n}{\sigma_ {P} ^ {2}}} \Delta_ {2, P}\right) + o (1)
$$
where  $o(1)$  denotes uniform convergence over all  $P\in \mathcal{P}_0$  as  $m\to \infty$
(Type-II error) In general, we have
$$
1 - \mathbb {E} _ {P} \left[ \varphi_ {\alpha} ^ {\text {S T F R}} \left(\mathcal {D} _ {t e} ^ {(n)}, \mathcal {D} _ {t r} ^ {(m)}\right) \right] \leq \Phi \left(\tau_ {\alpha} - \sqrt {\frac {n}{\sigma_ {P} ^ {2}}} \left(\Delta_ {P} - \Delta_ {1, P}\right)\right) + o (1)
$$
where  $o(1)$  denotes uniform convergence over all  $P\in \mathcal{P}$  as  $m\to \infty$  and  $\Delta_P\triangleq \mathbb{E}_P[\ell (f_{2,P}^* (Z),Y)] - \mathbb{E}_P[\ell (f_{1,P}^* (X,Z),Y)]$
4We assume  $f_{1,P}^{*}$  and  $f_{2,P}^{*}$  to be well-defined and unique.
4 A robust regression-based conditional independence test
This section introduces the Rao-Blackwellized Predictor Test (RBPT), a misspecification robust conditional independence test based on ideas from both regression and simulation-based CI tests. The RBPT assumes that we can implicitly or explicitly approximate the conditional distribution of  $X \mid Z$  and does not require inductive biases to be correctly specified. Because RBPT involves comparing the performance of two predictors and requires an approximation of the distribution of  $X \mid Z$ , we can directly compare it with the STFR [7] and the conditional randomization/permutation tests (CRT/CPT) [5, 4]. The RBPT can control Type-I error under relatively weaker assumptions compared to other tests, allowing some misspecified inductive biases.
The RBPT can be summarized as follows: (i) we train  $\hat{g}^{(m)}$  that predicts  $Y$  given  $(X,Z)$  using  $\mathcal{D}_{tr}^{(m)}$ ; (ii) we obtain the Rao-Blackwellized predictor  $h^{(m)}$  by smoothing  $\hat{g}^{(m)}$ , i.e.,
$$
h ^ {(m)} (z) \triangleq \int \hat {g} ^ {(m)} (x, z) \mathrm {d} P _ {X \mid Z = z} (x),
$$
then (iii) compare its performance with  $\hat{g}^{(m)}$ 's using the test set  $\mathcal{D}_{te}^{(n)}$  and a convex loss function  $\ell$  (not necessarily used to train  $\hat{g}^{(m)}$ ), and (iv) if the performance of  $\hat{g}^{(m)}$  is statistically better than  $h^{(m)}$ 's, we reject  $H_0: X \perp Y \mid Z$ . The proce-
dure described here bears a resemblance to the Rao-Blackwellization of estimators. In classical statistics, the Rao-Blackwell theorem [17] states that by taking the conditional expectation of an estimator with respect to a sufficient statistic, we can obtain a better estimator if the loss function is convex. In our case, the variable  $Z$  can be seen as "sufficient statistic" for  $Y$  under the assumption of conditional independence  $H_0: X \perp Y \mid Z$ . If  $H_0$  holds and the loss  $\ell(\hat{y}, y)^4$  is convex in its first argument, we can show using Jensen's inequality that the resulting model  $h^{(m)}$  has a lower risk relative to the initial model  $\hat{g}^{(m)}$ , i.e.,  $\mathbb{E}_P[\ell(h^{(m)}(Z), Y)] \mid \mathcal{D}_{tr}^{(m)}] - \mathbb{E}_P[\ell(\hat{g}^{(m)}(X, Z), Y) \mid \mathcal{D}_{tr}^{(m)}] \leq 0$ . Then, the risk gap in RBPT is non-positive under  $H_0$  in contrast with STFR's risk gap,
Algorithm 1: Obtaining  $p$ -value for the RBPT
Input: (i) Test set  $\mathcal{D}_{te}^{(n)} = \{(X_i,Y_i,Z_i)\}_{i = 1}^n$ , (ii) initial predictor  $\hat{g}^{(m)}$ , (iii) conditional distribution estimate  $\hat{Q}_{X|Z}^{(m)}$ , (iv) convex loss function  $\ell$ ;
Output: p-value  $p$
For each  $i \in [n]$ , get
$$
\hat {h} ^ {(m)} (Z _ {i}) = \int \hat {g} ^ {(m)} (x, Z _ {i}) \mathrm {d} \hat {Q} _ {X | Z = Z _ {i}} ^ {(m)} (x);
$$
Compute  $\Xi^{(n,m)}\triangleq \sqrt{n}\bar{T}^{(n,m)} / \hat{\sigma}^{(n,m)}$  where  $\bar{T}^{(n,m)}\triangleq \frac{1}{n}\sum_{i = 1}^{n}T_{i}^{(m)}$  with
$$
T _ {i} ^ {(m)} \triangleq \ell (\hat {h} ^ {(m)} (Z _ {i}), Y _ {i}) - \ell (\hat {g} ^ {(m)} (X _ {i}, Z _ {i}), Y _ {i})
$$
and  $\hat{\sigma}^{(n,m)}$  being  $\{T_i\}$  's sample std dev (Eq. 3.1).
return  $p = 1 - \Phi (\Xi^{(n,m)})$
which we should expect to be always non-negative given the definition of  $\hat{g}_2^{(m)}$  in that case. That fact negatively biases the RBPT test statistic, enabling better Type-I error control.
In practice, we cannot compute  $h^{(m)}$  exactly because  $P_{X|Z}$  is usually unknown. Then, we use an approximation  $\hat{Q}_{X|Z}^{(m)}$ , which can be given explicitly, e.g., using probabilistic classifiers or conditional density estimators [13], or implicitly, e.g., using generative adversarial networks (GANs) [22, 3]. We assume that  $\hat{Q}_{X|Z}^{(m)}$  is obtained using the training set. The approximated  $h^{(m)}$  is
$$
\hat {h} ^ {(m)} (z) \triangleq \int \hat {g} ^ {(m)} (x, z) \mathrm {d} \hat {Q} _ {X | Z = z} ^ {(m)} (x)
$$
where the integral can be solved numerically in case  $\hat{Q}_{X|Z}^{(m)}$  has a known probability mass function or Lebesgue density (e.g., via trapezoidal rule) or via Monte Carlo integration in case we can only sample from  $\hat{Q}_{X|Z}^{(m)}$ . Finally, for a fixed significance level  $\alpha \in (0,1)$ , the test  $\varphi_{\alpha}^{\mathrm{RBPT}}$  is given by Equation 3.2 where the  $p$ -value is obtained via Algorithm 1.
Before presenting RBPT results, we introduce some assumptions. Let  $Q_{X|Z}^{*}$  represent the limiting model for  $\hat{Q}_{X|Z}^{(m)}$ . The conditional distribution  $Q_{X|Z}^{*}$  depends on the underlying distribution  $P$ , but we omit additional subscripts for ease of notation. Assumption 4.1 defines the limiting models and fixes a convergence rate.
5The loss function  $\ell (\hat{y},y)$  needs to be convex with respect to its first entry  $(\hat{y})$  for all  $y$ . Both the test set and training set sizes, and the loss function  $\ell$  can be chosen using the heuristics introduced by Dai et al. [7].
Assumption 4.1. There is a function  $g_{P}^{*}$ , a conditional distribution  $Q_{X|Z}^{*}$ , and a constant  $\gamma > 0$  s.t.
$$
\mathbb {E} _ {P} \left[ \left\| \hat {g} ^ {(m)} (Z) - g _ {P} ^ {*} (Z) \right\| _ {2} ^ {2} \Big | \mathcal {D} _ {t r} ^ {(m)} \right] = \mathcal {O} _ {\mathcal {P}} (m ^ {- \gamma}) a n d \mathbb {E} _ {P} \left[ d _ {I V} (\hat {Q} _ {X | Z} ^ {(m)}, Q _ {X | Z} ^ {*}) \mid \mathcal {D} _ {t r} ^ {(m)} \right] = \mathcal {O} _ {\mathcal {P}} (m ^ {- \gamma})
$$
where  $d_{TV}$  denotes the total variation (TV) distance. Additionally, assume that both  $\hat{Q}_{X|Z}^{(m)}$  and  $Q_{X|Z}^*$  are dominated by a common  $\sigma$ -finite measure which does not depend on  $Z$  or  $m$ .
The common dominating measure in Assumption 4.1 could be, for example, the Lebesgue measure in  $\mathbb{R}^{d_X}$ . Next, Assumption 4.2 imposes additional constraints on the limiting model  $Q_{X|Z}^*$ . Under that assumption, the limiting misspecification level must be uniformly bounded over all  $P\in \mathcal{P}$ .
Assumption 4.2. For all  $P \in \mathcal{P}$ , the chi-square divergence
$$
\chi^ {2} \left(Q _ {X | Z} ^ {*} | | P _ {X | Z}\right) \triangleq \int \frac {\mathrm {d} Q _ {X | Z} ^ {*}}{\mathrm {d} P _ {X | Z}} \mathrm {d} Q _ {X | Z} ^ {*} - 1
$$
is a well-defined integrable random variable and  $\sup_{P\in \mathcal{P}}\mathbb{E}_P\left[\chi^2\left(Q_{X|Z}^*\| P_{X|Z}\right)\right] < \infty$
Now, assume  $\hat{g}^{(m)}$  is chosen from a model class  $\mathcal{G}^{(m)}$ . Assumption 4.3 imposes constraints on the model classes  $\{\mathcal{G}^{(m)}\}$  and loss function  $\ell$ .
Assumption 4.3. Assume (i)  $\sup_{g\in \mathcal{G}^{(m)}}\sup_{(x,z)\in \mathcal{X}\times \mathcal{Z}}\| g(x,z)\| _1\leq M < \infty$  for some real and positive  $M > 0$ , uniformly for all  $m$ , and (ii) that  $\ell$  is a  $L$ -Lipschitz loss function (with respect to its first argument) for a certain  $L > 0$ , i.e., for any  $\hat{y},\hat{y}^{\prime},y\in \mathcal{Y}$ , we have that  $|\ell (\hat{y},y) - \ell (\hat{y}^{\prime},y)|\leq L\| \hat{y} -\hat{y}^{\prime}\|_{2}$ .
Assumption 4.3 is valid by construction since we choose  $\mathcal{G}^{(m)}$  and the loss function  $\ell$ . That assumption is satisfied when, for example, (a) models in  $\cup_{m}\mathcal{G}^{(m)}$  are uniformly bounded, (b)  $\ell (\hat{y},y) = \| \hat{y} -y\| _p^p$  with  $p\geq 1$ , and (c)  $\mathcal{V}$  is a bounded subset of  $\mathbb{R}^{d_Y}$ , i.e., in classification problems and most of the practical regression problems. The loss  $\ell (\hat{y},y) = \| \hat{y} -y\| _p^p$ , with  $p\geq 1$ , is also convex with respect to its first entry and then a suitable loss for RBPT. It is important to emphasize that  $\ell$  does not need to be the same loss function used during the training phase. For example, we could use  $\ell (\hat{y},y) = \| \hat{y} -y\| _2^2$  in classification problems, where  $y$  is a one-hot encoded class label and  $\hat{y}$  is a vector of predicted probabilities given by a model trained using the cross-entropy loss.
Theorem 4.4. Suppose that Assumptions 3.2, 3.3, 4.1, 4.2, and 4.3 hold. If  $n$  is a function of  $m$  such that  $n \to \infty$  and  $n = o(m^{\gamma})$  as  $m \to \infty$ , then
$$
\mathbb {E} _ {P} \left[ \varphi_ {\alpha} ^ {\mathrm {R B P T}} \left(\mathcal {D} _ {t e} ^ {(n)}, \mathcal {D} _ {t r} ^ {(m)}\right) \right] = 1 - \Phi \left(\tau_ {\alpha} - \sqrt {\frac {n}{\sigma_ {P} ^ {2}}} \Omega_ {P} ^ {\mathrm {R B P T}}\right) + o (1)
$$
where  $o(1)$  denotes uniform convergence over all  $P\in \mathcal{P}$  as  $m\to \infty$  and  $\Omega_P^{\mathrm{RBPT}} = \Omega_{P,1}^{\mathrm{RBPT}} - \Omega_{P,2}^{\mathrm{RBPT}}$  with
$$
\Omega_ {P, 1} ^ {\mathrm {R B P T}} \triangleq \mathbb {E} _ {P} \left[ \ell \left(\int g _ {P} ^ {*} (x, Z) \mathrm {d} Q _ {X | Z} ^ {*} (x), Y\right) \right] - \mathbb {E} _ {P} \left[ \ell \left(\int g _ {P} ^ {*} (x, Z) \mathrm {d} P _ {X | Z} (x), Y\right) \right]
$$
$$
\underbrace{\Omega_{P,2}^{\mathrm{RBPT}}}_{\substack{\text{Jensen's gap}}} \triangleq \mathbb{E}_{P}\left[\ell (g_{P}^{*}(X,Z),Y)\right] - \mathbb{E}_{P}\left[\ell \left(\int g_{P}^{*}(x,Z) \mathrm{d}P_{X|Z}(x),Y\right)\right]
$$
When  $H_0: X \perp Y \mid Z$  holds and  $\ell$  is a strictly convex loss function (w.r.t. its first entry), we have that  $\Omega_{P,2}^{\mathrm{RBPT}} > 0$ , allowing some room for the "incorrectness" of  $Q_{X|Z}^{*}$ . That is, from Theorem 4.4, as long as  $\Omega_{P}^{\mathrm{RBPT}} \leq 0$ , i.e., if  $Q_{X|Z}^{*}$ 's incorrectness (measured by  $\Omega_{P,1}^{\mathrm{RBPT}}$ ) is not as big as Jensen's gap  $\Omega_{P,2}$ , RBPT has asymptotic Type-I error control. Uniform asymptotic Type-I error control is possible if  $\sup_{P \in \mathcal{P}_0} \Omega_{P}^{\mathrm{RBPT}} \leq 0$ . This is a great improvement of previous work (e.g., STFR, GCM, RESIT, CRT, CPT) since there is no need for any model to converge to the ground truth if  $\Omega_{P,1}^{\mathrm{RBPT}} \leq \Omega_{P,2}^{\mathrm{RBPT}}$ , which is a weaker condition. See however that a small  $\Omega_{P,2}^{\mathrm{RBPT}}$  reduces the room for  $Q_{X|Z}^{*}$  incorrectness. In the extreme case, when  $g_{P}^{*}$  is the Bayes predictor, and therefore does not depend on  $X$  under  $H_0$ , we need  $Q_{X|Z}^{*} = P_{X|Z}$  almost surely. On the other hand, if  $g_{P}^{*}$  is close to the Bayes predictor, RBPT has better power. That imposes an expected trade-off between Type-I error control and
In practice, we do not need  $\ell$  to be strictly convex for the Jensen's gap to be positive. Assuming that  $g_{P}^{*}$  depends on  $X$  under  $H_0$  is necessary, though. That condition is usually true when  $g_{P}^{*}$  is not the Bayes predictor.
In this case, Assumption 3.3 is not true. We need to include artificial noises in the definition of  $T_{i}$  as it was done in STFR by Dai et al. [7] in case we have high confidence that models converge to the ground truth.
Figure 2: Type-I error rates  $(c = 0)$ . In the first two plots, we set  $\theta = 0$  for RBPT, permitting Type-I error control across different  $d_{Z}$  values (Theorem 4.4), while RBPT2 allows Type-I error control for moderate  $d_{Z}$ . All the baselines fail to control Type-I errors regardless of  $d_{Z}$ . The last two plots illustrate that CRT emerges as the least robust test in this context, succeeded by RBPT and CPT.
power. To make a comparison with Berrett et al. [4] 's results in the case of CRT and CPT, we can express our remark in terms of the TV distance between  $Q_{X|Z}^{*}$  and  $P_{X|Z}$ . It can be shown that if  $\mathbb{E}_P[d_{\mathrm{TV}}(Q_{X|Z}^*, P_{X|Z})] \leq \Omega_{P,2}^{\mathrm{RBPT}} / (2ML)$ , then Type-I error control is guaranteed (see Appendix A.5). This contrasts with Berrett et al. [4] 's results because  $\mathbb{E}_P[d_{\mathrm{TV}}(Q_{X|Z}^*, P_{X|Z})] = 0$  is not needed.
We conclude this section with some relevant observations related to the RBPT.
On RBPT's power. Like STFR, non-trivial power is guaranteed if the predictor  $g_{P}^{*}$  is good enough. Indeed, the second part of Corollary 3.6 can be applied for an upper bound on RBPT's Type-II error.
Semi-supervised learning. Let  $Y$  denote a label variable. Situations in which unlabeled samples  $(X_{i}, Z_{i})$  are abundant while labeled samples  $(X_{i}, Y_{i}, Z_{i})$  are scarce happen in real applications of conditional independence testing [5, 4]. RBPT is well suited for those cases because the practitioner can use the abundant data to estimate  $P_{X|Z}$  flexibly. The semi-supervised learning scenario also applies to RBPT2, which we describe next.
Running RBPT when it is hard to estimate  $P_{X|Z}$ : the RBPT2. There might be situations in which it is hard to estimate the full conditional distribution  $P_{X|Z}$ . An alternative approach would be estimating the Rao-Blackwellized predictor directly using a second regressor. After training  $\hat{g}^{(m)}$ , we could use the training set, organizing it in pairs  $\{(Z_i, \hat{g}^{(m)}(Z_i, X_i))\}$ , to train a second predictor  $\hat{h}^{(m)}$  to predict  $\hat{g}^{(m)}(Z, X)$  given  $Z$ . That predictor could be trained to minimize the mean-squared error. The model  $\hat{h}^{(m)}$  should be more complex than  $\hat{g}^{(m)}$ , in the sense that we should hope that the first model performs better than the second under  $H_0$  in predicting  $Y$ . Consequently, this approach is effective when unlabeled samples are abundant, and we can train  $\hat{h}$  using both unlabeled data and the given training set. After obtaining  $\hat{h}^{(m)}$ , the test is conducted normally. We name this version of RBPT as "RBPT2". We include a note on how to adapt Theorem 4.4 for RBPT2 in Appendix A.6.
We empirically<sup>8</sup> analyze RBPT/ARBPT2 in the following experiments and compare them with relevant benchmarks, especially when the used models are misspecified. We assume  $\alpha = 10\%$  and  $\ell(\hat{y}, y) = (\hat{y} - y)^2$ . The benchmarks encompass STFR [7], GCM [31], and RESIT [42], which represent regression-based CI tests. Furthermore, we examine the conditional randomization/permutation tests (CRT/CPT) [5, 4] that necessitate the estimation of  $P_{X|Z}$ .
Artificial data experiments. Our setup takes inspiration from Berrett et al. [4], and the data is generated as
$$
Z \sim N \left(0, I _ {d _ {Z}}\right), \quad X \mid Z \sim N \left(\left(b ^ {\top} Z\right) ^ {2}, 1\right), \quad Y \mid X, Z \sim N \left(c X + a ^ {\top} Z + \gamma \left(b ^ {\top} Z\right) ^ {2}, 1\right).
$$
Here,  $d_Z$  denotes the dimensionality of  $Z$ ,  $a$  and  $b$  are sampled from  $N(0, I_{d_Z})$ , the constant  $c$  determines the conditional dependence of  $X$  and  $Y$  on  $Z$ , and the parameter  $\gamma$  dictates the hardness of conditional independence testing: a non-zero  $\gamma$  implies potential challenges in Type-I error control as there might be a pronounced marginal dependence between  $X$  and  $Y$  under  $H_0$ . Moreover, the training (resp. test) dataset consists of 800 (resp. 200) entries, and every predictor we employ operates on linear regression. RESIT employs Spearman's correlation between residuals as a test statistic while
$^{8}$ Code in https://github.com/felipemaiapolo/cit.
CRT and CPT $^9$  deploy STFR's test statistic, all of them with  $p$ -values determined by conditional sampling/permutations executed 100 times ( $B = 100$ ), considering  $\hat{Q}_{X|Z} = N((b^\top Z)^2 + \theta, 1)$ . The value of  $\theta$  gives the error level in approximating  $P_{X|Z}$ . To get  $\hat{h}$  in the two variations of RBPT, we either use  $\hat{Q}_{X|Z}$  (RBPT) or kernel ridge regression (KRR) equipped with a polynomial kernel to predict  $\hat{g}_1(X,Z)$  from  $Z$  (RBPT2). We sample generative parameters  $(a,b)$  five times using different random seeds, and for each iteration, we conduct 480 Monte Carlo simulations to estimate Type-I error and power. The presented results are the average ( $\pm$  standard deviation) estimated Type-I error/power across iterations.
Figure 3: Making RBPT2 more robust using unlabeled data. With  $d_Z = 40$ , we gradually increase the unlabeled sample size from 0 to 1000 when fitting  $\hat{h}$ . The results show that a larger unlabeled sample size leads to effective Type-I error control. Even though we present this result for RBPT2, the same pattern is expected for RBPT in the presence of unlabeled data.
In Figure 2 (resp. 4) we compare our methods' Type-I error rates (with  $c = 0$ ) (resp. power) against benchmarks. Regarding Figure 2, we focus on regression-based tests (STFR, GCM, and RESIT) in the first two plots and on simulation-based tests (CRT and CPT) in the last two plots. Regarding the first two plots, it is not straightforward to compare the level of misspecification between our methods and the benchmarks, so we use this as an opportunity to illustrate Theorem 4.4 and results from Section 3 and Appendix A. Fixing  $\theta = 0$  for RBPT, the Rao-Blackwellized predictor  $h$  is perfectly obtained, permitting Type-I error control regardless of the chosen  $d_Z$ . Using KRR for RBPT2 makes  $\hat{h}$  close to  $h$  when  $d_Z$  is not big and permits Type-I error control. When  $d_Z$  is big, more data is needed to fit  $\hat{h}$ , which can be accomplished using unlabeled data, as demonstrated in Figure 3 and commented in Section 4. On the other hand, Type-I error control is always violated for STFR, GCM, and RESIT when  $\gamma$  grows. Regarding the final two plots, we can more readily assess the robustness of
the methods when discrepancies arise between  $\hat{Q}_{X|Z}$  and  $P_{X|Z}$  as influenced by varying  $\theta$ . Figure 2 illustrates that CRT is the least robust test in this context, succeeded by RBPT and CPT. In Figure 4, we investigate how powerful RBPT and RBPT2 can be in practice when  $d_Z = 30$ . We compare our methods with CPT (when  $\theta = 0$ ), which seems to have practical robustness against misspecified inductive biases. Figure 4 shows that RBPT2 and CPT have similar power while RBPT is slightly more conservative.
Some concluding remarks are needed. First, RBPT and RBPT2 have shown to be practical and robust alternatives to conditional independence testing, exhibiting reasonable Type-I error control, mainly when employed in conjunction with a large unlabeled dataset, and power. Second, while CPT demonstrates notable robustness and relatively good power, its practicality falls short compared to RBPT (or RBPT2). This is because CPT needs a known density functional form for  $\hat{Q}_{X|Z}$  (plus the execution of MCMC chains) whereas RBPT (resp. RBPT2) can rely on conventional Monte Carlo integration using samples from  $\hat{Q}_{X|Z}$  (resp. supervised learning).
Real data experiments. For our subsequent experiments, we employ the car insurance dataset examined by Angwin et al. [2]. This dataset encompasses four US states (California, Illinois, Missouri, and Texas) and includes information from numerous insurance providers compiled at the ZIP code granularity. The data offers a risk metric and the insurance price levied on a hypothetical customer with consistent attributes from every ZIP
code. ZIP codes are categorized as either minority or non-minority, contingent on the percentage of non-white residents. The variables in consideration are  $Z$ , denoting the driving risk;  $X$ , an indicator for minority ZIP codes; and  $Y$ , signifying the insurance price. A pertinent question revolves around
Figure 4: Power curves for different methods. We compare our methods with CPT (when  $\theta = 0$ ), which seems to have practical robustness against misspecified inductive biases. RBPT2 and CPT have similar power, while RBPT is slightly more conservative.
For CPT execution, the Python script at http://www.stat.uchicago.edu/~rina/cpt.html was used, operating a single MCMC chain and preserving all other parameters as defined by the original authors.
Figure 5: Type-I error control and power analysis using car insurance data [2]. The first plot shows that RBPT and RBPT2 have better control over Type-I errors compared to all other methods, including CPT. The second plot reveals that all methods give the same qualitative result, corroborating the findings of Angwin et al. [2], suggesting that RBPT and RBPT2 can have good power while being more robust to Type-I errors.
the validity of the null hypothesis  $H_0: X \perp Y \mid Z$ , essentially questioning if demographic biases influence pricing.
We split our experiments into two parts. In the initial part, our primary goal is to compare the Type-I error rate across various tests. To ensure that  $H_0$  is valid, we discretize  $Z$  into twenty distinct values and shuffle the  $Y$  values corresponding to each discrete  $Z$  value. If a test maintains Type-I error control, we expect it to reject  $H_0$  for at most  $\alpha = 10\%$  of the companies in each state. In the second part, we focus on assessing the power of our methods. Given our lack of ground truth, we qualitatively compare RBPT and RBPT2 findings with those obtained by baseline methods and delineated by Angwin et al. [2], utilizing a detailed and multifaceted approach. In this last experiment, we aggregate the analysis for each state without conditioning on the firm. We resort to logistic regression for estimating the distribution of  $X \mid Z$  used by RBPT, GCM, CRT, and CPT. For RBPT2, we use a CatBoost regressor [26] to yield the Rao-Blackwellized predictor. We omit RESIT in this experiment as the additive model assumption is inappropriate. Both CRT and CPT methods utilize the same test metrics as STFR. The first plot<sup>10</sup> of Figure 5 shows that RBPT and RBPT2 methods have better control over Type-I errors compared to all other methods, including CPT. The second plot reveals that all methods give the same qualitative result that discrimination against minorities in ZIP codes is most evident in Illinois, followed by Texas, Missouri, and California. These findings corroborate with those of Angwin et al. [2], indicating that our methodology has satisfactory power while maintaining a robust Type-I error control.
In this work, we theoretically and empirically showed that widely used regression-based conditional independence tests are sensitive to the specification of inductive biases. Furthermore, we introduced the Rao-Blackwellized Predictor Test (RBPT), a misspecification-robust conditional independence test. RBPT is theoretically grounded and has been shown to perform well in practical situations compared to benchmarks.