SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Felipe Maia Polo
felipemaiapolo@gmail.com
Yuekai Sun
yuekai@umich.edu
Moulinath Banerjee
moulib@umich.edu
Conditional independence (CI) testing is a fundamental and challenging task in modern statistics and machine learning. Many modern methods for CI testing rely on powerful supervised learning methods to learn regression functions or Bayes predictors as an intermediate step; we refer to this class of tests as regression-based tests. Although these methods are guaranteed to control Type-I error when the supervised learning methods accurately estimate the regression functions or Bayes predictors of interest, their behavior is less understood when they fail due to misspecified inductive biases; in other words, when the employed models are not flexible enough or when the training algorithm does not induce the desired predictors. Then, we study the performance of regression-based CI tests under misspecified inductive biases. Namely, we propose new approximations or upper bounds for the testing errors of three regression-based tests that depend on misspecification errors. Moreover, we introduce the Rao-Blackwellized Predictor Test (RBPT), a regression-based CI test robust against misspecified inductive biases. Finally, we conduct experiments with artificial and real data, showcasing the usefulness of our theory and methods.
Conditional independence (CI) testing is fundamental in modern statistics and machine learning (ML). Its use has become widespread in several different areas, from (i) causal discovery [12, 24, 34, 11] and (ii) algorithmic fairness [28], to (iii) feature selection/importance [5, 37] and (iv) transfer learning [25]. Due to its growing relevance across different sub-fields of statistics and ML, new testing methods with different natures, from regression to simulation-based tests, are often introduced.
Regression-based CI tests, i.e., tests based on supervised learning methods, have become especially attractive in the past years due to (i) significant advances in supervised learning techniques, (ii) their suitability for high-dimensional problems, and (iii) their simplicity and easy application. However, regression-based tests usually depend on the assumption that we can accurately approximate the regression functions or Bayes predictors of interest, which is hardly true if (i) either the model classes are misspecified or if (ii) the training algorithms do not induce the desired predictors, i.e., if we have misspecified inductive biases. Misspecified inductive biases typically lead to inflated Type-I error rates but also can cause tests to be powerless. Even though these problems can frequently arise in practical situations, more attention should be given to theoretically understanding the effects of misspecification on CI hypothesis testing. Moreover, current regression-based methods are usually not designed to be robust against misspecification errors, making CI testing less reliable. In this work, we study the performance of three major regression-based conditional independence tests under misspecified inductive biases and propose the Rao-Blackwellized Predictor Test (RBPT), which is more robust against misspecification.
With more details, our main contributions are:
- We present new robustness results for three relevant regression-based conditional independence tests: (i) Significance Test of Feature Relevance (STFR) [7], (ii) Generalized Covariance Measure (GCM) test [31], and (iii) REgression with Subsequent Independence Test (RESIT) [42, 24, 12]. Namely, we derive approximations or upper bounds for the testing errors that explicitly depend on the level of misspecification.
- We introduce the Rao-Blackwellized Predictor Test (RBPT), a modification of the Significance Test of Feature Relevance (STFR) [7] test that is robust against misspecified inductive biases. In contrast with STFR and previous regression/simulation-based methods, the RBPT does not require models to be correctly specified to guarantee Type-I error control. We develop theoretical results about the RBPT, and experiments show that RBPT is robust when controlling Type-I error while maintaining non-trivial power.
Conditional independence testing. Let  $(X,Y,Z)$  be a random vector taking values in  $\mathcal{X}\times \mathcal{Y}\times \mathcal{Z}\subseteq$ $\mathbb{R}^{d_X\times d_Y\times d_Z}$  and  $\mathcal{P}$  be a fixed family of distributions on the measurable space  $(\mathcal{X}\times \mathcal{Y}\times \mathcal{Z},\mathcal{B})$ , where  $\mathcal{B} = \mathcal{B}(\mathcal{X}\times \mathcal{Y}\times \mathcal{Z})$  is the Borel  $\sigma$ -algebra. Let  $(X,Y,Z)\sim P$  and assume  $P\in \mathcal{P}$ . If  $\mathcal{P}_0\subset \mathcal{P}$  is the set of distributions in  $\mathcal{P}$  such that  $X\perp Y\mid Z$ , the problem of conditional independence testing can be expressed in the following way:
$$
H _ {0}: P \in \mathcal {P} _ {0} \quad H _ {1}: P \in \mathcal {P} \backslash \mathcal {P} _ {0}
$$
In this work, we also write  $H_0: X \perp Y \mid Z$  and  $H_1: X \not\perp Y \mid Z$ . We assume throughout that we have access to a dataset  $\mathcal{D}^{(n + m)} = \{(X_i, Y_i, Z_i)\}_{i=1}^{n + m}$  independent and identically distributed (i.i.d.) as  $(X, Y, Z)$ , where  $\mathcal{D}^{(n + m)}$  splits into a test set  $\mathcal{D}_{te}^{(n)} = \{(X_i, Y_i, Z_i)\}_{i=1}^n$  and a training set  $\mathcal{D}_{tr}^{(m)} = \{(X_i, Y_i, Z_i)\}_{i=n+1}^{n+m}$ . For convenience, we use the training set to fit models and the test set to conduct hypothesis tests, even though other approaches are possible.
Misspecified inductive biases in modern statistics and machine learning. Traditionally, misspecified inductive biases in statistics have been linked to the concept of model misspecification and then strictly related to the chosen model classes. For instance, if the best (Bayes) predictor for  $Y$  given  $X$ ,  $f^{*}$ , is a non-linear function of  $X$ , but we use a linear function to predict  $Y$ , then we say our model is misspecified because  $f^{*}$  is not in the class of linear functions. In modern machine learning and statistics, however, it is known that the training algorithm also plays a crucial role in determining the trained model. For example, it is known that training overparameterized neural networks using stochastic gradient descent bias the models towards functions with good generalization [14, 33]. In addition, D'Amour et al. [9] showed that varying hyperparameter values during training could result in significant differences in the patterns learned by the neural network. The researchers found, for instance, that models with different random initializations exhibit varying levels of out-of-distribution accuracy in predicting skin health conditions for different skin types, indicating that each model learned distinct features from the images. The sensitivity of the trained model concerning different training settings suggests that even models capable of universal approximation may not accurately estimate the target predictor if the training biases do not induce the functions we want to learn.
We present a toy experiment to empirically demonstrate how the training algorithm can prevent us from accurately estimating the target predictor even when the model class is correctly specified, leading to invalid significance tests. We work in the context of a high-dimensional (overparameterized) regression with a training set of 250 observations and 500 covariates. We use the Generalized Covariance Model (GCM) test[31] to conduct the CI test. The data are generated as
$$
Z \sim N (0, I _ {5 0 0}), X \mid Z \sim N \left(\beta_ {X} ^ {\top} Z, 1\right), \text {a n d} Y \mid X, Z \sim N \left(\beta_ {Y} ^ {\top} Z, 1\right),
$$
where the first five entries of  $\beta_{X}$  are set to 20, and the remaining entries are zero, while the last five entries of  $\beta_{Y}$  are set to 20, and the remaining entries are zero. This results in  $X$  and  $Y$  being conditionally independent given  $Z$  and depending on  $Z$  only through a small number of entries. Additionally,  $\mathbb{E}[X\mid Z] = \beta_X^\top Z$  and  $\mathbb{E}[Y\mid Z] = \beta_Y^\top Z$ , indicating that the linear model class is correctly specified. To
<sup>1</sup>Simulation-based tests usually rely on estimating conditional distributions.
See Appendix A.3 for more details
perform the GCM test, we use LASSO  $(\| \cdot \| _1$  penalization term added to empirical squared error) and the minimum-norm least-squares solution to fit linear models that predict  $X$  and  $Y$  given  $Z$ . In this problem, the LASSO fitting approach provides the correct inductive bias since  $\beta_{X}$  and  $\beta_{Y}$  are sparse.
We set the significance level to  $\alpha = 10\%$  and estimate the Type-I error rate for 100 different training sets. Figure 1 provides the Type-I error rate empirical distribution and illustrates that, despite using the same model class for both fitting methods, the training algorithm induces an undesired predictor in the minimum-norm case, implying an invalid test most of the time. On the other hand, the LASSO approach has better Type-I error control. In Appendix A, we give a similar example but using the Significance Test of Feature Relevance (STFR) [7].
In this work, we formalize the idea of misspecified inductive biases in the following way. Assume that a training algorithm  $\mathcal{A}$  is used to choose a model  $\hat{g}^{(m)} = \mathcal{A}(\mathcal{D}_{tr}^{(m)})$  from the class  $\mathcal{G}^{(m)}$ . We further assume that the sequence  $(\hat{g}^{(m)})_{m\in \mathbb{N}}$  converges to a limiting model  $g^{*}$  in a relevant context-dependent sense. We use different notions of convergence depending on the specific problem under consideration, which will be clear in the following sections. We say that  $g^{*}$  "carries misspecified inductive biases" if it does not equal the target Bayes predictor or regression function  $f^{*}$ . There are two possible reasons for  $g^{*}$  carrying misspecified
biases: either the limiting model class is small and does not include  $f^{*}$ , or the training algorithm  $\mathcal{A}$  cannot find the best possible predictor, even asymptotically.
Figure 1: Type-I error rate is contingent on the training algorithm and not solely on the model classes. Unlike the minimum-norm solution, the LASSO fit gives the correct inductive bias in high-dimensional regression, providing better Type-I error control.
Notation. We write  $\mathbb{E}_P$  and  $\mathsf{Var}_P$  for the expectation and variance of statistics computed using i.i.d. copies of  $(X,Y,Z) \sim P$ . Consequently,  $\mathbb{P}_P(A) = \mathbb{E}_P\mathbb{1}_A$ , where  $\mathbb{1}_A$  is the indicator of an event  $A$ . If  $\mathbb{E}_P$  and  $\mathsf{Var}_P$  are conditioned on some other statistics, we assume those statistics are also computed using i.i.d. samples from  $P$ . As usual,  $\Phi$  is the  $N(0,1)$  distribution function. If  $(a_m)_{m \in \mathbb{N}}$  and  $(b_m)_{m \in \mathbb{N}}$  are sequences of scalars, then  $a_m = o(b_m)$  is equivalent to  $a_m / b_m \to 0$  as  $m \to \infty$  and  $a_m = b_m + o(1)$  means  $a_m - b_m = o(1)$ . If  $(V^{(m)})_{m \in \mathbb{N}}$  is a sequence of random variables, where  $V^{(m)}$  as constructed using i.i.d. samples of  $P^{(m)} \in \mathcal{P}$  for each  $m$ , then (i)  $V^{(m)} = o_p(1)$  means that for every  $\varepsilon > 0$  we have  $\mathbb{P}_{P(m)}(|V^{(m)}| > \varepsilon) \to 0$  as  $m \to \infty$ , (ii)  $V^{(m)} = \mathcal{O}_p(1)$  means that for every  $\varepsilon > 0$  there exists a  $M > 0$  such that  $\sup_{m \in \mathbb{N}}\mathbb{P}_{P(m)}(|V^{(m)}| > M) < \varepsilon$ , (iii)  $V^{(m)} = a_m + o_p(1)$  means  $V^{(m)} - a_m = o_p(1)$ , (iv)  $V^{(m)} = o_p(a_m)$  means  $V^{(m)} / a_m = o_p(1)$ , and (v)  $V^{(m)} = \mathcal{O}_p(a_m)$  means  $V^{(m)} / a_m = \mathcal{O}_p(1)$ . Finally, let  $(V_P^{(m)})_{m \in \mathbb{N}, P \in \mathcal{P}}$  be a family of random variables that distributions explicitly depend on  $m \in \mathbb{N}$  and  $P \in \mathcal{P}$ . We give an example to clarify what we mean by "explicitly" depending on a specific distribution. Let  $V_P^{(m)} = \frac{1}{m} \sum_{i=1}^m (X_i - \mu_P)$ , where  $\mu_P = \mathbb{E}_P[X]$ . Here,  $V_P^{(m)}$  explicitly depends on  $P$  because of the quantity  $\mu_P$ . In this example,  $X_i$ 's outside the expectation can have an arbitrary distribution (unless stated), i.e., could be determined by  $P$  or any other distribution. With this context, (i)  $V_P^{(m)} = o_P(1)$  means that for every  $\varepsilon > 0$  we have  $\sup_{P \in \mathcal{P}}\mathbb{P}_P(|V_P^{(m)}| > \varepsilon) \to 0$  as  $m \to \infty$ , (ii)  $V_P^{(m)} = \mathcal{O}_P(1)$  means that for every  $\varepsilon > 0$  there exists a  $M > 0$  such that  $\sup_{m \in \mathbb{N}, P \in \mathcal{P}}\mathbb{P}_P(|V_P^{(m)}| > M) < \varepsilon$ , (iii)  $V_P^{(m)} = o_P(a_m)$  means  $V_P^{(m)} / a_m = o_P(1)$ , and (iv)  $V_P^{(m)} = \mathcal{O}_P(a_m)$  means  $V_P^{(m)} / a_m = \mathcal{O}_P(1)$ .
Related work. There is a growing literature on the problem of conditional independence testing regarding both theoretical and methodological aspects<sup>3</sup>. From the methodological point of view, there is a great variety of tests with different natures. Perhaps, the most important groups of tests are (i) simulation-based tests [5, 3, 4, 32, 35, 20], (ii) regression-based tests [40, 24, 42, 38, 31, 7], (iii) kernel-based tests [10, 8, 34, 30], and (iv) information-theoretic based tests [29, 16, 39]. Due to the advance of supervised and generative models in recent years, regression and simulation-based tests have become particularly appealing, especially when  $Z$  is not low-dimensional or discrete. A related but different line of research is constructing a lower confidence bound for conditional dependence of  $X$  and  $Y$  given  $Z$  [41]. In that work, the authors propose a method that relies on computing the conditional expectation of a possibly misspecified regression model, which can be related to
<sup>3</sup>See, for example, Marx and Vreeken [21], Shah and Peters [31], Li and Fan [19], Neykov et al. [23], Watson and Wright [37], Kim et al. [15], Shi et al. [32], Scutbon et al. [30], Tansey et al. [35], Zhang et al. [39], Ai et al. [1]
our method presented in Section 4. Despite the relationship between methods, their motivations, assumptions, and contexts are different.
Simulation-based tests depend on the fact that we can, implicitly or explicitly, approximate the conditional distributions  $P_{X|Z}$  or  $P_{Y|Z}$ . Two relevant simulation-based methods are the conditional randomization and conditional permutation tests (CRT/CPT) [5, 3, 4, 35]. For these tests, Berrett et al. [4] presents robustness results showing that we can approximately control Type I error even if our estimates for the conditional distributions are not perfect and we are under a finite-sample regime. However, it is also clear from their results that CRT and CPT might not control Type I error asymptotically when models for conditional distributions are misspecified. On the other hand, regression-based tests work under the assumption that we can accurately approximate the conditional expectations  $\mathbb{E}[X\mid Z]$  and  $\mathbb{E}[Y\mid Z]$  or other Bayes predictors, which is hardly true if the modeling and training inductive biases are misspecified. To the best of our knowledge, there are no published robustness results for regression-based CI tests like those presented by Berrett et al. [4]. We explore this literature gap.
3 Regression-based conditional independence tests under misspecified inductive biases
This section provides results for the Significance Test of Feature Relevance (STFR) [7]. Due to limited space, the results for the Generalized Covariance Measure (GCM) test [31] and the REgression with Subsequent Independence Test (RESIT) [42, 24, 12] are presented in Appendix A. From the results in Appendix A, one can easily derive a double robustness property for both GCM and RESIT, implying that not all models need to be correctly specified or trained with the correct inductive biases for Type-I error control.
3.1 Significance Test of Feature Relevance (STFR)
The STFR method studied by Dai et al. [7] offers a scalable approach for conducting conditional independence testing by comparing the performance of two predictors. To apply this method, we first train two predictors  $\hat{g}_1^{(m)}: \mathcal{X} \times \mathcal{Z} \to \mathcal{Y}$  and  $\hat{g}_2^{(m)}: \mathcal{Z} \to \mathcal{Y}$  on the training set  $\mathcal{D}_{tr}^{(m)}$  to predict  $Y$  given  $(X,Z)$  and  $Z$ , respectively. We assume that candidates for  $\hat{g}_2^{(m)}$  are models in the same class as  $\hat{g}_1^{(m)}$  but replacing  $X$  with null entries. Using samples from the test set  $\mathcal{D}_{te}^{(n)}$ , we conduct the test rejecting  $H_0: X \perp Y \mid Z$  if the statistic  $\Lambda^{(n,m)} \triangleq \sqrt{n}\bar{T}^{(n,m)}/\hat{\sigma}^{(n,m)}$  exceeds  $\tau_{\alpha} \triangleq \Phi^{-1}(1 - \alpha)$ , depending on the significance level  $\alpha \in (0,1)$ . We define  $\bar{T}^{(n,m)}$  and  $\hat{\sigma}^{(n,m)}$  as
$$
\bar {T} ^ {(n, m)} \triangleq \frac {1}{n} \sum_ {i = 1} ^ {n} T _ {i} ^ {(m)} \text {a n d} \hat {\sigma} ^ {(n, m)} \triangleq \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left(T _ {i} ^ {(m)}\right) ^ {2} - \left(\frac {1}{n} \sum_ {i = 1} ^ {n} T _ {i} ^ {(m)}\right) ^ {2} \right] ^ {1 / 2} \tag {3.1}
$$
with  $T_{i}^{(m)} \triangleq \ell(\hat{g}_{2}^{(m)}(Z_{i}), Y_{i}) - \ell(\hat{g}_{1}^{(m)}(X_{i}, Z_{i}), Y_{i}) + \varepsilon_{i}$ . Here,  $\ell$  is a loss function, typically used during the training phase, and  $\{\varepsilon_{i}\}_{i=1}^{n} \stackrel{iid}{\sim} N(0, \rho^{2})$  are small artificial random noises that do not let  $\hat{\sigma}^{(n,m)}$  vanish with a growing training set, thus allowing the asymptotic distribution of  $\Lambda^{(n,m)}$  to be standard normal under  $H_{0}: X \perp Y \mid Z$ . If the  $p$ -value is defined as  $p(\mathcal{D}_{te}^{(n)}, \mathcal{D}_{tr}^{(m)}) = 1 - \Phi(\Lambda^{(n,m)})$ , the test is equivalently given by
$$
\varphi_ {\alpha} ^ {\text {S T F R}} \left(\mathcal {D} _ {t e} ^ {(n)}, \mathcal {D} _ {t r} ^ {(m)}\right) \triangleq \left\{ \begin{array}{l l} 1, \text {i f} p \left(\mathcal {D} _ {t e} ^ {(n)}, \mathcal {D} _ {t r} ^ {(m)}\right) \leq \alpha \\ 0, \text {o t h e r w i s e} \end{array} \right. \tag {3.2}
$$
The rationale behind STFR is that if  $H_0: X \perp Y \mid Z$  holds, then  $\hat{g}_1^{(m)}$  and  $\hat{g}_2^{(m)}$  should have similar performance in the test set. On the other hand, if  $H_0$  does not hold, we expect  $\hat{g}_1^{(m)}$  to have significantly better performance, and then we would reject the null hypothesis. Said that, to control STFR's Type-I error, it is necessary that the risk gap between  $\hat{g}_1^{(m)}$  and  $\hat{g}_2^{(m)}$ ,  $\mathbb{E}_P[\ell(\hat{g}_2^{(m)}(Z), Y) \mid \mathcal{D}_{tr}^{(m)}] - \mathbb{E}_P[\ell(\hat{g}_1^{(m)}(X,Z), Y) \mid \mathcal{D}_{tr}^{(m)}]$ , under  $H_0$  vanishes as the training set size increases. Moreover, we need the risk gap to be positive for the test to have non-trivial power. These conditions can be met if the risk gap of  $g_{1,P}^*$  and  $g_{2,P}^*$ , the limiting models of  $\hat{g}_1^{(m)}$  and  $\hat{g}_2^{(m)}$ , is the same as the risk gap of the Bayes' predictors
$$
f _ {1, P} ^ {*} \triangleq \arg \min  _ {f _ {1}} \mathbb {E} _ {P} [ \ell (f _ {1} (X, Z), Y) ] \text {a n d} f _ {2, P} ^ {*} \triangleq \arg \min  _ {f _ {2}} \mathbb {E} _ {P} [ \ell (f _ {2} (Z), Y) ],
$$
where the minimization is done over the set of all measurable functions. However, the risk gap between  $\hat{g}_1^{(m)}$  and  $\hat{g}_2^{(m)}$  will typically not vanish if  $g_{1,P}^*$  and  $g_{2,P}^*$  are not the Bayes' predictors even under  $H_0$ . In general, we should expect  $g_{1,P}^*$  to perform better than  $g_{2,P}^*$  because the second predictor does not depend on  $X$ . Furthermore, their risk gap can be non-positive even if  $f_{1,P}^*$  performs better than  $f_{2,P}^*$ . In Appendix A.2, we present two examples in which model misspecification plays an important role when conducting STFR. The examples show that Type-I error control and/or power can be compromised due to model misspecification.
To derive theoretical results, we adapt the assumptions from Dai et al. [7]:
Assumption 3.1. There are functions  $g_{1,P}^*, g_{2,P}^*$ , and a constant  $\gamma > 0$  such that
$$
\begin{array}{l} \mathbb {E} _ {P} \left[ \ell (\hat {g} _ {2} ^ {(m)} (Z), Y) \mid \mathcal {D} _ {t r} ^ {(m)} \right] - \mathbb {E} _ {P} \left[ \ell \left(g _ {2, P} ^ {*} (Z), Y\right) \right] - \left(\mathbb {E} _ {P} \left[ \ell \left(\hat {g} _ {1} ^ {(m)} (X, Z), Y\right) \mid \mathcal {D} _ {t r} ^ {(m)} \right] - \mathbb {E} _ {P} \left[ \ell \left(g _ {1, P} ^ {*} (X, Z), Y\right) \right]\right) \\ = \mathcal {O} _ {\mathcal {P}} (m ^ {- \gamma}) \\ \end{array}
$$
Assumption 3.2. There exists a constant  $k > 0$  such that
$$
\mathbb {E} _ {P} \left[\left| T _ {1} ^ {(m)} \right| ^ {2 + k} \mid \mathcal {D} _ {t r} ^ {(m)} \right] = \mathcal {O} _ {\mathcal {P}} (1) a s m \rightarrow \infty
$$
Assumption 3.3. For every  $P \in \mathcal{P}$ , there exists a constant  $\sigma_P^2 > 0$  such that
$$
\mathsf {V a r} _ {P} \left[ T _ {1} ^ {(m)} \mid \mathcal {D} _ {t r} ^ {(m)} \right] - \sigma_ {P} ^ {2} = o _ {\mathcal {P}} (1) a s m \rightarrow \infty a n d \inf  _ {P \in \mathcal {P}} \sigma_ {P} ^ {2} > 0
$$
Finally, we present the results for this section. We start with an extension of Theorem 2 presented by Dai et al. [7] in the case of misspecified inductive biases.
Theorem 3.4. Suppose that Assumptions 3.1, 3.2, and 3.3 hold. If  $n$  is a function of  $m$  such that  $n \to \infty$  and  $n = o(m^{2\gamma})$  as  $m \to \infty$ , then
$$
\mathbb {E} _ {P} \left[ \varphi_ {\alpha} ^ {\text {S T F R}} \left(\mathcal {D} _ {t e} ^ {(n)}, \mathcal {D} _ {t r} ^ {(m)}\right) \right] = 1 - \Phi \left(\tau_ {\alpha} - \sqrt {\frac {n}{\sigma_ {P} ^ {2}}} \Omega_ {P} ^ {\text {S T F R}}\right) + o (1)
$$
where  $o(1)$  denotes uniform convergence over all  $P \in \mathcal{P}$  as  $m \to \infty$  and
$$
\Omega_ {P} ^ {\mathrm {S T F R}} \triangleq \mathbb {E} _ {P} [ \ell (g _ {2, P} ^ {*} (Z), Y) ] - \mathbb {E} _ {P} [ \ell (g _ {1, P} ^ {*} (X, Z), Y) ]
$$
Theorem 3.4 demonstrates that the performance of STFR depends on the limiting models  $g_{1,P}^{*}$  and  $g_{2,P}^{*}$ . Specifically, if  $\Omega_P^{\mathrm{STFR}} > 0$ , then  $\mathbb{E}_P[\varphi_\alpha^{\mathrm{STFR}}(\mathcal{D}_{te}^{(n)},\mathcal{D}_{tr}^{(m)})]\to 1$  even if  $H_0:X\perp Y|Z$  holds. In practice, we should expect  $\Omega_P^{\mathrm{STFR}} > 0$  because of how we set the class for  $\hat{g}_2^{(m)}$ . In contrast, we could have  $\Omega_P^{\mathrm{STFR}}\leq 0$ , and then  $\mathbb{E}_P[\varphi_\alpha^{\mathrm{STFR}}(\mathcal{D}_{te}^{(n)},\mathcal{D}_{tr}^{(m)})]\leq \alpha +o(1)$ , even if the gap between Bayes' predictors is positive. See examples in Appendix A.2 for both scenarios. Next, we provide Corollary 3.6 to clarify the relationship between testing and misspecification errors. This corollary formalizes the intuition that controlling Type-I error is directly related to misspecification of  $g_{2,P}^{*}$ , while minimizing Type-II error is directly related to misspecification of  $g_{1,P}^{*}$ .
Definition 3.5. For a distribution  $P$  and a loss function  $\ell$ , define the misspecification gaps:
$$
\Delta_ {1, P} \triangleq \mathbb {E} _ {P} [ \ell (g _ {1, P} ^ {*} (X, Z), Y) ] - \mathbb {E} _ {P} [ \ell (f _ {1, P} ^ {*} (X, Z), Y) ] a n d \Delta_ {2, P} \triangleq \mathbb {E} _ {P} [ \ell (g _ {2, P} ^ {*} (Z), Y) ] - \mathbb {E} _ {P} [ \ell (f _ {2, P} ^ {*} (Z), Y) ]
$$