Title: Covariate Shift Corrected Conditional Randomization Test
Venue: NeurIPS 2024
Accepted: yes
Keywords: conditional independence test

conditional randomization test

model-X framework

surrogate variable

importance weight

control covariates
Abstract: Conditional independence tests are crucial across various disciplines in determining the independence of an outcome variable $Y$ from a treatment variable $X$, conditioning on a set of confounders $Z$. The Conditional Randomization Test (CRT) offers a powerful framework for such testing by assuming known distributions of $X \mid Z$; it controls the Type-I error exactly, allowing for the use of flexible, black-box test statistics. In practice, testing for conditional independence often involves using data from a source population to draw conclusions about a target population. This can be challenging due to covariate shift---differences in the distribution of $X$, $Z$, and surrogate variables, which can affect the conditional distribution of $Y \mid X, Z$---rendering traditional CRT approaches invalid. To address this issue, we propose a novel Covariate Shift Corrected Pearson Chi-squared Conditional Randomization (csPCR) test. This test adapts to covariate shifts by integrating importance weights and employing the control variates method to reduce variance in the test statistics and thus enhance power. Theoretically, we establish that the csPCR test controls the Type-I error asymptotically. Empirically, through simulation studies, we demonstrate that our method not only maintains control over Type-I errors but also exhibits superior power, confirming its efficacy and practical utility in real-world scenarios where covariate shifts are prevalent. Finally, we apply our methodology to a real-world dataset to assess the impact of a COVID-19 treatment on the 90-day mortality rate among patients.

Covariate Shift Corrected Conditional Randomization Test

Abstract

Conditional independence tests are crucial across various disciplines in determining the independence of an outcome variable Y from a treatment variable X , conditioning on a set of confounders Z . The Conditional Randomization Test (CRT) offers a powerful framework for such testing by assuming known distributions of X | Z ; it controls the Type-I error exactly, allowing for the use of flexible, black-box test statistics. In practice, testing for conditional independence often involves using data from a source population to draw conclusions about a target population. This can be challenging due to covariate shift-differences in the distribution of X , Z , and surrogate variables, which can affect the conditional distribution of Y | X,Z -rendering traditional CRT approaches invalid. To address this issue, we propose a novel Covariate Shift Corrected Pearson Chi-squared Conditional Randomization (csPCR) test. This test adapts to covariate shifts by integrating importance weights and employing the control variates method to reduce variance in the test statistics and thus enhance power. Theoretically, we establish that the csPCR test controls the Type-I error asymptotically. Empirically, through simulation studies, we demonstrate that our method not only maintains control over Type-I errors but also exhibits superior power, confirming its efficacy and practical utility in real-world scenarios where covariate shifts are prevalent. Finally, we apply our methodology to a real-world dataset to assess the impact of a COVID-19 treatment on the 90-day mortality rate among patients.

1 Introduction

Conditional independence tests are important across diverse fields for determining whether an outcome variable Y is independent of a treatment variable X , conditioning on a potentially high-

dimensional vector of confounding variables Z . This type of testing is critical for understanding the complex relationships among variables. For instance, scientists may hope to understand whether a specific genetic feature influences disease outcomes, whether a particular treatment effectively extends life expectancy, or whether certain demographic factors impact college admissions.

Traditionally, these conditional testing problems are approached by modeling Y against X and Z through some parametric or semiparametric model. However, this strategy has been criticized due to potential model misspecification and limited observations of Y . As an alternative strategy, the model-X framework and Conditional Randomization Test (CRT) propose testing for the general conditional independence hypothesis H 0 : X K K Y | Z , free of any specific effect parameters [2]. The CRT assumes the distribution of X | Z to be known and can control the type-I error exactly, allowing for the choice of any flexible, black-box test statistic. This strategy is particularly useful when there is either strong and reliable scientific knowledge of the distribution of X | Z or an auxiliary dataset of p X,Z q of large sample size, known as the semi-supervised setting.

In practice, testing for conditional independence frequently involves using data from a source population to draw conclusions about a target population. This situation presents challenges due to potential differences in the distribution of variables between the two populations. For example, economists may be interested in whether college admission ( Y ) is independent of family income ( X ), conditioning on variables such as GPA, extracurricular activities, geographic location, and other demographics ( Z ). In the source population, the relationship might be influenced by factors like wealthy parents investing in SAT preparation, which boosts admission rates-a relationship that may not exist in a target population where such preparation is less common. Although Y may not appear independent of X given Z in the source population, the conclusion could vary significantly in the target population. This discrepancy underscores the need for a robust and flexible testing procedure that can adapt to shifts in distributions.

More specifically, we address the covariate shift scenario, where the distributions of the treatment variables X , the confounding variables Z , and some surrogate or auxiliary variables V (e.g., SAT scores) may differ between the source and target populations. However, the conditional distribution of Y given X,Z, and V remains the same between them. In such scenarios, our goal is to leverage information from the source to accurately test for conditional independence in the target population without the observation of Y on target. In the scenario we consider, the presence of V and potential differences in P p V | X,Z q between the source and target populations may lead to the conditional independence X K K Y | Z not holding simultaneously in the two populations. Specifically, because

the conditional distribution of Y given X and Z can vary between populations. This underscores why the problem is non-trivial.

See Figure 1 for an example of the consequences of such covariate shift.

In this paper, we propose a novel conditional independence test suitable for covariate shift scenarios. Our method builds upon the Pearson Chi-Squared Conditional Randomization (PCR) test, a powerful model-X testing procedure that effectively addresses a broader range of alternative p -value distributions than the vanilla CRT [5]. Methodologically, we make two major contributions. First, we introduce importance weights into the label counting steps of the original PCR test, making the new test valid under covariate shift. These weights adjust the importance of each sample according to its density ratio, effectively rebalancing the source data to match the target population's distribution. Second, we introduce a power enhancement method that employs the control variates method to reduce variance in the test statistics. Although importance weights can increase the variance in test statistics, especially when the density ratio can become extremely high, potentially reducing power, our power enhancement method effectively addresses this issue. Together, these innovations enable us to develop a PCR test that is both powerful and valid under covariate shifts.

The rest of the paper is organized as follows: In Section 2, we provide a formal introduction to the problem setup. In Section 3, we introduce the proposed Covariate Shift Corrected Pearson Chi-squared Conditional Randomization (csPCR) test and establish that the proposed csPCR test controls the Type-I error asymptotically. In Section 4, we demonstrate the empirical performance of the csPCR test through simulation studies. In Section 5, we apply the proposed csPCR test to a real-world dataset to assess the impact of a COVID-19 treatment on the 90-day mortality rate among patients.

1.1 Related Work

Our work builds upon the model-X framework and the conditional randomization test proposed by Candes et al. [2]. The particular method we develop is based on a variant of the vanilla CRT, the Pearson Conditional Randomization (PCR) test [5]. Recent advances in the CRT include improving computation time [7, 10], studying robustness [6, 11], and examining statistical power [19]. The focus of this paper, different from the above, is on how to build a valid CRT procedure when there is covariate shift. The paper is also complementary to the above literature: for example, we hope that future work can conduct theoretical power analysis for our procedure or develop a double robust version of the procedure just like in [6]. Finally, we note that surrogate variables play a crucial role in this paper: because the distribution of the surrogate variables is different in the source and the target population, naively testing the conditional independence hypothesis in the source population can yield invalid conclusions for the target population. A surrogate or silver standard label is a variable that is more feasible and accessible than Y in data collection and can be viewed as a noisy measure of Y . For example, tumor response rate is often used as an early endpoint surrogate for the long-term survival outcome [3], and blood pressure is commonly used as a surrogate for heart attacks. Surrogate variables are also commonly used in environmental studies and economics. Surrogate variables also play an important role in the paper by [6], albeit in a different way, where the surrogate variables are used to learn the distribution of Y | X,Z and to further improve the robustness of the CRT procedure.

Statistical learning and inference under covariate shift has been extensively studied over the past years. As a seminal work in addressing covariate shift bias, [4] proposed a density ratio weighting approach using kernel mean matching to characterize the adjusting weights. Their key idea of importance (re)weighting is intrinsically connected with early work in broader contexts like importance sampling [15, e.g.] and semiparametric inference [13, e.g.]. [8] extended this idea to a doubly robust framework accommodating surrogate variables like V and being more robust to the misspecification or poor quality of the density ratio models. [18] handled a more challenging scenario with severe shift and poor overlap between the source and target populations. Among this track of literature, [17] is the most closely related to our work as they also considered conditional independence testing under distributional shifts and proposed a general testing procedure base on importance sampling (IS) allowing for the use of CRT. Different from us, their work does not accommodate the covariate shifts of some surrogate or auxiliary V . Moreover, as will be shown in our numerical studies, their general IS testing strategy can encounter the loss of effective sample sizes and be less powerful than ours.

2 Problem Setup

2.1 Conditional Independence Testing under Covariate Shift

Let Y P R denote the outcome variable, X P R the treatment variable, Z P R p a vector of confounding variables, and V P R d a vector of surrogate variables. To make the problem more concrete, consider the following two examples:

Example 1 (College Admission) . Y is college admission, X is family income, Z includes a number of factors such as GPA, extracurricular activities, geographic location, and demographic information, V is the SAT score. In this case, V is easier to collect compared to Y as the college admission requires individual-level surveys.

Example 2 (Health Outcome) . Y is a long-term health outcome, X is a medical treatment, Z includes factors such as age, gender, and health history, V includes surrogate variables like blood pressure, BMI, and duration of hospital stays post the treatment, which can be measured within a much shorter term than Y .

(a) Source population.

(b) Target population.

Zs

Xs

Vs

Ys

ZT

XT

VT

YT

Consider a scenario involving two distinct populations: the source population S and the target population T . We collect data from the source population with the goal of making inferences about the target population. The source data contains n independent and identically distributed samples of p Y i , X i , Z i ¨ , V i ¨ q for i ' 1 , . . . , n . Let y ' p Y 1 , Y 2 , . . . , Y n q J P R n , x ' p X 1 , X 2 , . . . , X n q J P R n , Z ' p Z 1 ¨ , Z 2 ¨ , . . . , Z n ¨ q J P R n ˆ p , and V ' p V 1 ¨ , V 2 ¨ , . . . , V n ¨ q J P R n ˆ d . We are interested in testing the following conditional independence hypothesis in the target population:

We assume that the conditional distribution of Y | X,Z,V is the same in both populations; however, the distribution of p X,Z,V q varies between S and T . More precisely, the joint distribution of Y, X, Z, V can be described as follows:

This situation is referred to as the covariate shift scenario because the distribution of the covariates X , Z , and V in the source population S does not match that in the target population T .

Let's understand the above assumption and its implications through the two examples above. In the college admissions example, it is plausible to assume that the rate of college admissions remains consistent across the two populations when conditioned on the SAT score, family income, and other confounding variables. However, the joint distribution of X,V and Z can differ: in the source population, if wealthy parents frequently invest in SAT preparation, boosting admission rates, this relationship may not hold in a target population where such preparation is uncommon. In such cases, it is thus possible that X ✚ ✚ K K Y | Z in the source population but X K K Y | Z in the target population (see Figure 2 for such an example). In the health outcomes example, it is again plausible that the conditional distribution of long-term health outcomes given the treatment variable, confounding variables, and surrogates remains the same across the two populations. However, the assignment of the treatment may depend differently on the surrogate variables across the two populations. Therefore, it's possible that X K K Y | Z in one population, but not in the other.

In both examples, we can see that the result of naively applying a valid conditional independence test on the source population cannot guarantee a valid conclusion for testing H 0 in the target population. Therefore, we need to develop new tools for addressing covariate shifts in conditional independence tests.

2.2 Model-X Framework

In this paper, we operate within the model-X framework, as described by Candes et al. [2], which assumes that the joint distributions of covariates X,V,Z are perfectly known in both the source and target populations. This framework is particularly suited for scenarios where: (1) there is substantial prior domain knowledge about the covariates X,V, and Z , or (2) there is a significant amount of unsupervised data for these covariates in both populations, in addition to n labeled observations in the source population, characterizing a semi-supervised setting.

An example of the first scenario can be seen in genetics, where researchers have well-established models for the joint distributions of single nucleotide polymorphisms (SNPs). For the second scenario, consider our earlier example involving health outcomes. Here, the outcome variable Y represents a long-term health outcome that is more costly or sensitive to measure compared to the shorter-term

variables X,V, and Z . In such cases, the variables X,V, and Z are typically easier and less costly to collect, frequently resulting in a semi-supervised setting in these health-related studies.

3 Method: Covariate Shift Corrected PCR Test

3.1 Incorporating the Density Ratio into the PCR Test

In Section 2.1, we discussed how naively applying conditional independence tests to the source data cannot guarantee valid conclusions for the target population. To address this issue, we must incorporate information about the differences between the two populations into our testing procedure. In particular, we will make use of the density ratio defined as:

This ratio measures the relative likelihood of observing each combination of variables p X,Z,V q in the target population compared to the source population. By reweighting the data points in the source population using this density ratio, we effectively transform the source distribution to match the distribution of the target population, thereby addressing the covariate shift problem.

More specifically, we build our method upon the recently proposed Pearson Chi-Squared Conditional Randomization (PCR) test [5]. Compared to the vanilla CRT, the PCR test is designed to be more powerful across a broader range of alternative p -value distributions. At a high level, the PCR test assigns a label to each data point following a counterfeit sampling step and a subsequent score computation step. Under the null hypothesis that X K K Y | Z , the distribution of these labels should be uniform across all possible labels. The PCR test then rejects the null hypothesis if the empirical distribution of the labels deviates significantly from uniformity, as determined by a Pearson's chi-squared test.

Under distributional shift, if the data points were sampled from the target population, then the distribution of the labels would be uniform. However, since the data points are actually sampled from the source population, they must be reweighted using the density ratio. More specifically, in the final step of the PCR test, where the Pearson's chi-squared test is applied, we consider not the count of data points for each label, but the sum of the density ratios of the data points for each label instead. Under the null hypothesis, each sum should approximate n { L , where L is the total number of labels. Consequently, we modify the Pearson's chi-squared test to determine whether these weighted sums deviate significantly from n { L .

Based on the above intuition, we propose the Covariate Shift Corrected PCR (csPCR) Test, as outlined in Algorithm 1.

In Algorithm 1, lines 1-7 correspond to those in the original PCR test. These lines initiate the test by generating counterfeit samples ˜ X p m q j . Assuming the source and target populations were identical, under the null hypothesis, the random variables p X j , Y j , Z j q , p ˜ X p 1 q j , Y j , Z j q , . . . , p ˜ X p M q j , Y j , Z j q would be exchangeable. Consequently, the rank R j would be uniformly distributed over t 1 , . . . , M ` 1 u in the absence of ties, leading to a uniform distribution of the labels as well.

Lines 8-10 in Algorithm 1 address the covariate shift by incorporating density ratios as importance weights into W j . Due to this redefinition of W ℓ , the null distribution of the final test statistic U n,L is also different. Therefore, we also adjust the rejection threshold from the quantile of a chisquared distribution, as in the original PCR test, to the quantile of the weighted sum of chi-squared distributions.

3.2 Power Enhancement

To effectively address covariate shift, incorporating density ratios as importance weights into the PCR test is essential. However, when these ratios become large, they can increase the variance of the statistics W l . This elevated variance can diminish the test's power. Therefore, developing methods to reduce this variance is crucial for maintaining the power of the test.

Algorithm 1 Covariate Shift Corrected PCR (csPCR) Test.

Input: Data D T ' p y , x , Z , V q , the density ratio e , the test statistics T , integers K,L ě 1 , and the significance level α .

1: Take M ' KL ´ 1 .

2: for each data point j ' 1 to n do

3: Draw M i.i.d samples r X p 1 q j , . . . , r X p M q j from P T p X | Z q .

4: Use T to score the initial data point p X j , Y j , Z j q and its M counterfeits p r X p 1: M q j , Y j , Z j q

5: Let R j denote the rank of T j among t T j , r T p 1 q j , . . . , r T p M q j u , with ties broken randomly.

6: Partition t 1 , . . . , M ` 1 u ' S 1 Ť . . . Ť S L with S ℓ : ' tp ℓ ´ 1 q K ` 1 , . . . , ℓK u . Assign label ℓ j P t 1 , 2 , . . . , L u to sample j if R j P S ℓ j .

7: end for

8: Let w j ' e p X j , Z j , V j q for each j P t 1 , 2 , . . . , n u .

9: for each label ℓ P t 1 , 2 , . . . , L u : do

10: Let W ℓ be the sum of ℓ -labeled importance weights: W ℓ ' n ř j ' 1 w j ¨ ✶ t ℓ j ' ℓ u .

13: Let ˆ Ω n ' L n diag p D 1 , D 2 , ¨ ¨ ¨ , D L q ´ 1 L ¨ 1 L ˆ L .

11: Let D ℓ be the sum of ℓ -labeled squared importance weights: D ℓ ' n ř j ' 1 w 2 j ¨ ✶ t ℓ j ' ℓ u . 12: end for

14: Calculate the test statistic U n,L as follows U n,L ' L n L ř ℓ ' 1 ` W ℓ ´ n L ˘ 2 .

Output: Reject the null hypothesis if U n,L ě θ ˆ Ω n ,α ; otherwise, accept the null hypothesis. Here, θ ˆ Ω n ,α is the 1 ´ α quantile of the distribution χ 2 ˆ Ω n , where A ' χ 2 Ω denotes that A ' x T x for x ' N p 0 , Ω q .

To this end, we introduce a control variate function a , allowing a p X,Z,V q to serve as a control variate in reducing variance in W l [14]. Specifically, for a chosen γ ℓ , we define

We can then use Ă W ℓ instead of W ℓ in our algorithm.

We note that for any arbitrary choice of the function a and the parameter γ ℓ , the expectation of Ă W ℓ would be the same as that of W ℓ :

Therefore, even if we make a sub-optimal choice of the function a and the parameter γ ℓ in practice, the resulting test (under certain assumptions) will still remain asymptotically valid (see Section 3.3 for more details).

However, for effective variance reduction, it is preferable to have the control covariates a p X,Z,V q well-correlated with the outcome (See Section 4 for practical discussions on choices of the function a ). This is quite feasible, especially since the surrogate variable V is likely to be predictive of Y .

Input: Data D T ' p y , x , Z , V q , the density ratio e , the test statistics T , the control variate function a , integers K,L ě 1 , and the significance level α .

1: for each data point j ' 1 to n do

2: Compute the labels ℓ j as in Algorithm 1.

3: end for

4: Let w j ' e p X j , Z j , V j q for each j P t 1 , 2 , . . . , n u .

5: for each label ℓ P t 1 , 2 , . . . , L u : do

6: Compute ˆ γ ℓ , the regression coefficient obtained by a weighted linear regression of the indicator function ␣ t ℓ j ' ℓ u ( n j ' 1 on the control variate ␣ a p X j , Z j , V j q ( n j ' 1 with weights ␣ w j ( n j ' 1 .

✶ 7: Compute the augmented version of W ℓ as

8: end for 9: Let W ' ´ w j ¨ r ✶ t ℓ j ' ℓ u ´ ˆ γ ℓ a p X j , Z j , V j qs ` ˆ γ ℓ E T ' a p X,Z,V q ‰ ¯ ℓ,j for 1 ď ℓ ď L, 1 ď j ď n .

10: Calculate the sample covariance matrix r Ω n ' L n p W ´ 1 L ¨ 1 L ˆ n qp W ´ 1 L ¨ 1 L ˆ n q T .

11: Calculate the test statistic U n,L as follows r U n,L ' L n L ř ℓ ' 1 ´ Ă W ℓ ´ n L ¯ 2 .

Output: Reject the null hypothesis if r U n,L ě θ r Ω n ,α ; otherwise, accept the null hypothesis. Here, θ r Ω n ,α is the 1 ´ α quantile of the distribution χ 2 r Ω n , where A ' χ 2 Ω denotes that A ' x T x for x ' N p 0 , Ω q .

We would also like to discuss the choice of γ ℓ . According to the control covariate literature, with a fixed function a , the optimal choice of γ ℓ that minimizes variance is given by:

This coefficient is also the same as that obtained from a linear regression [14]. Thus, when implementing the algorithm, we take γ ℓ to be the regression coefficient obtained by running a weighted linear regression of the indicator function t ✶ ␣ ℓ j ' ℓ ( u n j ' 1 on the control variate ␣ a p X j , Z j , V j q ( n j ' 1 with weights ␣ w j ( n j ' 1 .

We have outlined the new csPCR test, including this power enhancement step, in Algorithm 2.

3.3 Theoretical Properties

In this section, we establish that the proposed tests control the type-I error asymptotically. Furthermore, we show that the power enhancement step effectively reduces the variance of the statistics W ℓ , which can typically improve the power.

Assumption 1 (Fourth moment) . The fourth moment of the density ratio e p X,Z,V q is finite: E S ' e p X,Z,V q 4 ‰ ă 8 . Furthermore, the fourth moment of product of the density ratio and the control variate function is also finite: E S ' e p X,Z,V q 4 a p X,Z,V q 4 ‰ ă 8 .

Theorem 1 (Valid Tests) . Under Assumption 1, assume that the null hypothesis of X K K Y | Z holds in the target population, then

Theorem 2 (Variance Reduction) . Let W l be the statistics computed in line 10 in Algorithm 1, and Ă W l be the statistics computed in line 7 in Algorithm 2. Under Assumption 1,

4 Numerical Simulation

In this section, we present simulation studies to assess the performance of our proposed csPCR method and its power enhancement version denoted csPCR(pe), and compare them to a benchmark method. The benchmark method adopted is an importance-resampling based method [17], denoted as the IS method. For fair comparison, we used the same PCR statistic as our method for the testing with IS. We use a significance level of α ' 0 . 05 .

4.1 Simulation Setup

Weconsider a semi-supervised setting where we have a large volume of unlabeled data of p X j , Z j , V j q from both the source and target populations. In addition, we have a small number of labeled data of p Y j , X j , Z j , V j q from the source population.

We separate confounding variables Z into two sets: Z ' p Z r , Z null q , where Z r is the relevant set and Z null is the null set. The relevant confounding variables Z r are generated as i.i.d. multivariate normal, with mean 0 for the source population and 1 for the target population to simulate the distributional shift in Z , where Z r P R p and we set p ' 5 . Null confounding variables Z null are generated independently with no correlation to other variables, modeled as N p 0 . 1 , I q q with q ' 50 for sparse high-dimensional settings in both populations.

The treatment variable X and the surrogate variable V are conditionally generated based on Z . Specifically, X is modeled identically across both the source and target populations as N p u J Z r , 1 q , where u is a predefined parameter vector that remains the same for both populations.

For V , it is modeled differently in the two populations, represented as N p v J S { T Z r `p 1 ´ θ q a S { T X ` θa S { T sin p X q , 1 q . Here, v S and v T are predefined parameter vectors for the source and target populations, respectively. The parameter a varies between populations ( a S for the source and a T for the target), controlling the effect of X on V , modeling the indirect effect. The factor θ modulates the nonlinear component of this relationship.

The outcome variable Y is generated for both populations using the same conditional model over p X,Z,V q :

where β and γ control the effects of V (indirect) and X (direct) on Y , respectively.

We generate 1000 unlabeled source and target samples to estimate the density ratio and generate 500 labeled source samples for testing. Moreover, in the simulation, we assume we have full knowledge of the joint distribution of p X,Z q and estimate V | X,Z using an Elastic net regression model with 5-fold cross-validation [20]. For the test statistic T in the algorithm, we choose a simple function T p ˜ X,Z,V,Y q ' Y ¨ ˜ X . For each parameter iteration, we conduct 1000 Monte Carlo simulations to estimate the Type-I error and power. We estimate the covariance matrix of the sequence of W i 's using the Monte Carlo method and use the momentchi2 package [1] for calculating the p -value. Additionally, we empirically choose the best hyperparameter L ' 3 for all our experiments through additional experiments shown in Appendix B.2.

4.2 Simulation Results

In Figure 3, we choose a S ' 1 and a T ' 0 to compare the Type-I error control of our methods with the benchmark. The left panel shows the Type-I error rate as the sample size of the data used to estimate the density ratio, n e , varies from small to large. There appears to be a slight Type-I error inflation for all three methods when the sample size n e is small, but the Type-I error quickly converges to the ideal level of 0.05 as n e grows larger. Moreover, our methods show more stable Type-I error control than the benchmark method when the estimation sample size is low. The right

panel shows that when the density ratio is well approximated, all three methods attain good Type-I error control regardless of the change in β , i.e., the strength of the indirect effect, but the csPCR and csPCR(pe) methods have more stable control.

0.25

0.25

CSPCR

CSPCR

0.20

IS

csPCR(pe)

0.20

IS

csPCR(pe)

0.15

Error

0.15

0.10

Type-1

0.10

0.05

0.05

0.00

400

500

600

700

800

900

1000

0.00

0

0.2

0.4

0.6

0.8

1

ne

β

To evaluate the statistical power of our csPCR test, we choose a S ' 0 and a T ' 2 , so that the null hypothesis holds true in the target population but not in the source population. As Figure 4a shows, both the csPCR and the csPCR(pe) methods have uniformly higher power than the benchmark method as we vary the indirect effect size β . For example, when β ' 1 . 4 , the benchmark IS method has a power of 0.33, the csPCR method has a power of 0.44, and the csPCR(pe) method can attain a power of 0.8.

When we fix the indirect effect β ' 2 and vary the direct effect of X ( γ ), as shown in Figure 4b, our methods still exceed the benchmark, and the power enhancement significantly improves the original version of the test. For example, when γ ' 1 , the benchmark IS method has a power of 0.4, the csPCR method has a power of 0.62, and the csPCR(pe) method can attain a power of 0.86.

(a) Increasing indirect effect

β

(b) Increasing direct effect

γ

(c) Varying nonlinear effect size

θ

1.0

1.0

1.0

CSPCR

CSPCR

0.8

csPCR(pe)

0.8

0.8

csPCR(pe)

IS

IS

Power

0.6

Power

0.6

Power

0.6

0.4

0.4

0.4

CSPCR

0.2

0.2

csPCR(pe)

0.2

IS

0.0

0

0.20.40.60.8

1

1.21.41.61.82

0.0

0

0.20.40.60.8

1.21.41.61.8

2

0.0

0

0.10.20.30.40.50.60.70.80.9

1

β

θ

We also test how adding a nonlinear component to the indirect effect affects the power when we assume a linear model of V | Z, X in the estimation stage. This can be helpful in assessing the performance of our methods under model misspecification. As Figure 4c indicates, as the nonlinear effect increases, the power of all three methods decreases, though our methods still significantly exceed the benchmark. Interestingly, we observe that as θ Ñ 1 , i.e., there is a full nonlinear component without a linear component, the advantage of the power-enhanced version over the original csPCR test disappears. This occurs because when the V | X,Z model is misspecified and the density ratio estimation is inaccurate, the variance reduction in the control variates step reduces variance in the 'wrong" direction, and thus does not improve the power of the original method.

4.3 Effective Sample Size

Wenotice a series of work in measuring the effective sample size (ESS) of the density ratio reweighting approaches [9]. Among them, one of the most common measure is n eff ' p ř n i ' 1 w i q 2 { ř n i ' 1 w 2 i . When the covariate shift between the source and target becomes stronger, the variance of the importance weight w i tends to be large and n eff will become smaller, which could result in lower power. We carry out simulation studies on the relationship between the power of csPCR and the ESS determined by the degree of covariate shift as discussed in Appendix B.3.

5 Real-World Application

The COVID-19 pandemic has presented unprecedented challenges to global health systems, with high variability in outcomes based on demographic and clinical characteristics. Early identification of patients at high risk for severe outcomes, such as mortality within 90 days of hospital admission, is crucial for timely and effective treatment interventions. This study leverages extensive hospital data to develop models predicting 90-day mortality following hospital admission due to COVID-19.

For this study, we extract patient data spanning from January 2020 to December 2023 from Duke University Health System (DUHS), focusing on individuals admitted with COVID-19. This period encompasses multiple waves of the pandemic, influenced by various circulating variants.

Our dataset comprises patient records for a total of N ' 3 , 057 individuals admitted with COVID-19. The outcome Y is defined as mortality within 90 days since hospital admission due to COVID-19. The treatment variable X is defined as binary, where 1 indicates the administration of any COVID-19 specific medication (explained in Appendix C) and 0 otherwise. The covariates Z include comorbidity indices (renal disease, diabetes without complication, diabetes with complication, local tumor, and metastatic tumor), age, gender, and race, which are critical for adjusting the risk models due to their known influence on COVID-19 outcomes. The length of hospitalization, denoted as V , is standardized to follow a standard normal distribution (with a mean of zero and a standard deviation of one), facilitating comparisons and integration into predictive models regardless of original scale or distribution.

The dataset is segmented into two distinct groups based on the date of hospital admission to align with pivotal changes in virus strain predominance and public health guidelines. The source data comprises COVID-19 admissions prior to November 30, 2021, with a sample size of N 1 ' 1 , 131 patients. The target data includes admissions from November 30, 2021, through December 2023, totaling N 2 ' 792 patients. This temporal division allows for the analysis of trends and outcomes associated with the evolving pandemic landscape. Prevalence of the 90-day mortality outcome within the source data is 14.3%, reflecting the impact of earlier virus strains and treatment protocols, while in the target data, the prevalence is substantially lower at 3.7%, possibly indicating the effect of improved treatments and vaccines, as well as the influence of different virus variants over time.

For the analysis, we divide 50% of the source data, comprising 565 individuals, alongside the entirety of the target data, to estimate the density ratio. Density ratios of X,Z are estimated using probabilistic classification method [12], while the density ratio of V | X,Z is determined through Elastic Net regression. For all three methods, the test statistic T is chosen to be T p ˜ X,Z,V,Y q ' Y ¨ ˜ X . As indicated in Table 1, both csPCR and csPCR(pe) give statistically significant results, whereas the IS method does not. The statistically significant results are consistent with biomedical literature. For example, through systematic review and meta-analysis, [21] reported that Bamlanivimab is effective in reducing the mortality rates of COVID patients. In a cohort study, [16] also found similar effectiveness for Nirmatrelvir-ritonavir.

These results align with our findings from the simulation study and demonstrate that our method has increased power compared with the benchmark IS method.

References

Dean A Bodenham and Niall M Adams. A comparison of efficient approximations for a weighted sum of chi-squared random variables. Statistics and Computing , 26(4):917-928, 2016.

Emmanuel Candes, Yingying Fan, Lucas Janson, and Jinchi Lv. Panning for gold:'modelx'knockoffs for high dimensional controlled variable selection. Journal of the Royal Statistical Society Series B: Statistical Methodology , 80(3):551-577, 2018.

Emerson Y Chen, Vikram Raghunathan, and Vinay Prasad. An overview of cancer drugs approved by the us food and drug administration based on the surrogate end point of response rate. JAMA Internal Medicine , 179(7):915-921, 2019.

Jiayuan Huang, Arthur Gretton, Karsten Borgwardt, Bernhard Schölkopf, and Alex Smola. Correcting sample selection bias by unlabeled data. Advances in neural information processing systems , 19, 2006.

Adel Javanmard and Mohammad Mehrabi. Pearson chi-squared conditional randomization test. arXiv preprint arXiv:2111.00027 , 2021.

Shuangning Li and Molei Liu. Maxway crt: improving the robustness of the model-x inference. Journal of the Royal Statistical Society Series B: Statistical Methodology , 85(5):1441-1470, 2023.

Molei Liu, Eugene Katsevich, Lucas Janson, and Aaditya Ramdas. Fast and powerful conditional randomization testing via distillation. Biometrika , 109(2):277-293, 2022.

Molei Liu, Yi Zhang, Katherine P Liao, and Tianxi Cai. Augmented transfer regression learning with semi-non-parametric nuisance models. Journal of Machine Learning Research , 24(293): 1-50, 2023.

Luca Martino, Víctor Elvira, and Francisco Louzada. Effective sample size for importance sampling based on discrepancy measures. Signal Processing , 131:386-401, 2017.

Binh T Nguyen, Bertrand Thirion, and Sylvain Arlot. A conditional randomization test for sparse logistic regression in high-dimension. Advances in Neural Information Processing Systems , 35:13691-13703, 2022.

Ziang Niu, Abhinav Chakraborty, Oliver Dukes, and Eugene Katsevich. Reconciling model-x and doubly robust approaches to conditional independence testing. arXiv preprint arXiv:2211.14698 , 2022.

Jing Qin. Inferences for case-control and semiparametric two-sample density ratio models. Biometrika , 85(3):619-630, 1998.

James M Robins, Andrea Rotnitzky, and Lue Ping Zhao. Estimation of regression coefficients when some regressors are not always observed. Journal of the American statistical Association , 89(427):846-866, 1994.

Sheldon M Ross. Simulation . academic press, 2022.

Donald B Rubin. The calculation of posterior distributions by data augmentation: Comment: A noniterative sampling/importance resampling alternative to the data augmentation algorithm for creating a few imputations when fractions of missing information are modest: The sir algorithm. Journal of the American Statistical Association , 82(398):543-546, 1987.

Kevin L Schwartz, Jun Wang, Mina Tadrous, Bradley J Langford, Nick Daneman, Valerie Leung, Tara Gomes, Lindsay Friedman, Peter Daley, and Kevin A Brown. Population-based evaluation of the effectiveness of nirmatrelvir-ritonavir for reducing hospital admissions and mortality from covid-19. Cmaj , 195(6):E220-E226, 2023.

Nikolaj Thams, Sorawit Saengkyongam, Niklas Pfister, and Jonas Peters. Statistical testing under distributional shifts. Journal of the Royal Statistical Society Series B: Statistical Methodology , 85(3):597-663, 2023.

Kaizheng Wang. Pseudo-labeling for kernel ridge regression under covariate shift. arXiv preprint arXiv:2302.10160 , 2023.

Wenshuo Wang and Lucas Janson. A high-dimensional power analysis of the conditional randomization test and knockoffs. Biometrika , 109(3):631-645, 2022.

Hui Zou and Trevor Hastie. Regularization and variable selection via the elastic net. Journal of the Royal Statistical Society Series B: Statistical Methodology , 67(2):301-320, 2005.

Ling Zuo, Guangyu Ao, Yushu Wang, Ming Gao, and Xin Qi. Bamlanivimab improves hospitalization and mortality rates in patients with covid-19: a systematic review and metaanalysis. The Journal of infection , 84(2):248, 2022.

A Proofs

A.1 Preliminaries

Throughout this section, we write S p x j , z j , v j q ' s j as the label assigned to sample j in Algorithms 1 and 2, instead of using ℓ j . This notation helps avoid confusion between different label choices.

Proposition 1. Assume that the conditional independence X K K Y | Z holds on the target population T . Let e p x j , z j , v j q denote the density ratio. For any integer ℓ P r 1 , L s , the following holds:

Proof of Proposition 1. For simplicity, denote w j ' e p X j , Z j , V j q and s j ' S T p X j , Z j , V j q .

The last equation follows from results in the non-covariate-shift scenario, e.g., from [5].

A.2 Proof of Theorem 1

A.2.1 Results for Algorithm 1

Let p W ℓ q ℓ ' 1 ,...,L be the sum of weights and ˆ Ω n be the sample covariance matrix in Algorithm 1. By Proposition 1, we have that

✶ Note that the W ℓ 's are sums of i.i.d. random variables, and thus by the Central Limit Theorem, as n Ñ8 ,

where for any ℓ, ℓ ˚ P t 1 , . . . , L u

Therefore,

Next, we will focus on the variance estimation part. We will show that ˆ Ω n p Ñ Ω as n Ñ8 . For any ℓ, ℓ ˚ P t 1 , . . . , L u ,

Up til now, we have that

Therefore,

A.2.2 Results for Algorithm 2

Let p Ă W ℓ q ℓ ' 1 ,...,L and r Ω n be the sum of weights and the sample covariance matrix in Algorithm 2. Let ˆ γ ℓ be the estimated coefficient in Algorithm 2.

Recall that in (7), we have identified the optimal choice of γ ℓ . We will start by working with this optimal choice and show that the ˆ γ ℓ is close to it. Define

Therefore, we have | W ℓ ' ř j K ℓ,j p γ ℓ q and Ă W ℓ ' ř j K ℓ,j p ˆ γ ℓ q ' | W ℓ ´p ˆ γ ℓ ´ γ ℓ q ř j H j .

Note that by (6), E p H j q ' 0 . By Proposition 1 and (6), we have E p | W ℓ q ' n L . Furthermore, because | W ℓ is a sum of i.i.d. random variables, we have that as n Ñ8 ,

where for each ℓ, ℓ ˚ P t 1 , . . . , L u ,

Next, we will show that the actual statistic r U n,L is close to q U n,L , and that the estimated variance matrix is also close to q Ω . We start with noting that the estimator ˆ γ ℓ from linear regression is close to the optimal choice γ ℓ defined in (7): by the Central Limit Theorem, ˆ γ ℓ ' γ ℓ ` O p p 1 { ? n q . And thus

Therefore,

The second-to-last line is because ˆ γ ℓ ' γ ℓ ` O p p 1 { ? n q and the terms inside the parenthesis, ř j H j , is a sum of n independent mean-zero random variables.

Therefore, together with (12), by Slusky's Theorem, we have that

and thus,

We will work on sample covariance matrix now. Recall that the sample covariance matrix r Ω n ' L n p W ´ 1 L ¨ 1 L ˆ n qp W ´ 1 L ¨ 1 L ˆ n q T , where W ℓ,j ' w j ¨ r ✶ t ℓ j ' ℓ u ´ ˆ γ ℓ a p X j , Z j , V j qs ` ˆ γ ℓ E T ' a p X,Z,V q ‰ ' K ℓ,j p ˆ γ ℓ q . Let's start with WW T . For any ℓ, ℓ ˚ P t 1 , . . . , L u ,

Therefore, by the law of large numbers,

Similarly, for W1 T , we have that for any ℓ, ℓ ˚ P t 1 , . . . , L u ,

Therefore, again by the law of large numbers,

Combining the above results gives,

Therefore, r Ω n p Ñ q Ω .

To summarize, we have that

Therefore,

A.3 Proof of Theorem 2

Similar to the proof of Theorem 1, we define

Therefore, we have | W ℓ ' ř j K ℓ,j p γ ℓ q and Ă W ℓ ' ř j K ℓ,j p ˆ γ ℓ q ' | W ℓ ´p ˆ γ ℓ ´ γ ℓ q ř j H j .

We know from the literature that γ ℓ is the optimal choice of γ and thus Var ' | W ℓ ı ď Var r W ℓ s . We will then move on to show that Var ' Ă W ℓ ı is close to Var ' | W ℓ ı and thus asymptotically no greater than Var r W ℓ s .

To this end, note that

But we also know from the proof of Theorem 1 that ˆ γ ℓ ´ γ ℓ p Ñ 0 . Then, because of the bounded fourth moment assumption, by the Dominated Convergence Theorem, we have that

Finally, we note that Var r W l s ' Ω p n q , and hence

B Additional Simulation Results

B.1 Running time

All experiments run on a Macbook Pro 2022 M2.

Artificial dataset : Regarding running time for one iteration including density ratio estimation and X | Z model fitting (on average), csPCR took 5.12s, csPCR(pe) took 14.95s, IS method took 1.5s, PCR took 1.25s.

Real-world application : Regarding running time for one test procedure, csPCR took 3.41s, csPCR(pe) took 11.32s, IS method took 0.81s.

B.2 Finding optimal hyperparameter L

We find the optimal L value for the testing algorithm by performing numerical simulations, evaluating its Type-I error control and power. We adopt the same numerical simulation setup as in the main text Section 4. We first choose a S ' 1 and a T ' 0 and also fix β ' 1 to compare the Type-I error rate for different choice of L of the csPCR method. We perform experiments with both true density ratio and estimated density ratio. The results are shown in Table 2.

Therefore,

1.0

0.8

Power

0.6

0.4

0.2

CSPCR

csPCR(pe)

0.0

2

3

4

5

6

7

8

9

10

L

We also test the power of the csPCR and csPCR(pe) method with different choices of L value. We choose a S ' 0 and a T ' 2 and fix β ' 2 .

As Table 2 and Figure 5 shows, as L value increases, the csPCR method become more conservative with more tight Type-I error control and lower power. We can observe that when we set L ' 3 , the csPCR method can achieve most stable Type-I error rate control and also highest power empirically. Therefore, in our simulation experiments and real world data experiments, we fix L ' 3 .

B.3 Role of effective sample size

We notice a series of work in measuring the effective sample size (ESS) of importance weight or sampling in the statistical computation literature, e.g., [Martino, et al, 2017] and others. Among them, one of the most common ways is to use the ratio n eff ' p ř i ' 1 w i q 2 ř i ' 1 w 2 i to approximate the ESS. When the covariate shift between the source and target becomes stronger, the variance of the importance weight w i tends to be large and n eff will become smaller, which can result in lower power. Our power enhancement method based on control variate could potentially alleviate this issue with properly specified control functions.

In the simulation study, we varied only µ z , the mean of the confounding variables Z T . A higher µ z signifies a stronger covariate shift between the source and target populations. From Figure 6 , it is evident that as µ z increases, the Effective Sample Size (ESS) required significantly decreases, while the power of the csPCR method concurrently declines. These results suggest that increasing covariate shift leads to a reduction in ESS and a corresponding decrease in statistical power.

B.4 Instability of the Importance Resampling (IS) method

In this section, we will use numerical simulations to ilustrate that the performance if the IS method is subject to the resample size heavily. IS method performs resampling without replacement and typically has to sample a much smaller subset (theoretically, in the order of o p ? n q ) of the source data to approximate the target. Consequently, the power of IS is substantially lower than our approach. If the resample size of IS is overly increased, it may fail to control the Type-I error due to excessive similarity between the resampled data and the original source data.

To further illustrate, we conducted additional experiments with varied resample sizes in IS to assess its effect on Type-I error control and power. From Figure7. one can observe that IS starts to show high Type-I error inflation when its resample size increases to 400 but still shows much lower power

1.0

250

CSPCR

0.8

csPCR(pe)

200

0.6

150

Powe

S

0.4

100

0.2

50

0.0

0

0.2

0.4

0.6

0.8

0.2

0.4

0.6

0.8

μz

μz

(by around 0.4) than our method with this resample size (or even larger ones). This indicates that our method achieves better statistical efficiency than IS (DRPL).

1.0

1.0

IS

IS

0.8

CSPCR

0.8

CSPCR

ErrorRate

cSPCR(pe)

cSPCR(pe)

0.6

Power

0.6

0.4

0.4

0.2

0.2

0.0

0.0

200

400

600

800

1000

200

400

600

800

1000

ResampleSize

ResampleSize

B.5 Choice of test statistic

In this section we explore the effect of test statistics on the algorithm performance. The main principle of choosing the test statistic is to characterize the conditional dependency between X and Y under the alternative hypothesis. The test statistic Y X may not be the optimal choice and that using p Y ´ ˆ E r Y | Z sqp X ´ E r X | Z sq could remove the confounding effect of Z .

Inspired by this, we used Y p X ´ E r X | Z sq as the test statistic to conduct additional simulations. As illustrated in Figure8, we find that Y p X ´ E r X | Z sq and Y X produce nearly the same power for both csPCR and csPCR(pe) with the change of effect size.

C Real-World Application

The specific medication indicated by the treatment variable X includes Ritonavir, Bamlanivimab, Casirivimab-Imdevimab, Remdesivir, Ritonavir Nirmatrelvir, Sotrovimab, Bamlanivimab Etesevimab. For simplicity, X ' 1 indicates any of these specific medication and X ' 0 otherwise.

(a) csPCR

(b) csPCR(pe)

1.0

1.0

Y·X

Y·X

Y·(X-E[X|Z])

Y·(X-E[X|Z])

0.8

0.8

0.6

0.6

Power

Power

0.4

0.4

0.2

0.2

0.0

0

0.20.40.60.811.21.41.61.8

2

0.0

0

0.20.40.60.811.21.41.61.82

Effect

Effect

C.1 Different outcome

In our real data experiment part, the outcome variable Y is defined as mortality within 90 days since hospital admission due to COVID-19. In addition, we also analyzed mortality within 30 days since hospital admission. As shown in Table 3, both csPCR and csPCR(pe) methods give significant results, aligning with biomedical literature.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: In the abstract and introduction, we mentioned that we propose a novel Covariate Shift Corrected Pearson Chi-squared Conditional Randomization (csPCR) test and discussed our methodological, theoretical, and empirical contributions.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]

Justification: We examine the performance of the algorithms under model misspecification in Section 4.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: The assumptions are listed in Section 3.3 and the proofs are provided in Appendix appendix:proof.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Details of the simulation studies and real-data application are included in Sections 4, 5 and Appendix B.2.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: Replication code for our simulation studies is submitted as supplementary material. It will also be made publicly available on GitHub once our paper is accepted. The COVID data set used for the real example in our paper is not publicly available due to privacy constraints.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Details of the simulation studies and real-data application are included in Sections 4, 5 and Appendix B.2.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: Type-I errors, power, and p-values are provided in Sections 4 and 5.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We record relevant information in Appendix B.1.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Yes, our research conforms to the NeurIPS Code of Ethics in every respect, including fairness, transparency, privacy, and social responsibility.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We have discussed the impact of the paper on the fields of healthcare and social sciences.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: We believe the paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We have cited the relevant papers and packages.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: We have provided detailed documentation of the newly proposed algorithm.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a novel approach to addressing covariate shift in conditional independence tests. By leveraging importance weights and the control variates method,the paper proposes Covariate Shift Corrected Pearson Chi-squared Conditional Randomization (csPCR) test, which maintains asymptotic Type-I error control. In addition, a power enhancement is proposed, which reduces the variance in the test statistic and thus improve the power towards alternatives. The methodology is validated both theoretically and empirically, demonstrating superior performance in simulation studies and practical application to a COVID-19 treatment dataset, highlighting its efficacy in real-world scenarios.\n\nSTRENGTHS:\nOriginality\nThe paper presents original contributions by introducing a new approach to conditional independence testing under covariate shift. This method combines the idea of conditional randomization test, importance weighting, the control variates technique. \n\nQuality\nThe work is of high quality, demonstrated through rigorous statistical theories on the validity and power, and comprehensive empirical validations.\n\nClarity\nThe paper is clearly written and well-organized. Each section has a clear message and readers can quickly grasp the main idea. \n\nSignificance\nI consider this work as a significant contribution to the field of conditional independence testing and covariate shift. The method is easy-to-implement, and is more powerful than existing methods as shown in numerical experiments.\n\nWEAKNESSES:\nThe paper does not leave serious technical weaknesses, but I am concerned about the limited applicability of this method in real-world problems. Based on my understanding, one would test for conditional independence directly on the target distribution if the outcomes in the target population were easy to collect. So csPCR is useful only when the outcomes are costly to collect. This questions several application examples mentioned in the paper. First, in the college admission example (from Line 37-53), why don't the economists just collect the college admission results from the target population and then conduct the vanilla PCR? Typically, the target outcomes have to be collected for followup analysis after the conditional independence test. Second, Section 5 introduces a real-world application of this method on Covid-19 pandemic data, where the source and the target data set are segmented based on time. Given such a sequential data collecting process, it is unclear whether the iid assumption still holds in the source and target distribution. See Questions for more details.\n\nDespite of the concerns above, I do believe that this method is useful for certain genetic and biomedical problems, where the outcomes are difficult to collect, but other covariates (such as SNPs) are relatively easy. Therefore, I suggest the authors clarify the application examples a bit by discussing more reasonable applications with some references.\n\nQUESTIONS:\n1. Page 5, Algorithm 1: Besides the variance issues considered in Algorithm 2, it is clear that the choice of test statistic in Eq (4) also affects the power. Can you comment on how to choose the test statistic? In the numerical experiment, the test statistic is specified to be the product of $Y$ and $X$, which makes sense as it measures the \"correlation\" between Y and X. However, $Z$ is not included in the test statistic, which seems to decrease its power. Have you tried any other test statistics? Do difference choices affect the power significantly? For instance, one alternative is to use $(Y - \\beta_y Z)(X - \\beta_x Z)$, where $\\beta_y and \\beta_x$ are regression coefficients for $Y\\sim Z$ and $X\\sim Z$ on an independent data set from source distribution.\n\n2. Page 4, Line 170-177: The paper argues that they used PCR as the baseline test because PCR is more powerful than the vanilla CRT. But on the other hand, CRT is also better than PCR in the sense that it has finite-sample type I error control, rather than asymptotic type I error control. It's worth mentioning this point as well. While I notice that the simulations in the paper shows satisfactory type I error control, the type I errors may get inflated if, for example, the data have heavy tails.\n\n3. Page 9, Section 9: Relating to the weaknesses, have you run any tests to check the iid assumption on the Covid-19 data set?\n\nLIMITATIONS:\nThe papers discuss about the limitations of their method under model misspecifications in Section 4, pointing out that the powerful enhancement can disappear when there is a full nonlinear component in the model. The authors have addressed the limitations adequately.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-04"}

comments: SUMMARY:
This paper proposes a new variation of CRT to be applied in the presence of covariate shifts. The paper presents a method and one extension for each, with higher power. Then, the authors present some needed theoretical results and finish with experiments.

STRENGTHS:
- The paper proposes a theoretically correct approach for testing CI under covariate shift;
- The paper presents their algorithms and theoretical results;
- The paper had convincing experiments;

WEAKNESSES:
- It would be interesting if the authors could estimate the full density ratio in their simulations (possibly in a high-d scenario) and then compare their results with the IS approach;
- The paper does not conduct any experiment where Type-I error control is shown in a real dataset, e.g., like for example in [1,2].

References

[1] Pogodin, R., Schrab, A., Li, Y., Sutherland, D. J., & Gretton, A. (2024). Practical Kernel Tests of Conditional Independence. arXiv preprint arXiv:2402.13196.

[2] Maia Polo, Felipe, Yuekai Sun, and Moulinath Banerjee. "Conditional independence testing under misspecified inductive biases." Advances in Neural Information Processing Systems 36 (2023): 58577-58612.

QUESTIONS:
- How can we understand the role of the effective sample sizes (weight imbalance) in the effectiveness of your method?
- Could you elaborate more on the surrogate variables? I am familiar with the literature on CI testing, but I haven't seen the presence of these variables in other works.

LIMITATIONS:
The authors comment on limitations.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-11

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces the Covariate Shift Corrected Pearson Chi-squared Conditional Randomization (csPCR) test, designed to address covariate shift in conditional independence testing. The csPCR method incorporates importance weights and employs the control variates method to enhance test power and reduce variance in the statistical analysis. Theoretical contributions demonstrate that csPCR controls Type-I error asymptotically. Empirical validations through simulation studies and a real-world application assessing COVID-19 treatment effectiveness showcase the method's effectiveness.\n\nSTRENGTHS:\n1. **Theoretical contribution:** The paper makes a theoretical contribution by addressing the covariate shift in conditional independence testing. \n\n2. **Methodological Innovation:** Introduction of the Covariate Shift Corrected Pearson Chi-squared Conditional Randomization (csPCR) test that incorporates importance weights and control variates method to manage covariate shifts effectively.\n\n3. **Empirical Validation:** Extensive simulation studies and real-world application (assessment of COVID-19 treatment on 90-day mortality) demonstrate the practical efficacy and superior power of the proposed csPCR test over traditional methods.\n\nWEAKNESSES:\n1. **Dependence on Accurate Estimations of Density Ratio:** The csPCR test's performance is critically dependent on the accurate estimation of density ratios. This dependence could pose significant challenges in practical scenarios characterized by limited, noisy, or high-dimensional data, where reliable density ratio estimation becomes inherently difficult.\n\n2. **Clarity on the Advantage Over Resampling Methods:** The paper does not explicitly clarify the advantages of the csPCR test over simpler resampling-based methods that also utilize estimated density ratios. While the csPCR integrates density ratios directly into the test statistic calculation and employs variance reduction techniques, it remains crucial for the authors to demonstrate why these features offer substantial improvements over traditional methods that adjust the sample weights during resampling. The discussion should address whether the complexity of csPCR provides tangible benefits, such as improved error rates or robustness in more varied practical applications, compared to potentially simpler resampling approaches.\n\nQUESTIONS:\n- Could the authors provide a detailed comparison between the csPCR test and traditional resampling methods that also use estimated density ratios? This would further strengthen the paper\n- The paper mentions the use of control variates to reduce variance introduced by importance weights. Could the authors discuss how this approach compares to variance reduction techniques used in resampling methods? \n- Could the authors elaborate on any theoretical limitations or potential failure modes of the csPCR method?\n- Are there additional empirical validations planned or underway to test the csPCR method across more varied datasets?\n\nLIMITATIONS:\nThe authors are encouraged to add a separate limitation section in either the main paper or appendix.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the issue of conditional independence under covariate shift. The authors' goal is to test the conditional independence for causal inference in the target data. The authors can use the source data whose distribution is potentially different from that of the target data. For this problem, the authors propose using the covariate-shift adaptation method and develop a method for testing the null hypothesis.\n\nSTRENGTHS:\nThis paper addresses the issue of conditional independence under covariate shift, which I find to be an intriguing problem. However, I was unable to comprehend the underlying assumptions of this problem, making it difficult to proceed with the reading.\n\nI understand the objective of this manuscript to be as follows:\n1. The authors aim to test the null hypothesis $H_0: X \\perp Y \\mid Z$ for the target data (Here, I denote the independence by $\\perp$). \n2. There is a possibility that $X \\perp Y \\mid Z$ does not hold simultaneously for both the source and target data.\n3. To test the null hypothesis for the target data, the authors employ an algorithm adapted to covariate shift.\nMy primary concern is whether the assumption \"there is a possibility that $X \\perp Y \\mid Z$ does not hold simultaneously for both the source and target data\" is justified in the first place.\n\nThe authors assume that, in order to use the covariate shift adaptation algorithm,\n- $Y \\mid X, Z, V$ is the same in both the source and target data.\nUnder this assumption, is it not the case that the situation where $X \\perp Y \\mid Z$ holds in one dataset and does not hold in the other cannot exist?\n\n$X \\perp Y \\mid Z$ implies that $p(y, x \\mid z) = p(y \\mid x, z) p(x \\mid z) = p(y \\mid z) p(x \\mid z)$, meaning that $p(y \\mid x, z) = p(y \\mid z)$.\nGiven that we assume $Y \\mid X, Z, V$ to be the same in both the source and target data, does this not imply that whether $X \\perp Y \\mid Z$ holds cannot differ between the source and target data?\n\nDue to this ambiguity, I was unable to further evaluate the paper. I would like the authors to clarify this point so that I can proceed with the evaluation.\n\nWEAKNESSES:\nSee above.\n\nQUESTIONS:\nSee above.\n\nLIMITATIONS:\nSee above.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 1, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-15"}