Covariate Shift Corrected Conditional Randomization Test
Authors: Bowen Xu
Yiwen Huang
Chuan Hong
Shuangning Li
Molei Liu
Venue: NeurIPS 2024 poster (2024)
Keywords: conditional independence test
conditional randomization test
model-X framework
surrogate variable
importance weight
control covariates
Primary area: causal_inference

Abstract: Conditional independence tests are crucial across various disciplines in determining the independence of an outcome variable $Y$ from a treatment variable $X$, conditioning on a set of confounders $Z$. The Conditional Randomization Test (CRT) offers a powerful framework for such testing by assuming known distributions of $X \mid Z$; it controls the Type-I error exactly, allowing for the use of flexible, black-box test statistics. In practice, testing for conditional independence often involves using data from a source population to draw conclusions about a target population. This can be challenging due to covariate shift---differences in the distribution of $X$, $Z$, and surrogate variables, which can affect the conditional distribution of $Y \mid X, Z$---rendering traditional CRT approaches invalid. To address this issue, we propose a novel Covariate Shift Corrected Pearson Chi-squared Conditional Randomization (csPCR) test. This test adapts to covariate shifts by integrating importance weights and employing the control variates method to reduce variance in the test statistics and thus enhance power. Theoretically, we establish that the csPCR test controls the Type-I error asymptotically. Empirically, through simulation studies, we demonstrate that our method not only maintains control over Type-I errors but also exhibits superior power, confirming its efficacy and practical utility in real-world scenarios where covariate shifts are prevalent. Finally, we apply our methodology to a real-world dataset to assess the impact of a COVID-19 treatment on the 90-day mortality rate among patients.

Covariate Shift Corrected Conditional Randomization Test
Bowen Xu*
Yiwen Huang Department of Statistics Peking University 20000107730stu, pku. edu. cn
Harvard University
bowenxuβg. harvard.
Chuan Hong
Shuangning Li Booth School of Business University of Chicago
Department of Biostatistics and Bioinformatics Duke University chuan hong@duke. edu
shuangning wi@chicagobooth..cooo edu
Molei Liu
Department of Biostatistics
Columbia Mailman School of Public Health m148908cumc. columbia.
Abstract
Conditional independence are crucial across various disciplines in determining the independence of an outcome variable Y from a treatment variable X. condition- ing set of confounders Z. The Conditional Randomization Test (CRT) offers a powerful framework for such testing by assuming known distributions of X Z; it controls the Type-I error exactly, allowing for the use of flexible, black-box test statistics. In practice, testing for conditional independence often involves using data from source population draw conclusions about target population. This can be challenging due to covariate differeiffee in the distribution of X, Z, and surrogate variables, which can affect the conditional distribution Y X, 7 Zeredering traditional CRT approaches invalid. To address this issue, we propose a novel Covariate Shift Corrected Pearson Chi-squared Conditional Randomization (csPCR) test. This test adapts to covariate shifts by integrating importance weights and employing the control variates method to reduce variance in the test statistics and thus enhance power. Theoretically, we establish that the csPCR test controls the Type-I error asymptotically. Empirically, through simu- lation studies, demonstrate that our method not only maintains control over Type-I errors but also exhibits superior power, confirming its efficacy and practical utility in real-world scenarios where covariate shifts prevalent. Finally, apply our methodology to a real-world dataset to assess the impact a COViD-19 treatment the 90-day mortality rate among patients.
1 Introduction
Conditional independence tests are important across diverse fields for determining whether an outcome variable Y independent of a treatment variable X, conditioning on potentially high-
*These authors contributed equally to this work.
Cooresponding author. To whom correspondence should be addressed.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

dimensional vector of confounding variables Z. This type of testing is critical for understanding the complex relationships among variables. For instance, scientists may hope to understand whether specific genetic feature influences disease outcomes, whether a particular treatment effectively extends life expectancy, or whether certain demographic factors impact college admissions.
Traditionally, these conditional testing problems are approached by modeling Y against X and Z through some parametric or semiparametrie model. However, this strategy has been criticized due to potential model misspecification and limited observations of Y. As an alternative strategy, the model-X framework and Conditional Randomization Test (CRT) propose testing for the general conditional independence hypothesis Ho X TL Y Z, free of any specific effect parameters [2]. The CRT assumes the distribution of X Z be known and can control the type-I error exactly, allowing for the choice of any flexible, black-box test statistic. This strategy is particularly useful when there is either strong and reliable scientific knowledge of the distribution of X Z or an auxiliary dataset of (X, Z) of large sample size, known as the semi-supervised setting.
In practice, testing for conditional independence frequently in- volves using data from source population to draw conclusions about target population. This situation presents challenges due to potential differences in the distribution of variables be- tween the two populations. For example, economists may be interested in whether college admission (Y) is independent of family income (XX), conditioning on variables such as GPA, extracurricular activities, geographic location, and other demo- graphics (Z). In the source population, the relationship might be influenced by factors like wealthy parents investing in SAT preparation, which boosts admission rates-a relationship that may not exist in target population where such preparation is less common. Although Y may not appear independent of X given Z in the source population, the conclusion could vary significantly in the target population. This discrepancy under- scores the need for robust and flexible testing procedure that can adapt to shifts in distributions.
csPCR PCR
Typea Bate
6.-
01
Figure 1: Type-I Error rates of our pro- posed csPCR and the source-only PCR on simulated example. The Type-I er- ror inflation of PCR demonstrates that source analysis is not valid or generaliz- able on the target due to covariate shift.
More specifically, we address the covariate shift scenario, where the distributions of the treatment variables X, the confounding variables Z, and some surrogate or auxiliary variables V (e.g.. SAT scores) may differ between the source and target populations. However, the conditional distribution of given X, Z, and V remains the same between them. In such scenarios, our goal is to leverage information from the source to accurately test for conditional independence in the target population without the observation of Y on target. In the scenario we consider, the presence of and potential differences in P(V X, Z) between the source and target populations may lead to the conditional independence X Y Z not holding simultancously in the two populations. Specifically, because
f P(Y x, Z,V.V x, X, dV,
P(Y X,
the conditional distribution of Y given X and Z can vary between populations. This underscores why the problem is non-trivial.
See Figure I for example of the consequences of such covariate shift.
In this paper, we propose a novel conditional independence test suitable for covariate shift scenarios. Our method builds upon the Pearson Chi-Squared Conditional Randomization (PCR) test, a powerful model-X testing procedure that effectively addresses broader range of alternative p-value distri- butions than the vanilla CRT [5]. Methodologically, we make two major contributions. First, we introduce importance weights into the label counting steps of the original PCR test, making the new test valid under covariate shift. These weights adjust the importance of each sample according to its density ratio, effectively rebalancing the source data to match the target population' 's distribution. Second, we introduce a power enhancement method that employs the control variates method to reduce variance in the test statistics. Although importance weights can increase the variance in test statistics, especially when the density ratio can become extremely high, potentially reducing power, our power enhancement method effectively addresses this issue. Together, these innovations enable us to develop a PCR test that is both powerful and valid under covariate shifts.

---

noisy measure The rest of the paper is organized as follows: In Section 2, we provide formal introduction to the problem setup. In Section 3, we introduce the proposed Covariate Shift Corrected Pearson Chi-squared Conditional Randomization (csPCR) test and establish that the proposed csPCR test controls the Type-I error asymptotically. In Section 4, we demonstrate the empirical performance of the csPCR test through simulation studies. In Section 5, we apply the proposed csPCR test to real-world dataset to assess the impact of a COVID-19 treatment on the 90-day mortality rate among patients.
1.1 Related Work
Our work builds upon the model-X framework and the conditional randomization test proposed by Candes et al. [2]. The particular method we develop is based on a variant of the vanilla CRT, the Pearson Conditional Randomization (PCR) test [5]. Recent advances in the CRT include improving computation time [7, 10J, studying robustness [6, 11], and examining statistical power [19]. The focus of this paper, different from the above, on how to build valid CRT procedure when there is covariate shift. The paper is also complementary to the above literature: for example, we hope that future work can conduct theoretical power analysis for our procedure or develop double robust version of the procedure just like in [6]. Finally, we note that surrogate variables play crucial role in this paper: because the distribution of the surrogate variables different in the source and the target population, naively testing the conditional independence hypothesis in the source population can yield invalid conclusions for the target population. A surrogate or silver standard label is a variable that is more feasible and accessible than Y in data collection and can be viewed of Y. For example, tumor response rate is often used as an early endpoint surrogate for the long-term survival outcome [3], and blood pressure is commonly used surrogate for heart attacks. Surrogate variables are also commonly used in environmental studies and economics. Surrogate variables also play important role in the paper by [6], albeit in different way, where the surrogate variables used to learn the distribution Y X, Z and to further improve the robustness of the CRT procedure.
Statistical learning and inference under covariate shift has been extensively studied over the past years. As seminal work in addressing covariate shift bias, [4] proposed density ratio weighting approach using kernel mean matching to characterize the adjusting weights. Their key idea of importance (re)weighting is intrinsically connected with early work in broader contexts like importance sampling [15, e.g. and semiparametric inference [13, e.g.l. [8] extended this idea to doubly robust framework accommodating surrogate variables like and being more robust to the misspecification or poor quality of the density ratio models. [18] handled more challenging scenario with severe shift and poor overlap between the source and target populations. Among this track of literature, [17] is the most closely related to our work as they also considered conditional independence testing under distributional shifts and proposed a general testing procedure base on importance sampling (IS) allowing for the use of CRT. Different from us, their work does accommodate the covariate shifts of some surrogate or auxiliary V. Moreover, as will be shown in our numerical studies, their general IS testing strategy can encounter the loss of effective sample sizes and be less powerful than ours.
2 Problem Setup
2.1 Conditional Independence Testing under Covariate Shift
Let Y E IR denote the outcome variable, X E IR the treatment variable, 2 E RP vector of confounding variables, and V E Rd a vector of surrogate variables. To make the problem more concrete, consider the following two examples:
Example 1 (College Admission). Y is college admission, X is family income, Z includes a number of factors such as GPA, extracurricular activities, geographic location, and demographic information, V is the SAT score. In this case, V is easier to collect compared to as the college admission requires individual-level surveys.
Example 2 (Health Outcome). Y is a long-term health outcome, X is a medical trealment, Z includes factors such as age, gender, and health history, V includes surrogate variables like blood pressure, BMI, and duration of hospital stays post the treatment, which can be measured within much shorter term than Y.
3

---

Zs
ZT
Xs
Vs
Ys
X
VT
YT
(a) Source population.
(b) Target population.
Figure 2: Direct acyclic graphs illustrating possible differences between the source and the target populations.
Consider scenario involving two distinct populations: the source population 8 and the target population T. We collect data from the source population with the goal making inferences about the target population. The source data contains independent and identically distributed samples of Y,) E R", x X2,........
(Y, ,, Z, V.). for .....,,7. n. Let y = Raxp, and V (V,., V2....., VT.) Raxd We are interested in testing the following conditional independence hypothesis in the target population:
E R",
(Z, 72....., 7n..)
Ho : X IL Y Y Z.
We assume that the conditional distribution of Y X, Z, V is the same in both populations; however, the distribution of (X, Z, V) varies between S and T. More precisely, the joint distribution of Y, X,X, Z, V can be described as follows:
Ps(Y, X, V) = Ps(X, Z, P) on S, Pr
Z, 2) PY
T.
This situation is referred to as the covariate shift scenario because the distribution of the covariates X, Z, and V in the source population S does match that in the target population T.
Let's understand the above assumption and its implications through the two examples above. In the college admissions example, it is plausible to assume that the rate of college admissions remains consistent across the two populations when conditioned on the SAT score, family income, and other confounding variables. However, the joint distribution of X, V and Z can differ: in the source population, if wealthy parents frequently invest in SAT preparation, boosting admission rates, this relationship may not hold in target population where such preparation is uncommon. In such cases, is thus possible that X Z in the source population but X IL Y Z in the target population (see Figure 2 for such example). In the health outcomes example, it is again plausible that the conditional distribution of long-term health outcomes given the treatment variable, confounding variables, and surrogates remains the same across the two populations. However, the assignment of the treatment may depend differently on the surrogate variables across the two populations. Therefore, it's possible that X IL Y in one population, but in the other.
In both examples, we can see that the result of naively applying a valid conditional independence test on the source population cannot guarantee valid conclusion for testing Ho in the target population. Therefore, we need to develop new tools for addressing covariate shifts conditional independence tests.
2.2 Model-X Framework
In this paper, we operate within the model-X framework, as described by Candes et al. [2], which assumes that the joint distributions of covariates X, V, Z are perfectly known in both the source and target populations. This framework is particularly suited for scenarios where: (I) there is substantial prior domain knowledge about the covariates X, V. and Z, (2) there is significant amount of unsupervised data for these covariates in both populations, in addition to n labeled observations in the source population, characterizing semi-supervised setting.
An example of the first scenario can seen in genetics, where researchers have well-established models for the joint distributions of single nucleotide polymorphisms (SNPs). For the second scenario, consider our earlier example involving health outcomes. Here, the outcome variable Y represents fong-term health outcome that is more costly sensitive to measure compared to the shorter-term

---

variables X, V. and Z. In such cases, the variables X, V, and Z are typically easier and less costly to collect, frequently resulting in a semi-supervised setting these health-related studies.
3 Method: Covariate Shift Corrected PCR Test
3.1 Incorporating the Density Ratio into the PCR Test
In Section 2.1, we discussed how naively applying conditional independence tests the source data cannot guarantee valid conclusions for the target population. To address this issue, we must incorporate information about the differences between the two populations into our testing procedure. In particular, we will make use of the density ratio defined as:
Pr(X, e(x, = 2s(X,2,V)
(3)
This ratio measures the relative likelihood of observing each combination of variables (X, V) in the target population compared to the source population. By reweighting the data points in the source population using this density ratio, we effectively transform the source distribution match the distribution of the target population, thereby addressing the covariate shift problem.
More specifically, build method upon the recently proposed Pearson Chi-Squared Conditional Randomization (PCR) test [5]. Compared to the vanilla CRT, the PCR test is designed to be more powerful across a broader range of alternative p-value distributions. At high level, the PCR test assigns a label to each data point following a counterfeit sampling step and a subsequent score computation step. Under the null hypothesis that X IL Y Z, the distribution of these labels should be uniform across all possible labels. The PCR test then rejects the null hypothesis the empirical distribution of the labels deviates significantly from uniformity, as determined by Pearson's 's chi-squared test.
Under distributional shift, if the data points were sampled from the target population, then the distribution of the labels would be uniform. However, since the data points are actually sampled from the source population, they must be reweighted using the density ratio. More specifically, in the final step of the PCR test, where the Pearson's 's chi-squared test is applied, we consider not the count of data points for each label, but the sum of the density ratios of the data points for each label instead. Under the null hypothesis, each sum should approximate n/ L,, where L is the total number of labels. Consequently, modify the Pearson's chi-squared test to determine whether these weighted sums deviate significantly from n/L.
Based the above intuition, propose the Covariate Shift Corrected PCR (csPCR) Test, as outlined in Algorithm
In Algorithm 1, lines 1-7 correspond to those in the original PCR test. These lines initiate the test by
generating counterfeit samples X ( Assuming the source and target populations were identical, un- (M)
der the null hypothesis, the random variables (X Y, 2,), (X) Yj. Z,)..... .........2 (X) (X(M) Y,, Z,) would
be exchangeable. Consequently, the rank R2 would be uniformly distributed over (1,..., M + 1) in
the absence of ties, leading to a uniform distribution of the labels as well.
Lines 8- 10 in Algorithm address the covariate shift by incorporating density ratios importance weights into W,. Due to this redefinition of W,, the null distribution of the final test statistic Un,L is also different. Therefore, we also adjust the rejection threshold from the quantile of chi- squared distribution, as in the original PCR test, to the quantile of the weighted sum of chi-squared distributions.
3.2 Power Enhancement
To effectively address covariate shift, incorporating density ratios as importance weights into the PCR test is essential. However, when these ratios become large, they can increase the variance of the statistics W, This elevated variance can diminish the test's power. Therefore, developing methods to reduce this variance is crucial for maintaining the power of the test.
5

---

Algorithm Covariate Shift Corrected PCR (csPCR) Test.
Input: Data Dy (y, x. Z, V), the density ratio the test statistics T, integers K, L > 1, and the significance level oz.
1: Take M KT 1.
2: for each data point j
do 3: Draw M i.d samples x ........ (MM from Pr(X) Z).
4: Use T to score the initial data point (Xj. y, 2z) and its M counterfeits (X) .y. Zz)
T T(2,, ,, Y,, [x)
T T(X,) .Y, Zz), for E (L....,,M}
(0)
5: Let R3 denote the rank of T, among (T,, (1 ..... T(A)), with ties broken randomly.
6: Partition (l,..., M 1} Si U... US, with S₁ 1 1l.,,,..... (K3. Assign label
l 2, 21. L} to sample j if Rg E St.-
7: end for
1,XX,,2, Z,. V2) for each E {1. nz). 9: for each label L e {1. 2..................... ...., :): do
8: Let w/y = e.
10:
Let W₁ be the sum of L-labeled importance weights: W3 = Σ w 1[e = l}. j-1
11: Let De be the sum of (-labeled squared importance weights: De Σ w 1lly ej.
12: end for
13:
Let Q2
L diag D1, , - 1 LxL-
14: Calculate the test statistic L as follows Un,L = Σ (e 2)²
Output: Reject the null hypothesis if Un,L 3 ii otherwise, accept the null hypothesis. Here, 0 a is the a quantile of the distribution x where A xa denotes that xTxx for S).
To this end, we introduce a control variate function a, allowing a(X, Z, V) to serve as control variate in reducing variance in Wi [14]. Specifically, for a chosen Ye, we define
W
Σ wy [Lte, = e) - -eea(X, Z,, [3, +3] + [a(x,
(5)
We can then use We instead of We in our algorithm.
We note that for any arbitrary choice of the function and the parameter Ye, the expectation of W would be the same that of Wc:
We
Σ E
W. 1fe, = 0} Σ
Te W.j((X, Zg, + ny la(X, 24) 2,
j-1
(6)
E [e]] mi (Es [e(x, V)a(X -7 [
=
Therefore, even we make a sub-optimal choice of the function a and the parameter Ye in practice, the resulting test (under certain assumptions) will still remain asymptotically valid (see Section 3.3 for more details).
However, for effective variance reduction, it is preferable to have the control covariates a(X, Z, V) well-correlated with the outcome (See Section for practical discussions on choices of the function a). This is quite feasible, especially since the surrogate variable V is likely to be predictive of Y.

---

Algorithm 2 Covariate Shift Corrected PCR Test with Power Enhancement.
Input: Data Dy (v, x Z, V), the density ratio the test statistics T. the control variate function a, integers K; 1, and the significance level a.
1: for each data point j to do
2: Compute the labels E in Algorithm
3: end for
4: Let W/ e Z,, V2)) for each j n1...
5: for each label e 21, ....,
L): ...,.....
do
6: Compute 9e, the regression coefficient obtained by a weighted linear regression of the indicator
function {I( } on the control variate {a(X,, Z, Vz) with weights {w, (w,})=. 3
7:
Compute the augmented version of Wi as
We Σ w [1te, 2) Sea Sea Vy)1 Mi [aX, [a(]]
8: end for
9: Let W
wy [1 f) 5x Z, Vi)l + 9elT [a(X; 2, V111
for & k L,
j E n.
10: Calculate the sample covariance matrix i., n (W 1xxx) (W - L 1xxa)
11: Calculate the test statistic Un,r as follows Ü.,, 1
" I.) We
Output: Reject the null hypothesis if Ün, 2 a otherwise, accept the null hypothesis. Here,
00 a is the 1-a a quantile of the distribution X where A x² denotes that A x'x for S2..
We would also like discuss the choice of ye. According the control covariate literature, with fixed function a, the optimal choice of 7e that minimizes variance is given by:
Cov Wy { O. wja( X,, Zzj W2)
Var [w,a((X, Zj. Wj)]
This coefficient is also the same as that obtained from a linear regression [14]. Thus, when implement- ing the algorithm, we take to be the regression coefficient obtained by running a weighted linear regression of the indicator function {1 {Ez e} }ブーı on the control variate {a(X,, 2,, V weights {
} with
We have outlined the new esPCR test, including this power enhancement step, in Algorithm 2.
3.3 Theoretical Properties
In this section, we establish that the proposed tests control the type-I error asymptotically. Further- more, show that the power enhancement step effectively reduces the variance of the statistics We. which can typically improve the power.
Assumption 1 (Fourth moment). The fourth moment of the density ratio e(X,ZVV) is finite: Es [e(X, control variate function is also finite: Es [e(X, Z, a.Vaa((XX Z, Z,V)* s.
vy*] < cc. Furthermore, the fourth moment of product of the density ratio and the 22,V) 2,X2,
Theorem 1 (Valid Tests), Under Assumption assume that the null hypothesis of X JL Y holds in the target population, then
lim I [Algorithm rejects]
(8)
lim PP [Algorithm 2 rejects] = a. m=
(9)
7

---

Theorem 2 (Variance Reduction). Let Wi be the statistics computed in line 10 in Algorithm 1, and Wi be the statistics computed in line 7 in Algorithm 2. Under Assumption 1,
lim sup 72-00
Var
W
Var (wi))
(10)
4 Numerical Simulation
In this section, we present simulation studies to assess the performance of our proposed csPCR method and its power enhancement version denoted csPCR(pe), and compare them to benchmark method. The benchmark method adopted is an importance-res rrootaanceresamplling based method [17], denoted as the IS method. For fair comparison, used the same PCR statistic as method for the testing with IS. We use a significance level of a = 0.05.
4.1 Simulation Setup
We consider a semi-supervised setting where we have large volume of unlabeled data of (Xj, Zy, V/)) from both the source and target populations. In addition, we have a small number of labeled data of X,, Zz, Vj) from the source population.
We separate confounding variables into two sets: = (Z,, Zmul), where Z, is the relevant set and Zuull is the null set. The relevant confounding variables Z, are generated i.d. multivariate normal, with mean for the source population and I for the target population to simulate the distributional shift in Z, where Z, RRP and set p 5. Null confounding variables Zuull generated independently with no correlation to other variables, modeled as (0.1,, , IT) with 50 for sparse high-dimensional settings in both populations.
The treatment variable X and the surrogate variable V are conditionally generated based on Z. Specifically, X is modeled identically across both the source and target populations as N(w Zr, 1), where u is a predefined parameter vector that remains the same for both populations.
For V, it is modeled differently in the two populations, represented as (1 8ag,7 sin(X). X): 1). Here, us and er are predefined parameter vectors for the source and target populations, respectively. The parameter a varies between populations (ag for the source and ar for the target), controlling the effect of X on V, modeling the indirect effect. The factor 0 modulates the nonlinear component of this relationship.
The outcome variable Y is generated for both populations using the same conditional model over (X,
YI(X, Z, V) ~ (NT(T BV + 7X,
where 3 and y control the effects of V (indirect) and X (direct) on Y, respectively.
We generate 1000 unlabeled source and target samples to estimate the density ratio and generate 500 labeled source samples for testing. Moreover, in the simulation, we assume we have full knowledge of the joint distribution of (X, Z) and estimate V|X, Z using Elastic net regression model with 5-fold cross-validation [20]. For the test statistic T in the algorithm, we choose a simple function T(X, Z, V) = Y X. For each parameter iteration, we conduct 1000 Monte Carlo simulations estimate the Type-I error and power. We estimate the covariance matrix of the sequence of W,'s using the Monte Carlo method and use the momentchi2 package [1] for calculating the p-value. Additionally, we empirically choose the best hyperparameter L for all our experiments through additional experiments shown in Appendix B.2.
4.2 Simulation Results
In Figure 3, we choose as and ay with the benchmark. The left panel shows the Type-I error rate as the sample size of the data used to estimate the density ratio, nh. varies from small to large. There appears be a slight Type-I enror inflation for all three methods when the sample size n. is small, but the Type-I error quickly converges to the ideal level of 0.05 as n. grows larger. Moreover, our methods show more stable Type-I error control than the benchmark method when the estimation sample size is low. The right
compare the Type-I error control of our methods
8

---

panel shows that when the density ratio is well approximated, all three methods attain good Type-I error control regardless of the change B, i.c., the strength of the indirect effect, but the csPCR and cpPCR(pe) methods have more stable control.
To evaluate the statistical power of our csPCR test, we choose = 0 and at 2, that the null hypoth- esis holds true in the target popula- tion but in the source population. As Figure 4a shows, both the csPCR and the csPCR(pe) methods have uni- formly higher power than the bench- mark method as we vary the indirect effect size B. For example, when / = 1.4, the benchmark IS method has power of 0.33, the esPCR method has a power of 0.44, and the esPCR(pe) method can attain a power of 0.8.
6ub
CSPCR(er
100
642
802
16.0
1000
0.00
Figure 3: Comparison of Type-1 error control across three methods.
When we fix the indirect effect 3 = 2 and vary the direct effect of X (7), as shown in Figure 4b, our methods still exceed the benchmark, and the power enhancement significantly improves the original version of the test, For example, when = 1, the benchmark IS method has a power of 0.4, the csPCR method has a power of 0.62, and the cpPCR(pe) method can attain power of 0.86.
CSPCR
EPPCR
exPCRipp)
0.8
csPCRipe!
0.6
0.6
02.
0.
3.
38
0.0
02
04
0.6
0.8
12
14
16
0.0
01
02
03
0.6
0.5
0.6
088 0.9
(a) Increasing indirect effect |
(b) Increasing direct effect 7
(c) Varying nonlinear effect size 0
Figure 4: Comparison of statistical power of three methods as effect size varies: (a) indirect effect (3, (b)
direct effect 7, and (c) nonlinear effect size e.
We also test how adding a nonlinear component to the indirect effect affects the power when we assume linear model of V X in the estimation stage. This can be helpful in assessing the performance of our methods under model misspecification. As Figure 4c indicates, as the nonlinear effect increases, the power of all three methods decreases, though our methods still significantly exceed the benchmark. Interestingly, we observe that as 0 - 1, i.e., there is a full nonlinear component without linear component, the advantage of the power-enhanced version original csPCR test disappears. This occurs because when the V X, Z model is misspecified and the density ratio estimation inaccurate, the variance reduction in the control variates step reduces variance in the "wrong" direction, and thus does not improve the power of the original method.
the
4.3 Effective Sample Size
We notice a series of work in measuring the effective sample size (ESS) of the density ratio reweighting approaches [9]. Among them, one of the most common measure is nei (E wx)) / Σx= w When the covariate shift between the source and target becomes stronger, the variance of the importance weight w, tends to be large and neff will become smaller, which could result in lower power. We carry out simulation studies on the relationship between the power of csPCR and the ESS determined by the degree of covariate shift as discussed in Appendix B.3.
5 Real- World Application
The COVID-19 pandemic has presented unprecedented challenges to global health systems, with high variability in outcomes based on demographic and clinical characteristics. Early identification of patients at high risk for severe outcomes, such as mortality within 90 days of hospital admission, crucial for timely and effective treatment interventions. This study leverages extensive hospital data to develop models predicting 90-day mortality following hospital admission due to COVID-19.

---

For this study, we extract patient data spanning from January 2020 to December 2023 from Duke University Health System (DUHS), focusing on individuals admitted with COVID-19. This period encompasses multiple waves of the pandemic, influenced by various circulating variants.
Our dataset comprises patient records for a total of N 3, 057 individuals admitted with COVID-19. The outcome Y is defined as mortality within 90 days since hospital admission due to COVID-19, The treatment variable X is defined binary, where indicates the administration of any COVID-19 specific medication (explained in Appendix C) and otherwise. The covariates Z include comorbidity indices (renal disease, diabetes without complication, diabetes with complication, local tumor, and metastatic tumor), age. gender, and race, which are critical for adjusting the risk models due to their known influence on COVID-19 19 outcomes. The length of hospitalization, denoted as V, is standardized to follow a standard normal distribution (with a mean of zero and standard deviation of one), facilitating comparisons and integration into predictive models regardless of original scale or distribution.
The dataset is segmented into two distinct groups based on the date of hospital admission align with pivotal changes in virus strain predominance and public health guidelines. The source data comprises COVID-19 admissions prior to November 30, 2021, with sample size of N₁ 1, 131 patients. The target data includes admissions from November 30, 2021, through December 2023, totaling N₂ 792 patients. This temporal division allows for the analysis of trends and outcomes associated with the evolving pandemic landscape. Prevalence of the 90-day mortality outcome within the source data 14.3%, reflecting the impact of earlier virus strains and treatment protocols, while in the target data, the prevalence is substantially lower at 3.7%, possibly indicating the effect of improved treatments and vaccines, as well as the influence of different virus variants over time.
Table 1: p-values of diffferent methods on COVID-19 dataset
Method csPCR csPCR(pe)
IS
p-value 0.025
0.032
0.663
For the analysis, we divide 50% of the source data, comprising 565 individuals, alongside the entirety of the target data, to estimate the density ratio. Density ratios X, Z are estimated using probabilistic classification method [12], while the density ratio of V|X, Z is determined through Elastic Net regression. For all three methods, the test statistic T is chosen to be T(X, V,) Y xxX. As indicated in Table 1, both csPCR and csPCR(pe) give statistically significant results, whereas the IS method does not. The statistically significant results are consistent with biomedical literature. For example, through systematic review and meta-analysis, [21] reported that Bamlanivimab is effective in reducing the mortality rates of COVID patients. In cohort study, [16] also found similar effectiveness for Nirmatrelvir-rioonvii
These results align with our findings from the simulation study and demonstrate that our method has increased power compared with the benchmark method.
10

---

References
[1] Dean A Bodenham and Niall M Adams. A comparison of efficient approximations for a weighted sum of chi-squared random variables. Statistics and Computing. 26(4):917-92) 2016.
[2] Emmanuel Candes, Yingying Fan, Lucas Janson, and Jinchi L.v. Panning for gold: model-
x "knockoffs for high dimensional controlled variable selection. Journal of the Royal Statistical Society Series B: Statistical Methodology, 80(3):511-577, 2018.
[3] Emerson Chen, Vikram Raghunathan, and Vinay Prasad. An overview of cancer drugs approved by the us food and drug administration based on the surrogate end point of response rate. JAMA Internal Medicine, 179(7):915-921, 2019.
[4] Jiayuan Huang, Arthur Gretton, Karsten Borgwardt, Bernhard Schölkopf, and Alex Smola. Correcting sample selection bias by unlabeled data. Advances in neural information processing systems, 19, 2006.
[5] Adel Javanmard and Mohammad Mehrabi. Pearson chi-squared conditional randomization test. arXiv preprint arXiv:2111.000 11.0022, 2021.
[6] Shuangning Li and Molei Liu. Maxway ert: improving the robustness of the model-x inference. Journal of the Royal Statistical Society Series B: Statistical Methodology, 8415):144 2023.
[7] Molei Liu, Eugene Katsevich, Lucas Janson, and Aaditya Ramdas. Fast and powerful conditional randomization testing via distillation. Biometrika, 109(2):277-291 2022.
[8] Molei Liu, Yi Zhang, Katherine P Liao, and Tianxi Cai. Augmented transfer regression learning with semi-non-
semi-non-paramettric parametric nuisance models. Journal of Machine Learning Research, 24(293): 1 1-50, 2023.
[9] Luca Martino, Víctor Elvira, and Francisco Louzada. Effective sample size for importance sampling based on discrepancy measures. Signal Processing, 131:386-401,2000
[10] Binh T Nguyen, Bertrand Thirion, and Sylvain Arlot. A conditional randomization test for sparse logistic regression in high-dimension. Advances in Neural Information Processing Systems, 35: 113703,,,, 2022.
[11] Ziang Niu, Abhinav Chakraborty, Oliver Dukes, and Eugene Katsevich. Reconciling model-x and doubly robust approaches to conditional independence testing. arXiv preprint arXiv:2211. 14698, 2022.
[12] Jing Qin. Inferences for case-control and semiparametric two-sample density ratio models. Biometrika, 85(3):619-633 630, 1998.
[13] James M Robins, Andrea Rotnitzky, and Lue Ping Zhao. Estimation of regression coefficients when some regressors are not always observed. Journal of the American statistical Association, 89(427):44-866, 1994.
[14] Sheldon M Ross. Simulation. academic press, 2022.
[15] Donald B Rubin. The calculation of posterior distributions by data augmentatiin: Comment: A noniterative sampling/importance resampling alternative to the data augmentation algorithm for creating a few imputations when fractions missing information are modest: The sir algorithm. Journal of the American Statistical Association, 82(398):543-546 1987.
[16] Kevin L. Schwartz, Jun Wang, Mina Tadrous, Bradley J Langford, Nick Daneman, Valerie Leung, Tara Gomes, Lindsay Friedman, Peter Daley, and Kevin A Brown. Population- based evaluation of the effectiveness of durmatrelvir-rioonae for reducing hospital admissions and mortality from covid-19. Cmaj. 12206)6220-E226, 2023.
[17] Nikolaj Thams, Sorawit Saengkyongam, Niklas Pfister, and Jonas Peters. Statistical testing under distributional shifts. Journal of the Royal Statistical Society Series B: Statistical Methodology, 85(3):597-663, 2023.
11

---

[18] Kaizheng Wang. Pseudo-labeling for kernel ridge regression under covariate shift. arXiv preprint arXiv:2302.10160, 2023.
[19] Wenshuo Wang and Lucas Janson. A high-dimensionall power analysis of the conditional randomization test and knockoffs. Biometrika, 109(3):631-645 2022.
[20] Hui Zou and Trevor Hastie. Regularization and variable selection via the elastic net. Journal of the Royal Statistical Society Series B: Statistical Methodology, 67(2):301-320, 2005.
[21] Ling Zuo, Guangyu Ao, Yushu Wang, Ming Gao, and Xin Qi. Bamlanivimab improves hospitalization and mortality rates in patients with covid-19; systematic review and meta- analysis. The Journal of infection, 84(2):248, 2022.
12

---

A Proofs
A.1 Preliminaries
Throughout this section, we write Sazy, 2j, U1)) 8j as the label assigned to sample j in Algorithms 1 and 2, instead of using f, This notation helps avoid confusion between different label choices.
Proposition 1. Assume that the conditional independence X TL Y Z holds on the target population T. Let j Zj. Uz) denote the density ratio. For any integer & E [1, Lj. the following holds:
1
Uz) = f}] = L Esle(2z,2z,0) u,)) 25,
Proof of Proposition 1. For simplicity, denote wj = e(X,, Zz, Vi)) and 8j = Sy 2,, Vz)
E 2, V3) (X =]]]
Es
E W3 P(s; 8 Y,, 23, X3, [3)] 2j, V/3
Ess
Wy E [P(s; e V, X3, v5)]
X3, V3
wy Ps( Ls (3) (P((s) = el 2z X,, Vi)] dZ, dX, dv,
(2,X1.15)
(11)
Pz X,, VT) E P(s, Y,, Zy X,, Vy) dz, dx, av,
12,
= Er
[((ss) = e X, K3)1 X,
= E, [P(s) = El Y Z,,,, VD]
1
L
The last equation follows from results in the man--oovriate-shhie scenario, e.g., from [5].
A.2
Proof of Theorem
A.2.1 Results for Algorithm
Let (Wa)..,..., be the sum of weights and 22 be the sample covariance matrix in Algorithm By Proposition 1, we have that
E(Wa) = n E[w, 1[e, = e7l
L
Note that the Wy's are sums of i.i.d. random variables, and thus by the Central Limit Theorem, as n 00,
A
L V n
W
n W2
L,
n T
n Wt
T.
4 (0,
where for any l,tt [I)
w 1[s1 = 011181 = m9]-1
Dr = LCov(wi {sy = f) wi ist = =(+}) = LEs
LEs w/ 1(81 f) {e (*] I
Therefore,
Un = AJA,, d 2
Xn-
13

---

Next, we will focus on the variance estimation part. We will show that 22, - as n 00. For any E* E [I,...,L}.
Pnee - {e= {*} D
T. - -1e e*) - - w/ 11e f) - 1
2 1 {*} LEs w1 1/81 = e3] - T. Ogge+
Up now, we have that
U A,A, 4. Xi+
and 0 Ps 0z
Therefore,
P(Algorithm 1 rejects) = P(Un,L 3 8/ - P(x > d2,a) = a.
A.2.2 Results for Algorithm
Let (We)e=i,.... and Uin be the sum of weights and the sample covariance matrix in Algorithm 2. Let fye be the estimated coefficient in Algorithm 2.
Recall that in (7), have identified the optimal choice of ye. We will start by working with this optimal choice and show that the 'y is close to it. Define
We = Σ wy (1te, = 2 - rea Zz. VJ)) + mye V)], j-1
(e,(7) = w (1(,, >aa(X, 2z, V2)) + 7 Eaa(X V)]. and
H3 = w x, Zz. V₁) ET [a Z, V)].
H, Therefore, have We Σ, (e)( and We Σ, Key(Se) We (er 76 E, H Note that by (6), B(H₂) = O. By Proposition and (6), we have B(We) E. Furthermore, because We is a sum of i...d random variables, we have that as n + 0.
V
L Wi
n W2 T
n T. WL
n T.)
4 (0)
(12)
where for each e, e (1,....,II) L},
00 = LCov (Kes (7e), Kes ((r=))))))
Therefore,
d X ÚN ÄTÄ,
Next, we will show that the actual statistic Ün,r. is close to Ün.,. and that the estimated variance matrix also close to L.. We start with noting that the estimator î from linear regression is close to the optimal choice "/ defined in (7): by the Central Limit Theorem, de = 7 + Op(1/ /n). And thus
11
We Σ wy (1f6, E3 Rea(X Zz, VJ)) (ax
Σ 3-1
wylllr = ffee Σ w. ww,a(X, 4z, 13) nEronE Z
wn1(( = T
Σ wwya(Xx Zz, V3) - O,(11)
J=1
W,+
14

---

The second-to- lo-last line is because 8 + (1//vnn) and the terms inside the parenthesis, E, H, is a sum of n independent mean-zero random variables. Therefore, together with (12), by Slusky's Theorem, we have that
An = V
L
W1
n
W2
n L
WL
n 7)
L)
$ N2,
L
and thus,
Ún,L- = A,A,, d X
We will work on sample covariance matrix now. Recall that the sample covariance matrix 82 = L L(W 1. (W 1, 1Lxx) where Wes wy [i[e, f} ) Gj, Vj]] + SolET
= Keo(fe). Let's start with WWT For any E, l (1....,4).
(WWT) ),,e
Σ (Keelre) (ffe - (e) (Kee,(1e) - (9c* - 7ea ){²
- D Keng(t))Kom - 7e) - H, ((70m) -19e* 7(0) Σ He
+ +fr - 7e((re -7** Σ H3
Kesltt)) +O(())
Therefore, by the law of large numbers,
I. "(WW)) Je.* (ew))e.e I. Keslt))) ((m))))))kk,,()))0000))0 LE (o,()))Kn,,(( n n
Similarly, for WI have that for any e, 10 E [I,...,,.}
(W1 )e,e) kes(te) (e)((r)) (fr -1) = +kp((t)+
Therefore, again by the law of large numbers,
L W (WI)),.e Je.0
L Kes(ro) + O LE +
N
N
Combining the above results gives,
in,,,e
L (w I' 1x I
1
1 [xx)
1,80
LE [Kg (Te) (16e)] LE [Kr [ + O,(1/vm
[T
L Cov [Ke,(1e), Ke (1e4) + (p/1//)
= -ee (p(//nn).
Therefore, Ωr L Ω
To summarize, we have that
Ün A,A.A d X and i P, 0i
Therefore,
P(Algorithm 2 rejects) P(Un.L 2 ( P(x2 0itaa)
15

---

A.3 Proof of Theorem 2
Similar to the proof of Theorem 1, we define
We Σ wy (1fe, - ( + ny)BB/a(XX,,
5-1
(e)(7) = W (1fe) = 2) - xa 2z v3) + (a 2, v)]. and
Hj = w w (XX, Zz. V2) - Ey [\)
Therefore, we have We 2; (e)( and Wi = Σn Ked = W - (ir - 7e) Enj Md.
We know from the literature that Tt is the optimal choice of and thus Var
W < Var [We]. We
will then move to show that Var We is close to Var [e]] and thus asymptotically greater
than Var [We]-
To this end, note that
Var
We
Var
We (9/ - 7e) Σ Hr
Var
We
2 Cov
We, (ire
3e) "n)CH, H1} Var [(fr 7e) Hj] HHp
Var
W
+ 2
Var
W
(15x 7e) H,₂ + ((fe-11)) -1) H3)² H1)
7m)
But we also know from the proof of Theorem that te - 7t 2. Then, because of the bounded fourth moment assumption, by the Dominated Convergence Theorem, we have that
E ((5t-70)) 1e) 7 (<
Therefore,
lim sup
Var
W
Var
We
n- 12
Finally, we note that Var [Wi] = 2(n). and hence
lim supsuupuaaaa n=
Wi
Var [Wi]) & 1.
B
Additional Simulation Results
B.1 Running time
All experiments run Macbook Pro 2022 M2.
Artificial dataset: Regarding running time for one iteration including density ratio estimation and X model fitting (on average), esPCR took 5.12s, csPCR(pe) took 14.95s, IS method took 1.5s, PCR took 1.25s.
Real-world application: Regarding running time for one test procedure, csPCR took 3.41s, csPCR(pe) took 11.32s, IS method took 0.81s.
B.2 Finding optimal hyperparameter
We find the optimal L value for the testing algorithm by performing numerical simulations, evaluating its Type-I error control and power. We adopt the same numerical simulation setup as in the main text Section 4. We first choose as 1 and at 0 and also fix ß 1 to compare the Type-I error rate for different choice of of the csPCR method. We perform experiments with both true density ratio and estimated density ratio. The results are shown in Table 2.
16

---

Table 2: Type-I Error Rates Different Levels of of csPCR Method
L
2
5
10
15
20
True Density Ratio
0,05125
0.05000
0.04575
0.03675
0.02825
0.02425
Estimated Density Ratio
0.04620
0.05025
0.04425
0.03905
0.02725
0.02175
0.8
Power
0.6
CSPCR
CsPCR(pe)
0.0
Figure Comparison statistical power the three methods as the parameter varies.
We also test the power of the esPCR and csPCR(pe) method with different choices of value. We choose ag = 0 and at = 2 and fix 3 = 2.
As Table and Figure shows, L value increases, the csPCR method become conservative with more tight Type-I error control and lower power. We can observe that when we set L = 3, the csPCR method can achieve most stable Type-I error rate control and also highest power empirically. Therefore, in our simulation experiments and real world data experiments, we fix 3.
B.3 Role of effective sample size
notice a series work measuring effective sample size (ESS) of importance weight or sampling in the statistical computation literature, e.g., [Martino, et al, 2017] and others. Among them, (g sx, one of the most common ways is to use the ratio Ref. 2 w5 to approximate the ESS. When the covariate shift between the source and target becomes stronger, the variance of the importance weight w; tends to be large and neff will become smaller, which can result in lower power. Our power enhancement method based control variate could potentially alleviate this issue with properly specified control functions.
In the simulation study, we varied only Ps. the mean of the confounding variables Zy. A higher μ signifies a stronger covariate shift between the source and target populations. From Figure 6, it is evident that as fl increases, the Effective Sample Size (ESS) required significantly decreases, while the power of the esPCR method concurrently declines. These results suggest that increasing covariate shift leads to reduction in ESS and a corresponding decrease statistical power.
B.4 Instability of the Importance Resampling (IS) method
In this section, we will use numerical simulations to ilustrate that the performance the method is subject to the resample size heavily. IS method performs resampling without replacement and typically has to sample a much smaller subset (theoretically, in the order of o( the source data to approximate the target. Consequently, the power of IS is substantially lower than our approach. If
the resample size of IS is overly increased, may fail to control the Type-I error due to excessive similarity between the resampled data and the original source data.
To further illustrate, we conducted additional experiments with varied resample sizes in IS to assess its effect on Type-I error control and power. From Figure7. one can observe that IS starts show high Type-I error inflation when its resample size increases to 400 but still shows much lower power
17

---

1.0
250
CSPCR
0.8
csPCR(pe)
Power
0.6
150
0.4
100
0.2
50
0.0
0.2
0.4
0.6
0.8
0.2
0.4
0.6
0.8
F
Figure 6: The left panel shows the comparison of statistical power of esPCR and csPCR(pe) method as the covariate shift gets stronger. The right panel illustrates how the Effective Sample Size(ESS) changes as covariate shift scale becomes larger.
(by around 0.4) than our method with this resample size (or even larger ones). This indicates that our method achieves better statistical efficiency than IS (DRPL).
15
csPCR
CSPCR
Type-1 Error Rate
csPCR(pe)
csPCR(pe)
0.2
0.2
0.0
0.0
200
400
600
8D0
1000
200
400
600
800
1000
Resample Size
Resample Size
Figure 7: Detailed comparison of Type-I error rate and power of csPCR and the IS method. With the source sample size n, = 1000, we gradually increase the resample size for the IS method from 100 to 1000. The horizontal lines represent the Type-I error rate and power, respectively, of the csPCR and cpPCR(pe) methods (they do not change with the tuning of IS).
B.5 Choice of test statistic
In this section explore the effeet of test statistics the algorithm performance. The main principle of choosing the test statistic to characterize the conditional dependency between and Y under the alternative hypothesis. The test statistic YX may not be the optimal choice and that using (Y E Z)) E[X Z]) could remove the confounding effect of Z.
Inspired by this, used Y(X E| Z1) as the test statistic to conduct additional simulations. As illustrated in Figure8, we find that ((X ( and YX produce nearly the same power for both csPCR and csPCR(pe) with the change of effect size.
C Real-World Application
The specific medication indicated by the treatment variable X includes Ritonavir, Bamlanivimab, Dasirivimab-IIddevinee Remdesivir, Ritonavir Nirmatrelvir, Sotrovimab, Bamlanivimab Etesevimab. For simplicity, X indicates any of these specific medication and X 0 otherwise.
18

---

1.0
1.0
Y-X
V-X
Y. -1x22]
V-JX ((122
0.8
Power
Power
0.0
0.2
1.6
1.2
1.8
Effect
Effect
(a) csPCR
(b) csPCR(pe)
Figure 8: Power against effect size for csPCR and csPCR(pe) with different test statistics X
and (X E[X] Y.)Y. We observe that the power is very similar with the two different test statistics.
C.1 Different outcome
In our real data experiment part, the outcome variable Y is defined mortality within 90 days since hospital admission due to COVID-19 19, In addition, we also analyzed mortality within 30 days since hospital admission. As shown in Table 3, both csPCR and csPCR(pe) methods give significant results, aligning with biomedical literature.
Table 3: p-values of different methods on COVID-19 dataset (mortality 30
Method
csPCR
csPCR(pe)
p-value
0.029
0.013
19

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the
paper's contributions and scope?
Answer: [Yes]
Justification: In the abstract and introduction, we mentioned that we propose a novel Covariate Shift Corrected Pearson Chi-squared Conditional Randomization (csPCR) test. and discussed our methodological, theoretical, and empirical contributions.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We examine the performance of the algorithms under model misspecification in Section 4.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
20

---

Answer: [Yes]
Justification:
The assumptions are listed in Section 3.3 and the proofs are provided in
Appendix appendix.proof.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Details of the simulation studies and real-data application are included in Sections 4, 5 and Appendix B.2.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
21

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: Replication code for our simulation studies is submitted as supplementary material. It will also be made publicly available on GitHub once our paper accepted. The COVID data set used for the real example in our paper is not publicly available due to privacy constraints.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/
httpii/gguiiees//eeuunntttt colobiiissioooicyll for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: // /niiss.... paips.cc/ppll http:/ubllcggguuddee http////giideeeoeeeuubbiiiiii htt/////oobbissioooooo
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer:
[Yes]
Justification: Details of the simulation studies and real-data application are included in Sections 4, 5 and Appendix B.2.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Type-I errors, power, and p-values are provided in Sections 4 and 5.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
22

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We record relevant information in Appendix B.1.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https https://mntt /ttpa:///////pp///eeee cc cctppblic////cccccccccccccccclcl htt.h///iehhishheeeeeee.
Answer: [Yes]
Justification: Yes, our research conforms to the NeurIPS Code of Ethics in every respect, including faimess,, transparency, privacy, and social responsibility.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We have discussed the impact of the paper on the fields of healthcare and social sciences.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
23

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We believe the paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We have cited the relevant papers and packages.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URI.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
24

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasttss has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We have provided detailed documentation of the newly proposed algorithm.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
25

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
26


Reviews:
[{"review_id": "AZQAmRZihI", "reviewer": "Reviewer_G2Ah", "summary": "This paper addresses the issue of conditional independence under covariate shift. The authors' goal is to test the conditional independence for causal inference in the target data. The authors can use the source data whose distribution is potentially different from that of the target data. For this problem, the authors propose using the covariate-shift adaptation method and develop a method for testing the null hypothesis.", "questions": "See above.", "limitations": "See above.", "rating": 4, "confidence": 1, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "This paper addresses the issue of conditional independence under covariate shift, which I find to be an intriguing problem. However, I was unable to comprehend the underlying assumptions of this problem, making it difficult to proceed with the reading.\n\nI understand the objective of this manuscript to be as follows:\n1. The authors aim to test the null hypothesis $H_0: X \\perp Y \\mid Z$ for the target data (Here, I denote the independence by $\\perp$). \n2. There is a possibility that $X \\perp Y \\mid Z$ does not hold simultaneously for both the source and target data.\n3. To test the null hypothesis for the target data, the authors employ an algorithm adapted to covariate shift.\nMy primary concern is whether the assumption \"there is a possibility that $X \\perp Y \\mid Z$ does not hold simultaneously for both the source and target data\" is justified in the first place.\n\nThe authors assume that, in order to use the covariate shift adaptation algorithm,\n- $Y \\mid X, Z, V$ is the same in both the source and target data.\nUnder this assumption, is it not the case that the situation where $X \\perp Y \\mid Z$ holds in one dataset and does not hold in the other cannot exist?\n\n$X \\perp Y \\mid Z$ implies that $p(y, x \\mid z) = p(y \\mid x, z) p(x \\mid z) = p(y \\mid z) p(x \\mid z)$, meaning that $p(y \\mid x, z) = p(y \\mid z)$.\nGiven that we assume $Y \\mid X, Z, V$ to be the same in both the source and target data, does this not imply that whether $X \\perp Y \\mid Z$ holds cannot differ between the source and target data?\n\nDue to this ambiguity, I was unable to further evaluate the paper. I would like the authors to clarify this point so that I can proceed with the evaluation.", "weaknesses": "See above.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We extend our sincere thanks for your reviewing work and important comments. We believe your main confusion lies in the existence of V and the fact that $P(V \\mid X, Z)$ can be different between the source and target. This can possibly cause the situation that $H_0: X \\perp Y \\mid Z$ does not hold simultaneously on the source and target (note that V is an auxiliary feature not included in the hypothesis of our primary interest). For illustration, consider the following simplified example:\n\nLet $X \\sim N(0, 1)$ and $Z = X + \\epsilon_z$ on both source and target populations, where $\\epsilon_z$ is an independent noise term. For the source, let $V_{\\mathcal{S}} = -X_{\\mathcal{S}} + \\epsilon_v$, while on the target, let $V_{\\mathcal{T}} = \\epsilon_v$, where $\\epsilon_v$ is a noise term independent of X and Z. On both source and target, let $Y = X + Z + V + \\epsilon$, corresponding to our assumption that $P(Y \\mid X, Z, V)$ holds the same between the source and target. \n\nIn this case, one can derive that (i) on the source, $Y_{\\mathcal{S}} = Z + \\epsilon_v + \\epsilon$; on target, $Y_{\\mathcal{T}} = X + Z + \\epsilon_v + \\epsilon$. Thus, $X \\perp Y \\mid Z$ holds on the source but not on the target, which underscores the importance of our setup and method.\n\nA more general data generation setup can be found in Figure 2 of our paper. In this diagram, $V$ could be interpreted as a mediator or early endpoint surrogate seen in real-world studies. Taking clinical trials as an example, suppose we are interested in testing the treatment effect of some drug $X$ on some long-term outcome $Y$ such as five-year survival, adjust for some baseline confounder $Z$. In this case, researchers collect $V$ as some early endpoint surrogate that can be measured within a short term, usually a few weeks or months post treatment, e.g., tumor response rate in cancer treatment. Thus, $V$ is easier to collect and also informative to $Y$. Then it is reasonable to assume that $P(Y\\mid X,Z,V)$ is shared by the source and target populations while $P(V\\mid X,Z)$ has a distributional shift between the two populations [Kallus and Mao, 2020]. Please find other real-world examples in our response to Weakness 1 from Reviewer REEU. We hope this clarification addresses your concerns and enables you to proceed with the evaluation. Thank you!\n\n$\\textbf{References}$\n\nKallus, Nathan, and Xiaojie Mao. \"On the role of surrogates in the efficient estimation of treatment effects with limited outcome data.\" arXiv preprint arXiv:2003.12408 (2020).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nEp4uiHFi6", "reviewer": "Reviewer_MwvS", "summary": "This paper introduces the Covariate Shift Corrected Pearson Chi-squared Conditional Randomization (csPCR) test, designed to address covariate shift in conditional independence testing. The csPCR method incorporates importance weights and employs the control variates method to enhance test power and reduce variance in the statistical analysis. Theoretical contributions demonstrate that csPCR controls Type-I error asymptotically. Empirical validations through simulation studies and a real-world application assessing COVID-19 treatment effectiveness showcase the method's effectiveness.", "questions": "- Could the authors provide a detailed comparison between the csPCR test and traditional resampling methods that also use estimated density ratios? This would further strengthen the paper\n- The paper mentions the use of control variates to reduce variance introduced by importance weights. Could the authors discuss how this approach compares to variance reduction techniques used in resampling methods? \n- Could the authors elaborate on any theoretical limitations or potential failure modes of the csPCR method?\n- Are there additional empirical validations planned or underway to test the csPCR method across more varied datasets?", "limitations": "The authors are encouraged to add a separate limitation section in either the main paper or appendix.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. **Theoretical contribution:** The paper makes a theoretical contribution by addressing the covariate shift in conditional independence testing. \n\n2. **Methodological Innovation:** Introduction of the Covariate Shift Corrected Pearson Chi-squared Conditional Randomization (csPCR) test that incorporates importance weights and control variates method to manage covariate shifts effectively.\n\n3. **Empirical Validation:** Extensive simulation studies and real-world application (assessment of COVID-19 treatment on 90-day mortality) demonstrate the practical efficacy and superior power of the proposed csPCR test over traditional methods.", "weaknesses": "1. **Dependence on Accurate Estimations of Density Ratio:** The csPCR test's performance is critically dependent on the accurate estimation of density ratios. This dependence could pose significant challenges in practical scenarios characterized by limited, noisy, or high-dimensional data, where reliable density ratio estimation becomes inherently difficult.\n\n2. **Clarity on the Advantage Over Resampling Methods:** The paper does not explicitly clarify the advantages of the csPCR test over simpler resampling-based methods that also utilize estimated density ratios. While the csPCR integrates density ratios directly into the test statistic calculation and employs variance reduction techniques, it remains crucial for the authors to demonstrate why these features offer substantial improvements over traditional methods that adjust the sample weights during resampling. The discussion should address whether the complexity of csPCR provides tangible benefits, such as improved error rates or robustness in more varied practical applications, compared to potentially simpler resampling approaches.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "$\\textbf{Weakness}$ 1: Dependence on Accurate Estimations of Density Ratio \\& $\\textbf{Question 3}$: Theoretical Limitations?\n\nResponse: We believe our method's limitation lies mainly when the model-X assumption fails, i.e., when the distribution of covariates cannot be accurately learned. In such settings, Type-I error inflation may occur. The robustness of CRT and PCR tests under this condition has been studied extensively [Berrett et al., 2020; Javanmard and Mehrabi, 2021; Li and Liu, 2023]. Theoretical upper bounds for Type-I error inflation have been given; moreover, simulation studies show that CRT and PCR tests are usually more robust than these bounds suggest.\n\nFor our csPCR test, we theoretically believe we can show similar results to those in Section 6 of [Javanmard and Mehrabi, 2021]. Empirically, we conducted simulation studies (some included in our submission, plus new ones) when the covariate distribution is estimated from data: \nOur studies include two scenarios: (i) one has the knowledge of $P(X\\mid Z)$ and needs to estimate $P(V \\mid X, Z)$; (ii) one has to estimate the full $P(X, V, Z)$ without any knowledge. For (i), please refer to Figure 3 for our numerical results showing that even with moderate sample sizes used for learning $P(V \\mid X, Z)$ (e.g., $n_e=500$, equal to the labeled sample size used for testing), the csPCR test still maintains good Type-I error control (nearly no inflation above 0.05). For the more challenging scenario (ii), we have added additional experiments with the full distribution density (or density ratio) estimated using the unlabeled samples. The results can be found in Figure R2 of the supplementary PDF file (attached to the global rebuttal response). In this case, our csPCR(pe) approach can still achieve Type-I error control at $n_e=400$ as well as good power, and the vanilla csPCR shows proper Type-I error control at $n_e=700$, which is still not large compared to the testing sample size of 500. \n\n$\\textbf{Weakness 2}$: Advantage Over Resampling Methods\n& $\\textbf{Question 1}$: Comparison with traditional resampling methods?\n\nResponse: First, we would like to clarify that we are benchmarking against the specific DRPL resampling method (referred to as IS in our paper) proposed in [Nikolaj et al., 2023]. This approach is proposed for the general purpose of testing under covariate shift and is, to our best knowledge, the only existing strategy for conditional independence testing under covariate shift. It is essentially different from traditional bootstrap resampling or importance sampling procedures. Specifically, IS performs resampling without replacement and typically has to sample a much smaller subset (theoretically, in the order of $o(\\sqrt{n})$) of the source data to approximate the target. Consequently, the power of IS is substantially lower than our approach. If the resample size of IS is overly increased, it may fail to control the Type-I error due to excessive similarity between the resampled data and the original source data.\nTo further illustrate, we conducted additional experiments with varied resample sizes in IS to assess its effect on Type-I error control and power; see Figure R1 in the supplementary PDF file (attached to the global rebuttal response). One can observe that IS starts to show high Type-I error inflation when its resample size increases to 400 but still shows much lower power (by around 0.4) than our method with this resample size (or even larger ones). This indicates that our method achieves better statistical efficiency than IS (DRPL). We also find that similar results hold regardless of whether the density ratio and model of $X$ are known or estimated.\n\n$\\textbf{Question 2}$: Variance reduction techniques used in resampling methods?\n\nResponse: As highlighted in our response to your Question 1, the IS method is essentially different from traditional resampling methods. The authors in [Nikolaj et al., 2023] have not proposed any variance reduction methods such as control variate, and we do not see a natural way to accommodate control variate in their framework. Therefore, we do not see a feasible way to make a direct comparison.\n\n$\\textbf{Question 4}$: Additional empirical validations?\n\nResponse: We plan to add the following new experiments with the real-world data:\n1. Expansion of the target dataset. We will adjust the cutoff date for segmentation to an earlier point, expanding the target dataset. For example, moving the cutoff date from November 30, 2021, to a mid-2021 date, increases the sample size in the target dataset while still capturing significant changes in the pandemic landscape.\n2. With the COVID data, we will generate multiple permuted datasets by randomly shuffling the outcome variable Y while keeping the treatment variable X and covariates Z unchanged. This process ensures that the relationship between Y and X, given Z, is broken, simulating the null hypothesis for use to test the robustness of our method in Type-I error control.\n3. Different outcomes. In addition to readmission, we will evaluate different outcomes such as mortality. Specifically, we will analyze mortality within 30 and 90 days of hospital admission due to COVID-19.\n\n\n$\\textbf{References}$\n\nBerrett, Thomas B., et al. \"The conditional permutation test for independence while controlling for confounders.\" Journal of the Royal Statistical Society Series B: Statistical Methodology 82.1 (2020): 175-197.\n\nJavanmard, Adel, and Mohammad Mehrabi. \"Pearson chi-squared conditional randomization test.\" arXiv preprint arXiv:2111.00027 (2021).\n\nLi, Shuangning, and Molei Liu. \"Maxway CRT: improving the robustness of the model-X inference.\" Journal of the Royal Statistical Society Series B: Statistical Methodology 85.5 (2023): 1441-1470.\n\nThams, Nikolaj, et al. \"Statistical testing under distributional shifts.\" Journal of the Royal Statistical Society Series B: Statistical Methodology 85.3 (2023): 597-663.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TGX6yqSlna", "reviewer": "Reviewer_r237", "summary": "This paper proposes a new variation of CRT to be applied in the presence of covariate shifts. The paper presents a method and one extension for each, with higher power. Then, the authors present some needed theoretical results and finish with experiments.", "questions": "- How can we understand the role of the effective sample sizes (weight imbalance) in the effectiveness of your method?\n- Could you elaborate more on the surrogate variables? I am familiar with the literature on CI testing, but I haven't seen the presence of these variables in other works.", "limitations": "The authors comment on limitations.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper proposes a theoretically correct approach for testing CI under covariate shift;\n- The paper presents their algorithms and theoretical results;\n- The paper had convincing experiments;", "weaknesses": "- It would be interesting if the authors could estimate the full density ratio in their simulations (possibly in a high-d scenario) and then compare their results with the IS approach;\n- The paper does not conduct any experiment where Type-I error control is shown in a real dataset, e.g., like for example in [1,2].\n\nReferences\n\n[1] Pogodin, R., Schrab, A., Li, Y., Sutherland, D. J., & Gretton, A. (2024). Practical Kernel Tests of Conditional Independence. arXiv preprint arXiv:2402.13196.\n\n[2] Maia Polo, Felipe, Yuekai Sun, and Moulinath Banerjee. \"Conditional independence testing under misspecified inductive biases.\" Advances in Neural Information Processing Systems 36 (2023): 58577-58612.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We extend our sincere thanks for your reviewing work and insightful comments. Please see our responses to your comments and questions as below. \n\n$\\textbf{Weakness 1}$: It would be interesting if the authors could estimate the full density ratio in their simulations (possibly in a high-d scenario) and then compare their results with the IS approach.\n\nResponse: We have added additional experiments with the full density ratio estimated, as shown in Figure R2 of the supplementary PDF file (attached to the global rebuttal response). We chose a high-dimensional setting where the dimension of $Z$ is 50. We fit high-dimensional regression to estimate the joint density ratio $e(X, Z, V)$ and the conditional model $X \\sim Z$ with the unlabeled data.\nThe results show that when the sample size is larger than 800, all three methods have good Type-I error control. With a relatively small sample size for estimation, the IS method and the power enhancement version of our method retain good Type-I error rate control, but the vanilla csPCR method may have an inflated Type-I error rate (e.g., when $n_e = 400$, the Type-I error rate of csPCR is 0.108, while the IS method and csPCR(pe) retain 0.052). The statistical power follows a similar pattern as presented in the paper (csPCR(pe) > csPCR > IS) but is uniformly lower for all three methods because of the lower estimation accuracy (e.g., when $\\beta = 2$, the power of csPCR(pe) is 0.833, while in our original experiments it is 0.867).\n\n$\\textbf{Weakness 2}$: The paper does not conduct any experiment where Type-I error control is shown in a real dataset, e.g., like for example in [1,2].\n\nResponse: We propose the following steps to validate Type-I error control in the real dataset. (1) We will adjust the cutoff date for segmentation to an earlier point, expanding the target dataset. For example, moving the cutoff date from November 30, 2021, to a mid-2021 date, increases the sample size in the target dataset while still capturing significant changes in the pandemic landscape. With the expanded dataset, we will rerun our csPCR test and calculate the empirical Type-I error rate. This involves comparing the test results against the true outcomes in the expanded target dataset. (2) Additionally, we will generate multiple permuted datasets by randomly shuffling Y while keeping X and Z unchanged. For each permuted dataset, we will apply the csPCR test and record the results to calculate the empirical Type-I error rate.\n\n\n$\\textbf{Question 1}$: How can we understand the role of the effective sample sizes (weight imbalance) in the effectiveness of your method?\n\nResponse: We notice a series of work in measuring the effective sample size (ESS) of importance weight or sampling in the statistical computation literature, e.g., [Martino, et al, 2017] and others. Among them, one of the most common ways is to use the ratio $n_{eff}=(\\sum_{i=1}w_i)^2 / \\sum_{i=1}w_i^2$ to approximate the ESS. When the covariate shift between the source and target becomes stronger, the variance of the importance weight $w_i$ tends to be large and $n_{eff}$ will become smaller, which can result in lower power. Our power enhancement method based on control variate could potentially alleviate this issue with properly specified control functions. We plan to carry out additional simulation studies on the relationship between the power of csPCR and the effective sample size (affected by the degree of covariate shift) in the camera-ready version of the paper upon acceptance.\n\n$\\textbf{Question 2}$: Could you elaborate more on the surrogate variables? I am familiar with the literature on CI testing, but I haven't seen the presence of these variables in other works.\n\nResponse: A surrogate or silver standard label is a variable that is more feasible and accessible than $Y$ in data collection and can be viewed as a noisy measure of $Y$. In clinical trials, $Y$ is often a longer-term outcome; in such settings, surrogates are measures that can predict the effect of a treatment on the longer-term outcome $Y$. These surrogates can be biomarkers or clinical parameters measured relatively quickly, usually within a few weeks or months of starting treatment. For example, in clinical trials, tumor response rate is often used as a surrogate for overall survival, and blood pressure is commonly used as a surrogate for cardiovascular events such as heart attacks. Surrogate variables are also commonly used in environmental studies and economics.\n\nRegarding conditional independence testing, Li and Liu (2023) study how surrogate variables can improve the robustness of the Conditional Randomization Test (CRT). They propose a method called Maxway CRT, which leverages knowledge of the distribution of $Y|Z$ to enhance the robustness of the CRT. Surrogate variables are extremely helpful in learning the distribution of $Y|Z$ because there is usually much more data on surrogates than on the outcome variable. \n\nFinally, we want to emphasize that without the surrogate variable, even if there is a shift in the distribution of $X$, the original CRT on the source data remains valid for the target population, provided that the distribution of $Y | X, Z$ is assumed to be the same in both the source and target populations.\n\n$\\textbf{References}$\n\nMartino, Luca, Víctor Elvira, and Francisco Louzada. \"Effective sample size for importance sampling based on discrepancy measures.\" Signal Processing 131 (2017): 386-401.\n\nLi, Shuangning, and Molei Liu. \"Maxway CRT: improving the robustness of the model-X inference.\" Journal of the Royal Statistical Society Series B: Statistical Methodology 85.5 (2023): 1441-1470.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ADiSeCjIPA", "reviewer": "Reviewer_REEU", "summary": "The paper introduces a novel approach to addressing covariate shift in conditional independence tests. By leveraging importance weights and the control variates method,the paper proposes Covariate Shift Corrected Pearson Chi-squared Conditional Randomization (csPCR) test, which maintains asymptotic Type-I error control. In addition, a power enhancement is proposed, which reduces the variance in the test statistic and thus improve the power towards alternatives. The methodology is validated both theoretically and empirically, demonstrating superior performance in simulation studies and practical application to a COVID-19 treatment dataset, highlighting its efficacy in real-world scenarios.", "questions": "1. Page 5, Algorithm 1: Besides the variance issues considered in Algorithm 2, it is clear that the choice of test statistic in Eq (4) also affects the power. Can you comment on how to choose the test statistic? In the numerical experiment, the test statistic is specified to be the product of $Y$ and $X$, which makes sense as it measures the \"correlation\" between Y and X. However, $Z$ is not included in the test statistic, which seems to decrease its power. Have you tried any other test statistics? Do difference choices affect the power significantly? For instance, one alternative is to use $(Y - \\beta_y Z)(X - \\beta_x Z)$, where $\\beta_y and \\beta_x$ are regression coefficients for $Y\\sim Z$ and $X\\sim Z$ on an independent data set from source distribution.\n\n2. Page 4, Line 170-177: The paper argues that they used PCR as the baseline test because PCR is more powerful than the vanilla CRT. But on the other hand, CRT is also better than PCR in the sense that it has finite-sample type I error control, rather than asymptotic type I error control. It's worth mentioning this point as well. While I notice that the simulations in the paper shows satisfactory type I error control, the type I errors may get inflated if, for example, the data have heavy tails.\n\n3. Page 9, Section 9: Relating to the weaknesses, have you run any tests to check the iid assumption on the Covid-19 data set?", "limitations": "The papers discuss about the limitations of their method under model misspecifications in Section 4, pointing out that the powerful enhancement can disappear when there is a full nonlinear component in the model. The authors have addressed the limitations adequately.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "Originality\nThe paper presents original contributions by introducing a new approach to conditional independence testing under covariate shift. This method combines the idea of conditional randomization test, importance weighting, the control variates technique. \n\nQuality\nThe work is of high quality, demonstrated through rigorous statistical theories on the validity and power, and comprehensive empirical validations.\n\nClarity\nThe paper is clearly written and well-organized. Each section has a clear message and readers can quickly grasp the main idea. \n\nSignificance\nI consider this work as a significant contribution to the field of conditional independence testing and covariate shift. The method is easy-to-implement, and is more powerful than existing methods as shown in numerical experiments.", "weaknesses": "The paper does not leave serious technical weaknesses, but I am concerned about the limited applicability of this method in real-world problems. Based on my understanding, one would test for conditional independence directly on the target distribution if the outcomes in the target population were easy to collect. So csPCR is useful only when the outcomes are costly to collect. This questions several application examples mentioned in the paper. First, in the college admission example (from Line 37-53), why don't the economists just collect the college admission results from the target population and then conduct the vanilla PCR? Typically, the target outcomes have to be collected for followup analysis after the conditional independence test. Second, Section 5 introduces a real-world application of this method on Covid-19 pandemic data, where the source and the target data set are segmented based on time. Given such a sequential data collecting process, it is unclear whether the iid assumption still holds in the source and target distribution. See Questions for more details.\n\nDespite of the concerns above, I do believe that this method is useful for certain genetic and biomedical problems, where the outcomes are difficult to collect, but other covariates (such as SNPs) are relatively easy. Therefore, I suggest the authors clarify the application examples a bit by discussing more reasonable applications with some references.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We extend our sincere thanks for your reviewing work and insightful comments. Please see our responses to your comments and questions as below. \n\n$\\textbf{Weaknesses 1}$: In the college admission example, why don't the economists just collect the college admission results from the target population?\n\nResponse: In the college admission example, large-scale data on SAT scores is more easily accessible through high schools, exam preparation schools, or the College Board (the organization that administers the SAT exam). Conversely, college admission results are more difficult and costly to obtain, as they require individual-level surveys. Furthermore, students might be reluctant to disclose whether they were rejected in such surveys.\n\nFollowing your suggestion, we also plan to add another example about clinical study. Suppose we are interested in testing the treatment effect of some drug $X$ on some long-term outcome $Y$ such as five-year survival. In this case, researchers collect $V$ as some early endpoint surrogate that can be measured within a short term, usually a few weeks or months post treatment, e.g., tumor response rate in cancer treatment; see [e.g., VanderWeele 2013].\n\nVanderWeele, Tyler J. \"Surrogate measures and consistent surrogates.\" Biometrics 69.3 (2013): 561-565.\n\n\n$\\textbf{Weaknesses 2}$: Given such a sequential data collecting process of the COVID data, it is unclear whether the iid assumption still holds.\n& $\\textbf{Question 3}$: Have you run any tests to check the iid assumption?\n\nResponse: The period spanning from January 2020 to November 30, 2021, in our source dataset captures the early waves of the COVID-19 pandemic, including the original strain and early variants such as Alpha and Beta. After November 30, 2021, our target dataset includes admissions during subsequent waves dominated by variants like Delta and Omicron. The segmentation of the dataset at November 30, 2021, is intentional to account for significant shifts in virus characteristics, public health policies, and medical treatments, while within source or target, the distribution of these covariates is assumed to be consistent, reflecting the stabilization of public health responses and medical treatments specific to early/later variants and the increased coverage of vaccinations. In addition, we plan to run rigorous tests (e.g., Kolmogorov-Smirnov test) to examine this i.i.d assumption within the source and target upon the acceptance of the paper.\n\n\n$\\textbf{Question 1}$: Besides the variance issues considered in Algorithm 2, it is clear that the choice of test statistic in Eq (4) also affects the power. Can you comment on how to choose the test statistic? \n\nResponse: The main principle of choosing the test statistic is to characterize the conditional dependency between $X$ and $Y$ under the alternative hypothesis. We agree that $YX$ may not be the optimal choice for the test statistic and that using $(Y-\\hat{E}[Y\\mid Z])(X - E[X\\mid Z])$ could remove the confounding effect of $Z$. Inspired by this, we used $Y(X - E[X|Z])$ as the test statistic to conduct additional simulations. The results are presented in supplementary Figure R3. We find that $Y(X - E[X|Z])$ and $YX$ produce nearly the same power for both csPCR and csPCR(pe).\nWe did not use $(Y-\\hat{E}[Y\\mid Z])(X - E[X\\mid Z])$ because estimating $\\hat{E}[Y\\mid Z]$ requires sample splitting to estimate $\\hat{E}[Y\\mid Z]$ with some hold-out sample (otherwise, the theoretical Type-I error control of the PCR test cannot be guaranteed).\nAn alternative strategy is to get an estimate of $P(Y\\mid X,Z)$ on some hold-out training data as mentioned above, then naturally use $\\log{P(Y\\mid X,Z)}-\\log{P(Y\\mid Z)}$ as the test statistic [Tansey et al., 2022]. This could increase the ability to capture nonlinear dependence, but the sample splitting will generally cause a loss of power. The surrogate or auxiliary $V$ in our case could potentially help this hold-out training procedure and alleviate the power loss issue. We plan to add simulations on this upon the acceptance of our paper.\nTansey, Wesley, et al. \"The holdout randomization test for feature selection in black box models.\" Journal of Computational and Graphical Statistics 31.1 (2022): 151-162. \n\n\n$\\textbf{Question 2}$: Comparison of PCR with vanilla CRT?\n\nResponse:  You are perfectly correct that PCR does not achieve exact Type-I error control. Meanwhile, we would like to point out that unlike other asymptotic inference approaches, PCR and csPCR are less prone to issues with heavy-tailed data, as they rely on the binary indicator variable of each subject to construct the chi-squared statistics. Nevertheless, small sample sizes could still cause Type-I error inflation in csPCR. We note that the PCR paper [Javanmard and Mehrabi, 2021] introduced a finite-sample Type-I error control version of PCR, which is achieved using concentration inequalities. Their strategy could be naturally incorporated with the current csPCR to achieve better (exact) Type-I control. We will add a discussion on this extension.\nAdditionally, we agree that our statement regarding PCR being more powerful than CRT was overly broad. A more precise statement would be that PCR can handle some more challenging alternatives not addressed well by the vanilla CRT, as demonstrated in [Javanmard and Mehrabi, 2021]. In our numerical studies, we used the PCR construction on all benchmarks for a fairer comparison, and we found that our method outperforms the IS method with the vanilla CRT as well.\n\n$\\textbf{References}$\n\nJavanmard, Adel, and Mohammad Mehrabi. \"Pearson chi-squared conditional randomization test.\" arXiv preprint arXiv:2111.00027 (2021).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely thank all reviewers and chairs for their feedback. We have addressed each of the reviewers' points individually and have included a PDF with additional experiments. The first of the experiments demonstrated that our proposed csPCR method has more stable Type I error rate control and higher power without the need for tuning the resample size compared to the IS method. The second experiment shows that when estimating the full density ratio, the vanilla csPCR may suffer from slight Type I error inflation, while other results show similar patterns to our original experiments. The third experiment shows that with the different choice of test statistic $Y(X - E[X|Z])$, both the Type I error rate and power do not change significantly.

Meta-review: This paper addresses the problem of conditional independence testing under covariate shifts. The goal is to determine whether $p(y|x,z) = p(y|z)$ (i.e., whether X and Y are conditionally independent given Z) on the target data, by making use of another source dataset. The main challenge arises because there can be covariate shifts between the source and the target. The paper introduces importance weights into the Pearson Chi-Squared Conditional Randomization (PCR) test and makes use of the control variates method to address the potential covariate shifts. It is theoretically proven that the new test is consistent (asymptotically has maximum test power), and asymptotically has a well-controlled false positive rate.

**Exclude an official review from Reviewer G2Ah**

Reviewer G2Ah misunderstood the problem setting and could not go through the submission to provide a full review. In particular, in the paper, it is assumed that $P(Y|X, Z, V)$ remains the same from source to target data. The misunderstanding was that this would mean $P(Y|X,Z)$ is also the same and so the test can be conducted on the source data to draw conclusions on the target data. This statement is NOT true due to the presence of $V$. This misunderstanding was resolved during the reviewer discussion period with the AC. This official review is negative because of a misunderstanding. The AC decided to exclude this review. 

The authors are strongly advised to emphasize this point more. In particular, it may be useful to include the derivation to arrive at $P(Y|X,Z)=\int P(Y|X,Z,V)P(V|X,Z)\thinspace dV$ in the paper.


**Overall review**

The remaining three reviewers noted the originality of the method in this specific setting (REEU), and strong theoretical contribution (r237) with convincing experiments (MwvS). A common point raised is that the method depends on the accuracy of the estimated density ratio (for importance weighting) (MwvS, r237) which may limit applicability of the method in real problems (REEU). The authors sufficiently addressed this concern by providing more empirical results in the rebuttal. 

Recommendation: accept.

The authors are strongly suggested to add more results to questions raised by the reviewers e.g.,generate multiple permuted datasets on the Covid data experiment, evaluate different outcomes such as mortality, etc.
Decision: Accept (poster)