Covariate Shift Corrected Conditional Randomization Test

Authors: bowen  xu
yiwen  huang
chuan  hong
shuangning  li
molei  liu
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 04:50:06

Abstract: Conditional independence tests are crucial across various disciplines in determining the independence of an outcome variable $Y$ from a treatment variable $X$, conditioning on a set of confounders $Z$. The Conditional Randomization Test (CRT) offers a powerful framework for such testing by assuming known distributions of $X \mid Z$; it controls the Type-I error exactly, allowing for the use of flexible, black-box test statistics. In practice, testing for conditional independence often involves using data from a source population to draw conclusions about a target population. This can be challenging due to covariate shift---differences in the distribution of $X$, $Z$, and surrogate variables, which can affect the conditional distribution of $Y \mid X, Z$---rendering traditional CRT approaches invalid. To address this issue, we propose a novel Covariate Shift Corrected Pearson Chi-squared Conditional Randomization (csPCR) test. This test adapts to covariate shifts by integrating importance weights and employing the control variates method to reduce variance in the test statistics and thus enhance power. Theoretically, we establish that the csPCR test controls the Type-I error asymptotically. Empirically, through simulation studies, we demonstrate that our method not only maintains control over Type-I errors but also exhibits superior power, confirming its efficacy and practical utility in real-world scenarios where covariate shifts are prevalent. Finally, we apply our methodology to a real-world dataset to assess the impact of a COVID-19 treatment on the 90-day mortality rate among patients.

Main paper: 
 related work: 
 Our work builds upon the model-X framework and the conditional randomization test proposed by Candes et al. [2] . The particular method we develop is based on a variant of the vanilla CRT, the Pearson Conditional Randomization (PCR) test [5] . Recent advances in the CRT include improving computation time [7, 10] , studying robustness [6, 11] , and examining statistical power [19] . The focus of this paper, different from the above, is on how to build a valid CRT procedure when there is covariate shift. The paper is also complementary to the above literature: for example, we hope that future work can conduct theoretical power analysis for our procedure or develop a double robust version of the procedure just like in [6] . Finally, we note that surrogate variables play a crucial role in this paper: because the distribution of the surrogate variables is different in the source and the target population, naively testing the conditional independence hypothesis in the source population can yield invalid conclusions for the target population. A surrogate or silver standard label is a variable that is more feasible and accessible than Y in data collection and can be viewed as a noisy measure of Y . For example, tumor response rate is often used as an early endpoint surrogate for the long-term survival outcome [3] , and blood pressure is commonly used as a surrogate for heart attacks. Surrogate variables are also commonly used in environmental studies and economics. Surrogate variables also play an important role in the paper by [6] , albeit in a different way, where the surrogate variables are used to learn the distribution of Y | X, Z and to further improve the robustness of the CRT procedure.Statistical learning and inference under covariate shift has been extensively studied over the past years.As a seminal work in addressing covariate shift bias, [4] proposed a density ratio weighting approach using kernel mean matching to characterize the adjusting weights. Their key idea of importance (re)weighting is intrinsically connected with early work in broader contexts like importance sampling [15, e. g.] and semiparametric inference [13, e.g.] . [8] extended this idea to a doubly robust framework accommodating surrogate variables like V and being more robust to the misspecification or poor quality of the density ratio models. [18] handled a more challenging scenario with severe shift and poor overlap between the source and target populations. Among this track of literature, [17] is the most closely related to our work as they also considered conditional independence testing under distributional shifts and proposed a general testing procedure base on importance sampling (IS) allowing for the use of CRT. Different from us, their work does not accommodate the covariate shifts of some surrogate or auxiliary V . Moreover, as will be shown in our numerical studies, their general IS testing strategy can encounter the loss of effective sample sizes and be less powerful than ours.2 Problem Setup Consider a scenario involving two distinct populations: the source population S and the target population T . We collect data from the source population with the goal of making inferences about the target population. The source data contains n independent and identically distributed samples ofpY i , X i , Z i¨, V i¨q for i " 1, . . . , n. Let y " pY 1 , Y 2 , . . . , Y n q J P R n , x " pX 1 , X 2 , . . . , X n q J P R n ,Z " pZ 1¨, Z 2¨, . . . , Z n¨q J P R nˆp , and V " pV 1¨, V 2¨, . . . , V n¨q J P R nˆd . We are interested in testing the following conditional independence hypothesis in the target population:EQUATIONWe assume that the conditional distribution of Y | X, Z, V is the same in both populations; however, the distribution of pX, Z, V q varies between S and T . More precisely, the joint distribution of Y, X, Z, V can be described as follows:EQUATIONThis situation is referred to as the covariate shift scenario because the distribution of the covariates X, Z, and V in the source population S does not match that in the target population T .Let's understand the above assumption and its implications through the two examples above. In the college admissions example, it is plausible to assume that the rate of college admissions remains consistent across the two populations when conditioned on the SAT score, family income, and other confounding variables. However, the joint distribution of X, V and Z can differ: in the source population, if wealthy parents frequently invest in SAT preparation, boosting admission rates, this relationship may not hold in a target population where such preparation is uncommon. In such cases, it is thus possible that X K KY | Z in the source population but X K K Y | Z in the target population (see Figure 2 for such an example). In the health outcomes example, it is again plausible that the conditional distribution of long-term health outcomes given the treatment variable, confounding variables, and surrogates remains the same across the two populations. However, the assignment of the treatment may depend differently on the surrogate variables across the two populations. Therefore, it's possible that X K K Y | Z in one population, but not in the other.In both examples, we can see that the result of naively applying a valid conditional independence test on the source population cannot guarantee a valid conclusion for testing H 0 in the target population.Therefore, we need to develop new tools for addressing covariate shifts in conditional independence tests. 
model-x framework: 
 In this paper, we operate within the model-X framework, as described by Candes et al. [2] , which assumes that the joint distributions of covariates X, V, Z are perfectly known in both the source and target populations. This framework is particularly suited for scenarios where: (1) there is substantial prior domain knowledge about the covariates X, V, and Z, or (2) there is a significant amount of unsupervised data for these covariates in both populations, in addition to n labeled observations in the source population, characterizing a semi-supervised setting.An example of the first scenario can be seen in genetics, where researchers have well-established models for the joint distributions of single nucleotide polymorphisms (SNPs). For the second scenario, consider our earlier example involving health outcomes. Here, the outcome variable Y represents a long-term health outcome that is more costly or sensitive to measure compared to the shorter-term variables X, V, and Z. In such cases, the variables X, V, and Z are typically easier and less costly to collect, frequently resulting in a semi-supervised setting in these health-related studies.3 Method: Covariate Shift Corrected PCR Test In Section 2.1, we discussed how naively applying conditional independence tests to the source data cannot guarantee valid conclusions for the target population. To address this issue, we must incorporate information about the differences between the two populations into our testing procedure.In particular, we will make use of the density ratio defined as:EQUATION)This ratio measures the relative likelihood of observing each combination of variables pX, Z, V q in the target population compared to the source population. By reweighting the data points in the source population using this density ratio, we effectively transform the source distribution to match the distribution of the target population, thereby addressing the covariate shift problem.More specifically, we build our method upon the recently proposed Pearson Chi-Squared Conditional Randomization (PCR) test [5] . Compared to the vanilla CRT, the PCR test is designed to be more powerful across a broader range of alternative p-value distributions. At a high level, the PCR test assigns a label to each data point following a counterfeit sampling step and a subsequent score computation step. Under the null hypothesis that X K K Y | Z, the distribution of these labels should be uniform across all possible labels. The PCR test then rejects the null hypothesis if the empirical distribution of the labels deviates significantly from uniformity, as determined by aPearson's chi-squared test.Under distributional shift, if the data points were sampled from the target population, then the distribution of the labels would be uniform. However, since the data points are actually sampled from the source population, they must be reweighted using the density ratio. More specifically, in the final step of the PCR test, where the Pearson's chi-squared test is applied, we consider not the count of data points for each label, but the sum of the density ratios of the data points for each label instead.Under the null hypothesis, each sum should approximate n{L, where L is the total number of labels.Consequently, we modify the Pearson's chi-squared test to determine whether these weighted sums deviate significantly from n{L.Based on the above intuition, we propose the Covariate Shift Corrected PCR (csPCR) Test, as outlined in Algorithm 1.In Algorithm 1, lines 1-7 correspond to those in the original PCR test. These lines initiate the test by generating counterfeit samples Xpmq j . Assuming the source and target populations were identical, under the null hypothesis, the random variables pX j , Y j , Z j q, p Xp1q j , Y j , Z j q, . . . , p XpMq j , Y j , Z j q would be exchangeable. Consequently, the rank R j would be uniformly distributed over t1, . . . , M `1u in the absence of ties, leading to a uniform distribution of the labels as well.Lines 8-10 in Algorithm 1 address the covariate shift by incorporating density ratios as importance weights into W j . Due to this redefinition of W ℓ , the null distribution of the final test statisticU n,L is also different. Therefore, we also adjust the rejection threshold from the quantile of a chisquared distribution, as in the original PCR test, to the quantile of the weighted sum of chi-squared distributions.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: We examine the performance of the algorithms under model misspecification in Section 4.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.• The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.• The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.• The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.• The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.• If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.• While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
power enhancement: 
 To effectively address covariate shift, incorporating density ratios as importance weights into the PCR test is essential. However, when these ratios become large, they can increase the variance of the statistics W l . This elevated variance can diminish the test's power. Therefore, developing methods to reduce this variance is crucial for maintaining the power of the test.Algorithm 1 Covariate Shift Corrected PCR (csPCR) Test.Input: Data D T " py, x, Z, Vq, the density ratio e, the test statistics T , integers K, L ě 1, and the significance level α. 1: Take M " KL ´1. 2: for each data point j " 1 to n do 3: Draw M i.i.d samples r X p1q j , . . . , r X pM q j from P T pX | Zq.Use T to score the initial data point pX j , Y j , Z j q and its M counterfeits p r X p1:M q j , Y j , Z j qEQUATIONLet R j denote the rank of T j among tT j , r T p1q j , . . . , r T pM q j u, with ties broken randomly.Partition t1, . . . , M `1u " S 1 Ť . . . Ť S L with S ℓ :" tpℓ ´1qK `1, . . . , ℓKu. Assign label ℓ j P t1, 2, . . . , Lu to sample j if R j P S ℓj . 7: end for 8: Let w j " epX j , Z j , V j q for each j P t1, 2, . . . , nu. 9: for each label ℓ P t1, 2, . . . , Lu: do 10:Let W ℓ be the sum of ℓ-labeled importance weights: W ℓ " n ř j"1 w j ¨1tℓ j " ℓu.Let D ℓ be the sum of ℓ-labeled squared importance weights: D ℓ " n ř j"1 w 2 j ¨1tℓ j " ℓu.12: end for13: Let Ωn " L n diagpD 1 , D 2 , ¨¨¨, D L q ´1 L ¨1LˆL . 14: Calculate the test statistic U n,L as follows U n,L " L n L ř ℓ"1 `Wℓ ´n L ˘2 .Output: Reject the null hypothesis if U n,L ě θ Ωn,α ; otherwise, accept the null hypothesis. Here, θ Ωn,α is the 1 ´α quantile of the distribution χ 2, where A " χ 2 Ω denotes that A " x T x for x " N p0, Ωq.To this end, we introduce a control variate function a, allowing apX, Z, V q to serve as a control variate in reducing variance in W l [14] . Specifically, for a chosen γ ℓ , we defineEQUATIONWe can then use Ă W ℓ instead of W ℓ in our algorithm.We note that for any arbitrary choice of the function a and the parameter γ ℓ , the expectation of Ă W ℓ would be the same as that of W ℓ :EQUATIONTherefore, even if we make a sub-optimal choice of the function a and the parameter γ ℓ in practice, the resulting test (under certain assumptions) will still remain asymptotically valid (see Section 3.3 for more details).However, for effective variance reduction, it is preferable to have the control covariates apX, Z, V q well-correlated with the outcome (See Section 4 for practical discussions on choices of the function a). This is quite feasible, especially since the surrogate variable V is likely to be predictive of Y .Algorithm 2 Covariate Shift Corrected PCR Test with Power Enhancement.Input: Data D T " py, x, Z, Vq, the density ratio e, the test statistics T , the control variate function a, integers K, L ě 1, and the significance level α. 1: for each data point j " 1 to n do 2:Compute the labels ℓ j as in Algorithm 1. 3: end for 4: Let w j " epX j , Z j , V j q for each j P t1, 2, . . . , nu. 5: for each label ℓ P t1, 2, . . . , Lu: do6:Compute γℓ , the regression coefficient obtained by a weighted linear regression of the indicator function ␣ 1tℓ j " ℓu ( n j"1 on the control variate ␣ apX j , Z j , V j q ( n j"1 with weights ␣ w j ( n j"1 .Compute the augmented version of W ℓ asĂ W ℓ " n ÿ j"1 w j ¨r1tℓ j " ℓu ´γ ℓ apX j , Z j , V j qs `nγ ℓ E T " apX, Z, V q ‰ . 8: end for 9: Let W " ´wj ¨r1tℓ j " ℓu ´γ ℓ apX j , Z j , V j qs `γ ℓ E T " apX, Z, V q ‰ ¯ℓ,j for 1 ď ℓ ď L, 1 ď j ď n. 10: Calculate the sample covariance matrix r Ω n " L n pW ´1 L ¨1Lˆn qpW ´1 L ¨1Lˆn q T . 11: Calculate the test statistic U n,L as follows r U n,L " L n L ř ℓ"1 ´Ă W ℓ ´n L ¯2 . Output: Reject the null hypothesis if r U n,L ě θ rΩn,α ; otherwise, accept the null hypothesis. Here, θ r Ωn,α is the 1 ´α quantile of the distribution χ 2 r Ωn , where A " χ 2 Ω denotes that A " x T x for x " N p0, Ωq.We would also like to discuss the choice of γ ℓ . According to the control covariate literature, with a fixed function a, the optimal choice of γ ℓ that minimizes variance is given by:EQUATION)This coefficient is also the same as that obtained from a linear regression [14] . Thus, when implementing the algorithm, we take γ ℓ to be the regression coefficient obtained by running a weighted linear regression of the indicator function t1 ␣ ℓ j " ℓ ( u n j"1 on the control variate ␣ apX j , Z j , V j q ( n j"1 with weights ␣ w j ( n j"1 .We have outlined the new csPCR test, including this power enhancement step, in Algorithm 2.In this section, we establish that the proposed tests control the type-I error asymptotically. Furthermore, we show that the power enhancement step effectively reduces the variance of the statistics W ℓ , which can typically improve the power.Assumption 1 (Fourth moment). The fourth moment of the density ratio epX, Z, V q is finite:E S " epX, Z, V q 4 ‰ ă 8. Furthermore, the fourth moment of product of the density ratio and the control variate function is also finite: E S " epX, Z, V q 4 apX, Z, V q 4 ‰ ă 8.Theorem 1 (Valid Tests). Under Assumption 1, assume that the null hypothesis of X K K Y | Z holds in the target population, thenEQUATION)EQUATIONTheorem 2 (Variance Reduction). Let W l be the statistics computed in line 10 in Algorithm 1, and Ă W l be the statistics computed in line 7 in Algorithm 2. Under Assumption 1,EQUATIONQuestion: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes] Justification: Replication code for our simulation studies is submitted as supplementary material. It will also be made publicly available on GitHub once our paper is accepted.The COVID data set used for the real example in our paper is not publicly available due to privacy constraints.Guidelines:• The answer NA means that paper does not include experiments requiring code.• Please see the NeurIPS code and data submission guidelines (https://nips.cc/ public/guides/CodeSubmissionPolicy) for more details.• While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).• The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https://nips.cc/public/guides/CodeSubmissionPolicy) for more details.• The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.• The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.• At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).• Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
numerical simulation: 
 In this section, we present simulation studies to assess the performance of our proposed csPCR method and its power enhancement version denoted csPCR(pe), and compare them to a benchmark method. The benchmark method adopted is an importance-resampling based method [17] , denoted as the IS method. For fair comparison, we used the same PCR statistic as our method for the testing with IS. We use a significance level of α " 0.05.We consider a semi-supervised setting where we have a large volume of unlabeled data of pX j , Z j , V j q from both the source and target populations. In addition, we have a small number of labeled data of pY j , X j , Z j , V j q from the source population.We separate confounding variables Z into two sets: Z " pZ r , Z null q, where Z r is the relevant set and Z null is the null set. The relevant confounding variables Z r are generated as i.i.d. multivariate normal, with mean 0 for the source population and 1 for the target population to simulate the distributional shift in Z, where Z r P R p and we set p " 5. Null confounding variables Z null are generated independently with no correlation to other variables, modeled as N p0.1, I q q with q " 50 for sparse high-dimensional settings in both populations.The treatment variable X and the surrogate variable V are conditionally generated based on Z.Specifically, X is modeled identically across both the source and target populations asN pu J Z r , 1q,where u is a predefined parameter vector that remains the same for both populations.For V , it is modeled differently in the two populations, represented as N pv J S{T Z r `p1 ´θqa S{T X θa S{T sinpXq, 1q. Here, v S and v T are predefined parameter vectors for the source and target populations, respectively. The parameter a varies between populations (a S for the source and a T for the target), controlling the effect of X on V , modeling the indirect effect. The factor θ modulates the nonlinear component of this relationship.The outcome variable Y is generated for both populations using the same conditional model over pX, Z, V q: Y |pX, Z, V q S{T " N ppv T Z r q 2 `βV `γX, 1q, where β and γ control the effects of V (indirect) and X (direct) on Y , respectively.We generate 1000 unlabeled source and target samples to estimate the density ratio and generate 500 labeled source samples for testing. Moreover, in the simulation, we assume we have full knowledge of the joint distribution of pX, Zq and estimate V |X, Z using an Elastic net regression model with 5-fold cross-validation [20] . For the test statistic T in the algorithm, we choose a simple function T p X, Z, V, Y q " Y ¨X. For each parameter iteration, we conduct 1000 Monte Carlo simulations to estimate the Type-I error and power. We estimate the covariance matrix of the sequence of W i 's using the Monte Carlo method and use the momentchi2 package [1] for calculating the p-value.Additionally, we empirically choose the best hyperparameter L " 3 for all our experiments through additional experiments shown in Appendix B.2.In Figure 3 , we choose a S " 1 and a T " 0 to compare the Type-I error control of our methods with the benchmark. The left panel shows the Type-I error rate as the sample size of the data used to estimate the density ratio, n e , varies from small to large. There appears to be a slight Type-I error inflation for all three methods when the sample size n e is small, but the Type-I error quickly converges to the ideal level of 0.05 as n e grows larger. Moreover, our methods show more stable Type-I error control than the benchmark method when the estimation sample size is low. The right panel shows that when the density ratio is well approximated, all three methods attain good Type-I error control regardless of the change in β, i.e., the strength of the indirect effect, but the csPCR and csPCR(pe) methods have more stable control. To evaluate the statistical power of our csPCR test, we choose a S " 0 and a T " 2, so that the null hypothesis holds true in the target population but not in the source population.As Figure 4a shows, both the csPCR and the csPCR(pe) methods have uniformly higher power than the benchmark method as we vary the indirect effect size β. For example, when β " 1.4, the benchmark IS method has a power of 0.33, the csPCR method has a power of 0.44, and the csPCR(pe) method can attain a power of 0.8.When we fix the indirect effect β " 2 and vary the direct effect of X (γ), as shown in Figure 4b , our methods still exceed the benchmark, and the power enhancement significantly improves the original version of the test. For example, when γ " 1, the benchmark IS method has a power of 0.4, the csPCR method has a power of 0.62, and the csPCR(pe) method can attain a power of 0.86. We also test how adding a nonlinear component to the indirect effect affects the power when we assume a linear model of V | Z, X in the estimation stage. This can be helpful in assessing the performance of our methods under model misspecification. As Figure 4c indicates, as the nonlinear effect increases, the power of all three methods decreases, though our methods still significantly exceed the benchmark. Interestingly, we observe that as θ Ñ 1, i.e., there is a full nonlinear component without a linear component, the advantage of the power-enhanced version over the original csPCR test disappears. This occurs because when the V | X, Z model is misspecified and the density ratio estimation is inaccurate, the variance reduction in the control variates step reduces variance in the "wrong" direction, and thus does not improve the power of the original method.We notice a series of work in measuring the effective sample size (ESS) of the density ratio reweighting approaches [9] . Among them, one of the most common measure is n eff " př n i"1 w i q 2 { ř n i"1 w 2 i .When the covariate shift between the source and target becomes stronger, the variance of the importance weight w i tends to be large and n eff will become smaller, which could result in lower power. We carry out simulation studies on the relationship between the power of csPCR and the ESS determined by the degree of covariate shift as discussed in Appendix B.3. 
real-world application: 
 The COVID- For the analysis, we divide 50% of the source data, comprising 565 individuals, alongside the entirety of the target data, to estimate the density ratio. Density ratios of X, Z are estimated using probabilistic classification method [12] , while the density ratio of V |X, Z is determined through Elastic Net regression. For all three methods, the test statistic T is chosen to be T p X, Z, V, Y q " Y ¨X. As indicated in Table 1 , both csPCR and csPCR(pe) give statistically significant results, whereas the IS method does not. The statistically significant results are consistent with biomedical literature.For example, through systematic review and meta-analysis, [21] reported that Bamlanivimab is effective in reducing the mortality rates of COVID patients. In a cohort study, [16] also found similar effectiveness for Nirmatrelvir-ritonavir.These results align with our findings from the simulation study and demonstrate that our method has increased power compared with the benchmark IS method.Throughout this section, we write Spx j , z j , v j q " s j as the label assigned to sample j in Algorithms 1 and 2, instead of using ℓ j . This notation helps avoid confusion between different label choices.Proposition 1. Assume that the conditional independence X K K Y | Z holds on the target population T . Let epx j , z j , v j q denote the density ratio. For any integer ℓ P r1, Ls, the following holds:E S repx j , z j , v j q ¨1tS T px j , z j , v j q " ℓus " 1 L .Proof of Proposition 1. For simplicity, denote w j " epX j , Z j , V j q and s j " S T pX j , Z j , V j q.E S repX j , Z j , V j q ¨1tS T pX j , Z j , V j q " ℓusEQUATIONThe last equation follows from results in the non-covariate-shift scenario, e.g., from [5] .Let pW ℓ q ℓ"1,...,L be the sum of weights and Ωn be the sample covariance matrix in Algorithm 1. By Proposition 1, we have that EpW ℓ q " n ¨Erw j ¨1tℓ j " ℓus " n L .Note that the W ℓ 's are sums of i.i.d. random variables, and thus by the Central Limit Theorem, asn Ñ 8, A n " c L n ˆW1 ´n L , W 2 ´n L , . . . , W L ´n L ˙d Ñ N L p0, Ωq,where for any ℓ, ℓ ˚P t1, . . . , LuΩ ℓ,ℓ ˚" LCovpw 1 1 ts 1 " ℓu , w 1 1 ␣ s 1 " ℓ ˚(q " LE S " w 2 1 ¨1ts 1 " ℓu1ts 1 " ℓ ˚uı ´1 L " LE S " w 2 1 ¨1ts 1 " ℓu ı 1 ␣ ℓ " ℓ ˚( ´1 L .Therefore,U n,L " A T n A n d Ý Ñ χ 2 Ω .Next, we will focus on the variance estimation part. We will show that Ωn p Ñ Ω as n Ñ 8. For any ℓ, ℓ ˚P t1, . . . , Lu,Ωn,ℓ,ℓ ˚" 1 ␣ ℓ " ℓ ˚( L n D l ´1 L " 1 ␣ ℓ " ℓ ˚( L n n ÿ j"1 w 2 j ¨1tℓ j " ℓu ´1 L p Ñ 1 ␣ ℓ " ℓ ˚( LE S " w 2 1 ¨1ts 1 " ℓu ı ´1 L " Ω ℓ,ℓ ˚.Up til now, we have thatU n,L " A T n A n d Ý Ñ χ 2 Ω , and θ Ωn,α p Ñ θ Ω,α .Therefore, PpAlgorithm 1 rejectsq " PpU n,L ě θ Ωn,α q Ñ Ppχ 2 Ω ě θ Ω,α q " α.Let p Ă W ℓ q ℓ"1,...,L and r Ω n be the sum of weights and the sample covariance matrix in Algorithm 2.Let γℓ be the estimated coefficient in Algorithm 2.Recall that in (7) , we have identified the optimal choice of γ ℓ . We will start by working with this optimal choice and show that the γℓ is close to it. Define| W ℓ " n ÿ j"1w j `1tℓ j " ℓu ´γℓ apX j , Z j , V j q ˘`nγ ℓ E T rapX, Z, V qs, K ℓ,j pγq " w j `1tℓ j " ℓu ´γapX j , Z j , V j q ˘`γE T rapX, Z, V qs, andH j " w j apX j , Z j , V j q ´ET " apX, Z, V q ‰ . Therefore, we have | W ℓ " ř j K ℓ,j pγ ℓ q and Ă W ℓ " ř j K ℓ,j pγ ℓ q " | W ℓ ´pγ ℓ ´γℓ q ř j H j .Note that by (6), EpH j q " 0. By Proposition 1 and (6), we have Ep | W ℓ q " n L . Furthermore, because | W ℓ is a sum of i.i.d. random variables, we have that as n Ñ 8,EQUATIONwhere for each ℓ, ℓ ˚P t1, . . . , Lu,q Ω ℓ,ℓ ˚" LCov ␣ K ℓ,j pγ ℓ q, K ℓ ˚,j pγ ℓ ˚q( . Therefore, q U n,L " q A T n q A n d Ý Ñ χ 2 q Ω .Next, we will show that the actual statistic r U n,L is close to q U n,L , and that the estimated variance matrix is also close to q Ω. We start with noting that the estimator γℓ from linear regression is close to the optimal choice γ ℓ defined in (7) : by the Central Limit Theorem, γℓ " γ ℓ `Op p1{ ? nq. And thusĂ W ℓ " n ÿ j"1 w j `1tℓ j " ℓu ´γ ℓ apX j , Z j , V j q ˘`nγ ℓ E T rapX, Z, V qs " n ÿ j"1 w j 1tℓ j " ℓu ´γ ℓ ¨n ÿ j"1 w j apX j , Z j , V j q ´nE T rapX, Z, V qs ‚ " n ÿ j"1 w j 1tℓ j " ℓu ´γℓ ¨n ÿ j"1 w j apX j , Z j , V j q ´nE T rapX, Z, V qs ‚`O p p1q " | W ℓ `Op p1q.The second-to-last line is because γℓ " γ ℓ `Op p1{ ? nq and the terms inside the parenthesis, ř j H j , is a sum of n independent mean-zero random variables.Therefore, together with (12) , by Slusky's Theorem, we have thatr A n " c L n ˆĂ W 1 ´n L , Ă W 2 ´n L , . . . , Ă W L ´n L ˙d Ñ N L p0, q Ωq,and thus,r U n,L " r A T n r A n d Ý Ñ χ 2 q Ω .We will work on sample covariance matrix now. Recall that the sample covariance matrix r Ω n "L n pW ´1 L ¨1Lˆn qpW ´1 L ¨1Lˆn q T , where W ℓ,j " w j ¨r1tℓ j " ℓu ´γ ℓ apX j , Z j , V j qs γℓ E T " apX, Z, V q ‰ " K ℓ,j pγ ℓ q. Let's start with WW T . For any ℓ, ℓ ˚P t1, . . . , Lu,pWW T q ℓ,ℓ ˚" ÿ j K ℓ,j pγ ℓ qK ℓ ˚,j pγ ℓ ˚q " ÿ j`Kℓ,j pγ ℓ q ´pγ ℓ ´γℓ qH j ˘`K ℓ ˚,j pγ ℓ q ´pγ ℓ ˚´γ ℓ ˚qH j " ÿ j K ℓ,j pγ ℓ qK ℓ ˚,j pγ ℓ ˚q ´pγ ℓ ´γℓ qÿ j H j K ℓ ˚,j pγ ℓ ˚q ´pγ ℓ ˚´γ ℓ ˚q ÿ j H j K ℓ,j pγ ℓ q `pγ ℓ ´γℓ qpγ ℓ ˚´γ ℓ ˚q ÿ j H 2 j " ÿ j K ℓ,j pγ ℓ qK ℓ ˚,j pγ ℓ ˚q `Op p ? nqTherefore, by the law of large numbers,L n pWW T q ℓ,ℓ ˚" L n ÿ j K ℓ,j pγ ℓ qK ℓ ˚,j pγ ℓ ˚q`O p p1{ ? nq " LE " K ℓ,1 pγ ℓ qK ℓ ˚,1 pγ ℓ ˚q‰ `Op p1{ ? nq.Similarly, for W1 T , we have that for any ℓ, ℓ ˚P t1, . . . , Lu, pW1 T q ℓ,ℓ ˚" ÿ j K ℓ,j pγ ℓ q " ÿ j K ℓ,j pγ ℓ q ´pγ ℓ ´γℓ qH j " ÿ j K ℓ,j pγ ℓ q `Op p ? nq.Therefore, again by the law of large numbers,L N pW1 T q ℓ,ℓ ˚" L N ÿ j K ℓ,j pγ ℓ q `Op p1{ ? nq " LE " K ℓ,j pγ ℓ q ‰ `Op p1{ ? nq " 1 `Op p1{ ? nq.Combining the above results gives,r Ω n,ℓ,ℓ ˚" L n " pW ´1 L ¨1Lˆn qpW ´1 L ¨1Lˆn q T ȷ ℓ,ℓ " LE " K ℓ,1 pγ ℓ qK ℓ ˚,1 pγ ℓ ˚q‰ ´LE " K ℓ,j pγ ℓ q ‰ E " K ℓ ˚,j pγ ℓ ˚q‰ `Op p1{ ? nq " L Cov " K ℓ,1 pγ ℓ q, K ℓ ˚,1 pγ ℓ ˚q‰ `Op p1{ ? nq " q Ω ℓ,ℓ ˚`O p p1{ ? nq. Therefore, r Ω n p Ñ q Ω.To summarize, we have thatr U n,L " r A T n r A n d Ý Ñ χ 2 q Ω , and θ r Ωn,α p Ñ θ q Ω,α .Therefore,PpAlgorithm 2 rejectsq " Pp r U n,L ě θ r Ωn,α q Ñ Ppχ 2 q Ω ě θ q Ω,α q " α.Similar to the proof of Theorem 1, we define| W ℓ " n ÿ j"1w j `1tℓ j " ℓu ´γℓ apX j , Z j , V j q ˘`nγ ℓ E T rapX, Z, V qs, K ℓ,j pγq " w j `1tℓ j " ℓu ´γapX j , Z j , V j q ˘`γE T rapX, Z, V qs, andH j " w j apX j , Z j , V j q ´ET " apX, Z, V q ‰ .Therefore, we have | W ℓ " ř j K ℓ,j pγ ℓ q and Ă W ℓ " ř j K ℓ,j pγ ℓ q " | W ℓ ´pγ ℓ ´γℓ q ř j H j .We know from the literature that γ ℓ is the optimal choice of γ and thus Var" | W ℓ ı ď Var rW ℓ s.We will then move on to show that Var" Ă W ℓ ı is close to Var " | W ℓı and thus asymptotically no greater than Var rW ℓ s.To this end, note thatVar " Ă W ℓ ı " Var " | W ℓ ´pγ ℓ ´γℓ q ÿ j H j ı " Var " | W ℓ ı `2 Cov " | W ℓ , pγ ℓ ´γℓ q ÿ j H j ı `Var " pγ ℓ ´γℓ q ÿ j H j ‰ ď Var " | W ℓ ı `2c Var " | W ℓ ı c E " `pγ ℓ ´γℓ q ÿ j H j ˘2ı `E " `pγ ℓ ´γℓ q ÿ j H j ˘2ı .But we also know from the proof of Theorem 1 that γℓ ´γℓ p Ñ 0. Then, because of the bounded fourth moment assumption, by the Dominated Convergence Theorem, we have that1 n E " `pγ ℓ ´γℓ q ÿ j H j ˘2ı Ñ 0.Therefore,lim sup nÑ8 1 n ˆVar " Ă W ℓ ı ´Var " | W ℓ ı ˙ď 0.Finally, we note that Var rW l s " Ωpnq, and hencelim sup nÑ8 pVar " Ă W l ı { Var rW l sq ď 1.All experiments run on a Macbook Pro 2022 M2.Regarding running time for one iteration including density ratio estimation and X|Z model fitting (on average), csPCR took 5.12s, csPCR(pe) took 14.95s, IS method took 1.5s, PCR took 1.25s.Regarding running time for one test procedure, csPCR took 3.41s, csPCR(pe) took 11.32s, IS method took 0.81s.We find the optimal L value for the testing algorithm by performing numerical simulations, evaluating its Type-I error control and power. We adopt the same numerical simulation setup as in the main text Section 4. We first choose a S " 1 and a T " 0 and also fix β " 1 to compare the Type-I error rate for different choice of L of the csPCR method. We perform experiments with both true density ratio and estimated density ratio. The results are shown in Table 2 . We also test the power of the csPCR and csPCR(pe) method with different choices of L value. We choose a S " 0 and a T " 2 and fix β " 2.As Table 2 and Figure 5 shows, as L value increases, the csPCR method become more conservative with more tight Type-I error control and lower power. We can observe that when we set L " 3, the csPCR method can achieve most stable Type-I error rate control and also highest power empirically.Therefore, in our simulation experiments and real world data experiments, we fix L " 3.We notice a series of work in measuring the effective sample size (ESS) of importance weight or sampling in the statistical computation literature, e.g., [Martino, et al, 2017] and others. Among them, one of the most common ways is to use the ratio n ef f "p ř i"1 wiq 2 ř i"1 w 2 ito approximate the ESS. When the covariate shift between the source and target becomes stronger, the variance of the importance weight w i tends to be large and n ef f will become smaller, which can result in lower power. Our power enhancement method based on control variate could potentially alleviate this issue with properly specified control functions.In the simulation study, we varied only µ z , the mean of the confounding variables Z T . A higher µ z signifies a stronger covariate shift between the source and target populations. From Figure 6 , it is evident that as µ z increases, the Effective Sample Size (ESS) required significantly decreases, while the power of the csPCR method concurrently declines. These results suggest that increasing covariate shift leads to a reduction in ESS and a corresponding decrease in statistical power.In this section, we will use numerical simulations to ilustrate that the performance if the IS method is subject to the resample size heavily. IS method performs resampling without replacement and typically has to sample a much smaller subset (theoretically, in the order of op ? nq) of the source data to approximate the target. Consequently, the power of IS is substantially lower than our approach. If the resample size of IS is overly increased, it may fail to control the Type-I error due to excessive similarity between the resampled data and the original source data.To further illustrate, we conducted additional experiments with varied resample sizes in IS to assess its effect on Type-I error control and power. From Figure7. one can observe that IS starts to show high Type-I error inflation when its resample size increases to 400 but still shows much lower power (by around 0.4) than our method with this resample size (or even larger ones). This indicates that our method achieves better statistical efficiency than IS (DRPL).Figure 7 : Detailed comparison of Type-I error rate and power of csPCR and the IS method. With the source sample size n s " 1000, we gradually increase the resample size for the IS method from 100 to 1000. The two horizontal lines represent the Type-I error rate and power, respectively, of the csPCR and csPCR(pe) methods (they do not change with the tuning of IS).In this section we explore the effect of test statistics on the algorithm performance. The main principle of choosing the test statistic is to characterize the conditional dependency between X and Y under the alternative hypothesis. The test statistic Y X may not be the optimal choice and that using pY ´ÊrY | ZsqpX ´ErX | Zsq could remove the confounding effect of Z.Inspired by this, we used Y pX ´ErX|Zsq as the test statistic to conduct additional simulations. As illustrated in Figure8, we find that Y pX ´ErX|Zsq and Y X produce nearly the same power for both csPCR and csPCR(pe) with the change of effect size.The specific medication indicated by the treatment variable X includes Ritonavir, Bamlanivimab, Casirivimab-Imdevimab, Remdesivir, Ritonavir Nirmatrelvir, Sotrovimab, Bamlanivimab Etesevimab.For simplicity, X " 1 indicates any of these specific medication and X " 0 otherwise. In our real data experiment part, the outcome variable Y is defined as mortality within 90 days since hospital admission due to COVID-19. In addition, we also analyzed mortality within 30 days since hospital admission. As shown in Table 3 , both csPCR and csPCR(pe) methods give significant results, aligning with biomedical literature. Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper.• The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.• The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.• It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: Details of the simulation studies and real-data application are included in Sections 4, 5 and Appendix B.2.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.• The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes] Justification: Type-I errors, power, and p-values are provided in Sections 4 and 5.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).• The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)• The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean.• It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.• For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).• If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes]Justification: We record relevant information in Appendix B.1.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.• The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.• The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?Justification: Yes, our research conforms to the NeurIPS Code of Ethics in every respect, including fairness, transparency, privacy, and social responsibility.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.• The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes]Justification: We have discussed the impact of the paper on the fields of healthcare and social sciences.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.• The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.• If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: We believe the paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.• Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.• We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: We have cited the relevant papers and packages.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL.• The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.• For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.• If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [Yes]Justification: We have provided detailed documentation of the newly proposed algorithm.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.• The paper should discuss whether and how consent was obtained from people whose asset is used.• At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA]Justification: The paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.• Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.• According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.• Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. 
: 
 dimensional vector of confounding variables Z. This type of testing is critical for understanding the complex relationships among variables. For instance, scientists may hope to understand whether a specific genetic feature influences disease outcomes, whether a particular treatment effectively extends life expectancy, or whether certain demographic factors impact college admissions.Traditionally, these conditional testing problems are approached by modeling Y against X and Z through some parametric or semiparametric model. However, this strategy has been criticized due to potential model misspecification and limited observations of Y . As an alternative strategy, the model-X framework and Conditional Randomization Test (CRT) propose testing for the general conditional independence hypothesis H 0 : X K K Y | Z, free of any specific effect parameters [2] .The CRT assumes the distribution of X | Z to be known and can control the type-I error exactly, allowing for the choice of any flexible, black-box test statistic. This strategy is particularly useful when there is either strong and reliable scientific knowledge of the distribution of X | Z or an auxiliary dataset of pX, Zq of large sample size, known as the semi-supervised setting.Figure 1 : Type-I Error rates of our proposed csPCR and the source-only PCR on a simulated example. The Type-I error inflation of PCR demonstrates that source analysis is not valid or generalizable on the target due to covariate shift.In practice, testing for conditional independence frequently involves using data from a source population to draw conclusions about a target population. This situation presents challenges due to potential differences in the distribution of variables between the two populations. For example, economists may be interested in whether college admission (Y ) is independent of family income (X), conditioning on variables such as GPA, extracurricular activities, geographic location, and other demographics (Z). In the source population, the relationship might be influenced by factors like wealthy parents investing in SAT preparation, which boosts admission rates-a relationship that may not exist in a target population where such preparation is less common. Although Y may not appear independent of X given Z in the source population, the conclusion could vary significantly in the target population. This discrepancy underscores the need for a robust and flexible testing procedure that can adapt to shifts in distributions.More specifically, we address the covariate shift scenario, where the distributions of the treatment variables X, the confounding variables Z, and some surrogate or auxiliary variables V (e.g., SAT scores) may differ between the source and target populations. However, the conditional distribution of Y given X, Z, and V remains the same between them. In such scenarios, our goal is to leverage information from the source to accurately test for conditional independence in the target population without the observation of Y on target. In the scenario we consider, the presence of V and potential differences in P pV | X, Zq between the source and target populations may lead to the conditional independence X K K Y | Z not holding simultaneously in the two populations. Specifically, because P pY | X, Zq " ż P pY | X, Z, V qP pV | X, Zq dV, the conditional distribution of Y given X and Z can vary between populations. This underscores why the problem is non-trivial.See Figure 1 for an example of the consequences of such covariate shift.In this paper, we propose a novel conditional independence test suitable for covariate shift scenarios.Our method builds upon the Pearson Chi-Squared Conditional Randomization (PCR) test, a powerful model-X testing procedure that effectively addresses a broader range of alternative p-value distributions than the vanilla CRT [5] . Methodologically, we make two major contributions. First, we introduce importance weights into the label counting steps of the original PCR test, making the new test valid under covariate shift. These weights adjust the importance of each sample according to its density ratio, effectively rebalancing the source data to match the target population's distribution.Second, we introduce a power enhancement method that employs the control variates method to reduce variance in the test statistics. Although importance weights can increase the variance in test statistics, especially when the density ratio can become extremely high, potentially reducing power, our power enhancement method effectively addresses this issue. Together, these innovations enable us to develop a PCR test that is both powerful and valid under covariate shifts.The rest of the paper is organized as follows: In Section 2, we provide a formal introduction to the problem setup. In Section 3, we introduce the proposed Covariate Shift Corrected Pearson Chi-squared Conditional Randomization (csPCR) test and establish that the proposed csPCR test controls the Type-I error asymptotically. In Section 4, we demonstrate the empirical performance of the csPCR test through simulation studies. In Section 5, we apply the proposed csPCR test to a real-world dataset to assess the impact of a COVID-19 treatment on the 90-day mortality rate among patients. 
 
 Appendix:  


Decision: This paper addresses the problem of conditional independence testing under covariate shifts. The goal is to determine whether $p(y|x,z) = p(y|z)$ (i.e., whether X and Y are conditionally independent given Z) on the target data, by making use of another source dataset. The main challenge arises because there can be covariate shifts between the source and the target. The paper introduces importance weights into the Pearson Chi-Squared Conditional Randomization (PCR) test and makes use of the control variates method to address the potential covariate shifts. It is theoretically proven that the new test is consistent (asymptotically has maximum test power), and asymptotically has a well-controlled false positive rate.

**Exclude an official review from Reviewer G2Ah**

Reviewer G2Ah misunderstood the problem setting and could not go through the submission to provide a full review. In particular, in the paper, it is assumed that $P(Y|X, Z, V)$ remains the same from source to target data. The misunderstanding was that this would mean $P(Y|X,Z)$ is also the same and so the test can be conducted on the source data to draw conclusions on the target data. This statement is NOT true due to the presence of $V$. This misunderstanding was resolved during the reviewer discussion period with the AC. This official review is negative because of a misunderstanding. The AC decided to exclude this review. 

The authors are strongly advised to emphasize this point more. In particular, it may be useful to include the derivation to arrive at $P(Y|X,Z)=\int P(Y|X,Z,V)P(V|X,Z)\thinspace dV$ in the paper.


**Overall review**

The remaining three reviewers noted the originality of the method in this specific setting (REEU), and strong theoretical contribution (r237) with convincing experiments (MwvS). A common point raised is that the method depends on the accuracy of the estimated density ratio (for importance weighting) (MwvS, r237) which may limit applicability of the method in real problems (REEU). The authors sufficiently addressed this concern by providing more empirical results in the rebuttal. 

Recommendation: accept.

The authors are strongly suggested to add more results to questions raised by the reviewers e.g.,generate multiple permuted datasets on the Covid data experiment, evaluate different outcomes such as mortality, etc.

Reviews: review: paper_summary: paper_summary: This paper addresses the issue of conditional independence under covariate shift. The authors' goal is to test the conditional independence for causal inference in the target data. The authors can use the source data whose distribution is potentially different from that of the target data. For this problem, the authors propose using the covariate-shift adaptation method and develop a method for testing the null hypothesis.
strength_weakness: strength_weakness: See above.
questions: questions: See above.
limitations: limitations: See above.
paper_summary: paper_summary: This paper introduces the Covariate Shift Corrected Pearson Chi-squared Conditional Randomization (csPCR) test, designed to address covariate shift in conditional independence testing. The csPCR method incorporates importance weights and employs the control variates method to enhance test power and reduce variance in the statistical analysis. Theoretical contributions demonstrate that csPCR controls Type-I error asymptotically. Empirical validations through simulation studies and a real-world application assessing COVID-19 treatment effectiveness showcase the method's effectiveness.
strength_weakness: strength_weakness: 1. **Dependence on Accurate Estimations of Density Ratio:** The csPCR test's performance is critically dependent on the accurate estimation of density ratios. This dependence could pose significant challenges in practical scenarios characterized by limited, noisy, or high-dimensional data, where reliable density ratio estimation becomes inherently difficult.

2. **Clarity on the Advantage Over Resampling Methods:** The paper does not explicitly clarify the advantages of the csPCR test over simpler resampling-based methods that also utilize estimated density ratios. While the csPCR integrates density ratios directly into the test statistic calculation and employs variance reduction techniques, it remains crucial for the authors to demonstrate why these features offer substantial improvements over traditional methods that adjust the sample weights during resampling. The discussion should address whether the complexity of csPCR provides tangible benefits, such as improved error rates or robustness in more varied practical applications, compared to potentially simpler resampling approaches.
questions: questions: - Could the authors provide a detailed comparison between the csPCR test and traditional resampling methods that also use estimated density ratios? This would further strengthen the paper
- The paper mentions the use of control variates to reduce variance introduced by importance weights. Could the authors discuss how this approach compares to variance reduction techniques used in resampling methods? 
- Could the authors elaborate on any theoretical limitations or potential failure modes of the csPCR method?
- Are there additional empirical validations planned or underway to test the csPCR method across more varied datasets?
limitations: limitations: The authors are encouraged to add a separate limitation section in either the main paper or appendix.
paper_summary: paper_summary: This paper proposes a new variation of CRT to be applied in the presence of covariate shifts. The paper presents a method and one extension for each, with higher power. Then, the authors present some needed theoretical results and finish with experiments.
strength_weakness: strength_weakness: - It would be interesting if the authors could estimate the full density ratio in their simulations (possibly in a high-d scenario) and then compare their results with the IS approach;
- The paper does not conduct any experiment where Type-I error control is shown in a real dataset, e.g., like for example in [1,2].

References

[1] Pogodin, R., Schrab, A., Li, Y., Sutherland, D. J., & Gretton, A. (2024). Practical Kernel Tests of Conditional Independence. arXiv preprint arXiv:2402.13196.

[2] Maia Polo, Felipe, Yuekai Sun, and Moulinath Banerjee. "Conditional independence testing under misspecified inductive biases." Advances in Neural Information Processing Systems 36 (2023): 58577-58612.
questions: questions: - How can we understand the role of the effective sample sizes (weight imbalance) in the effectiveness of your method?
- Could you elaborate more on the surrogate variables? I am familiar with the literature on CI testing, but I haven't seen the presence of these variables in other works.
limitations: limitations: The authors comment on limitations.
paper_summary: paper_summary: The paper introduces a novel approach to addressing covariate shift in conditional independence tests. By leveraging importance weights and the control variates method,the paper proposes Covariate Shift Corrected Pearson Chi-squared Conditional Randomization (csPCR) test, which maintains asymptotic Type-I error control. In addition, a power enhancement is proposed, which reduces the variance in the test statistic and thus improve the power towards alternatives. The methodology is validated both theoretically and empirically, demonstrating superior performance in simulation studies and practical application to a COVID-19 treatment dataset, highlighting its efficacy in real-world scenarios.
strength_weakness: strength_weakness: The paper does not leave serious technical weaknesses, but I am concerned about the limited applicability of this method in real-world problems. Based on my understanding, one would test for conditional independence directly on the target distribution if the outcomes in the target population were easy to collect. So csPCR is useful only when the outcomes are costly to collect. This questions several application examples mentioned in the paper. First, in the college admission example (from Line 37-53), why don't the economists just collect the college admission results from the target population and then conduct the vanilla PCR? Typically, the target outcomes have to be collected for followup analysis after the conditional independence test. Second, Section 5 introduces a real-world application of this method on Covid-19 pandemic data, where the source and the target data set are segmented based on time. Given such a sequential data collecting process, it is unclear whether the iid assumption still holds in the source and target distribution. See Questions for more details.

Despite of the concerns above, I do believe that this method is useful for certain genetic and biomedical problems, where the outcomes are difficult to collect, but other covariates (such as SNPs) are relatively easy. Therefore, I suggest the authors clarify the application examples a bit by discussing more reasonable applications with some references.
questions: questions: 1. Page 5, Algorithm 1: Besides the variance issues considered in Algorithm 2, it is clear that the choice of test statistic in Eq (4) also affects the power. Can you comment on how to choose the test statistic? In the numerical experiment, the test statistic is specified to be the product of $Y$ and $X$, which makes sense as it measures the "correlation" between Y and X. However, $Z$ is not included in the test statistic, which seems to decrease its power. Have you tried any other test statistics? Do difference choices affect the power significantly? For instance, one alternative is to use $(Y - \beta_y Z)(X - \beta_x Z)$, where $\beta_y and \beta_x$ are regression coefficients for $Y\sim Z$ and $X\sim Z$ on an independent data set from source distribution.

2. Page 4, Line 170-177: The paper argues that they used PCR as the baseline test because PCR is more powerful than the vanilla CRT. But on the other hand, CRT is also better than PCR in the sense that it has finite-sample type I error control, rather than asymptotic type I error control. It's worth mentioning this point as well. While I notice that the simulations in the paper shows satisfactory type I error control, the type I errors may get inflated if, for example, the data have heavy tails.

3. Page 9, Section 9: Relating to the weaknesses, have you run any tests to check the iid assumption on the Covid-19 data set?
limitations: limitations: The papers discuss about the limitations of their method under model misspecifications in Section 4, pointing out that the powerful enhancement can disappear when there is a full nonlinear component in the model. The authors have addressed the limitations adequately.

Comments: title: Re: Official Comment by Authors
Re: Rebuttal by Authors
Reminder to engage in the discussion
comment: Thanks for your detailed response.  Most major concerns have been solved. I would like to keep my original rating.
Thank you for your response. I have gained a deeper understanding of the authors' contributions.

However, to be honest, I am not entirely clear about the motivation behind this study. If the authors assume that the conditional distributions are equal between the source and target data, wouldn't it suffice to test only for the source data? I am not convinced about the necessity of concerning with the Type 1 error in test for the target data. In the other words, it is unclear for me why we conduct test for the target data. 

I also have doubts about the claim that the Type 1 error changes. I think this is due to covariate shift, but if we consider covariates as non-stochastic, wouldn't the Type 1 error remain unchanged?

My background is actually in economics. While the authors provide examples from economics, in social science data analysis, it is common to estimate models using only the source data and then conduct counterfactual simulations on the target data under the assumption that the conditional distributions are equal. That is, it is enough to care only about the model on the source data, and we can transfer the results under the assumption that the conditional distribution is invariant. Given these typical analysis procedures, I find the problem setting somewhat difficult to understand.

I do not oppose acceptance, but I have some reservations about the problem setting, so I will keep my current score. In the future, since this problem setting may not be intuitively accepted, it might be worthwhile to focus more on justifying the problem setting itself more rather than on the technical issue of controlling the Type 1 error.
Thank you for your reply and clarification! You are correct that our goal is to test $X \perp Y \mid Z$ on the target data, with the assumption that $P(Y \mid X, Z, V)$ remains consistent between the source and target domains. However, even if $P(Y \mid X, Z, V)$ is the same for both the source and target, $P(V \mid X, Z)$ can still differ between them. As we demonstrated in our example in the rebuttal, this difference in $P(V \mid X, Z)$ led to $X \perp Y \mid Z$ holding true on the source data but not on the target. If we only test on the source data, this discrepancy can significantly inflate the Type-I error rate, as illustrated in Figure 1 of our paper.
Please Let us know if you have any further concerns.
I appreciate the authors' reply, which has deepened my understanding of your contributions.

It seems I might have caused some misunderstanding with my previous question. Here is what I intended to ask:

Objective of this study: To investigate whether conditional independence holds in the target data.

Assumption in this study: Covariate shift = meaning that P(Y | X, Z, V) remains unchanged between the source data and the target data.

Under this situation, wouldn't testing only the source data suffice to achieve the objective, given the assumption?

I wanted to confirm whether I misunderstood the problem setting.
Hello reviewer MwvS. The reviewer-author discussion period is between Aug 7-13 (AoE). Please read the authors’ rebuttal, see if it addresses your questions, and engage in a discussion with the authors. You are strongly encouraged to read the official reviews posted by other reviewers. Some of your questions may have already been answered. **You must acknowledge the authors’ rebuttal and give a reason for why it did or did not address your concerns. If the rebuttal does not change your rating, please also state the reason.** Thank you for your services.
We express sincere thanks to your insightful comments and positive recognition on our work. 

$\textbf{W1}$: Thanks for pointing out that an increasing dimension tends to cause higher variational importance weights and lower ESS. We will follow this to design our additional simulation on power v.s. ESS (changing with the variance of importance weights).

$\textbf{W2}$: Thanks! When shuffling $Y$, we think both (i) marginal permutation and (ii) conditioning on (the full set or a subset of) Z will cause Y and X to be independent conditional on Z. Thus, we will try both setups and you are correct that we can use binning if Z is continuous and relatively high-dimensional.

$\textbf{Q1}$: Thanks for the question! We think the effective sample size of our csPCR estimator (without control variate) can be defined in a similar way as their paper, i.e., the original source sample size * the ratio between the variances of (i) and (ii), where (i) stands for the unweighted indicator on target (which one would obtain if they could observe and use the same amount of labeled sample on target for PCR), and (ii) stands for the weighted indicator variable on source. 

This ratio can be shown to be smaller or equal to 1, with the “=” holding only when there is no covariate shift between the source and target. Thus, covariate shift will cause loss of the effective sample size to our method and larger degree of covariate shift tends to induce larger variance of the weighted estimator on source and larger loss of the ESS. This interpretation is quite analog to that in IS literature. Further, we can write down the ESS for our power enhanced (PE) estimator and apply our Theorem 2 to show that it is larger than the ESS of csPCR without PE. 

$\textbf{Q2}$: Thanks for the suggestion and we will highlight more on the specific setup with surrogate in the abstract and introduction section!
Thank you for your reply!

- W1: Thank you for your new experiment! (this suggestion is related to the ESS question because in high-d we expect the ESS to be lower)
- W2: I think the idea is good. When shuffling Y, you will need to do that within each value of Z, correct? If your Z is continuous, you would probably need to use some binning.
- Q1: My question was related to the paper you mentioned here by Martino et al. In that paper, they argue that the ESS can be defined as a variance ratio and the implications of that are clear in the IS literature. I was wondering if you could extract some more meaningful relationships here as well.
- Q2: Thank you for the detailed explanation! I would probably try to input some introduction on the surrogate variable in the abstract since you mention it but it might not be clear you are going to use them throughout the paper (you are working in this specific setup, which is not the same as the original CRT setup).

I have increased my score.
Thanks for the clarification. I don't have further concerns.
We extend our sincere thanks for your reviewing work and insightful comments. Please see our responses to your comments and questions as below in the rebuttal.