Title: An Efficient Doubly-Robust Test for the Kernel Treatment Effect
Venue: NeurIPS 2023
Accepted: yes
Keywords: kernel treatment effect

causal inference

maximum mean discrepancy
Abstract: The average treatment effect, which is the difference in expectation of the counterfactuals, is probably the most popular target effect in causal inference with binary treatments. However, treatments may have effects beyond the mean, for instance decreasing or increasing the variance. We propose a new kernel-based test for distributional effects of the treatment. It is, to the best of our knowledge, the first kernel-based, doubly-robust test with provably valid type-I error. Furthermore, our proposed algorithm is computationally efficient, avoiding the use of permutations.

An Efficient Doubly-Robust Test for the Kernel Treatment Effect

Abstract

The average treatment effect, which is the difference in expectation of the counterfactuals, is probably the most popular target effect in causal inference with binary treatments. However, treatments may have effects beyond the mean, for instance decreasing or increasing the variance. We propose a new kernel-based test for distributional effects of the treatment. It is, to the best of our knowledge, the first kernel-based, doubly-robust test with provably valid type-I error. Furthermore, our proposed algorithm is computationally efficient, avoiding the use of permutations.

1 Introduction

In the context of causal inference, potential outcomes (Rubin, 2005) are widely used to address counterfactual questions (e.g. what would have happened had some intervention been performed?). This framework considers

where X and Y represent the covariates and outcome respectively, and A ∈ { 0 , 1 } is a binary treatment. Furthermore, the following three conditions are assumed:

(Consistency) Y = AY ∗ 1 +(1 -A ) Y ∗ 0 (where Y ∗ 1 , Y ∗ 0 are the potential outcomes).

ii) (No unmeasured confounding) Y ∗ 0 , Y ∗ 1 ⊥ ⊥ A | X .

iii) (Overlap) For some ϵ > 0 , we have ϵ < π ( X ) := P A | X ( A = 1 | X ) < 1 -ϵ almost surely.

Such assumptions allow for identification of causal target parameters. For instance, one may be interested in the average variance-weighted treatment effects (Robins et al., 2008; Li et al., 2011), stochastic intervention effects (Muñoz and Van Der Laan, 2012; Kennedy, 2019), or treatment effect bounds (Richardson et al., 2014; Luedtke et al., 2015). However, most of the literature focuses on estimation and inference of the average treatment effect (Imbens, 2004; Hernan and Robins, 2020), defined as the difference in expectation of the potential outcomes

Given ( X i , A i , Y i ) N i =1 ∼ ( X,A,Y ) , there are three widespread estimators of ψ . First, the plug-in (PI) estimator:

where ˆ θ 1 ( X ) , ˆ θ 0 ( X ) estimate E [ Y ∗ 1 | X ] , E [ Y ∗ 0 | X ] . Second, the inverse propensity weighting (IPW) estimator:

where ˆ π ( X ) estimates the propensity scores P [ A = 1 | X ] . Third, the so called Augmented Inverse Propensity Weighted (AIPW) estimator:

Under certain conditions (e.g., consistent nuisance estimation at n -1 / 4 rates), the asymptotic mean squared error of the AIPW estimator is smaller than that of the IPW and PI estimator, and minimax optimal in a local asymptotic sense (Kennedy, 2022), hence it has become increasingly popular in the last decade. The AIPW estimator is often referred to as the doubly-robust estimator. We highlight that double-robustness is an intriguing property of an estimator that makes use of two models, in which the estimator is consistent even if only one of the two models is well-specified and the other may be misspecified; we refer the reader to Kang and Schafer (2007) for a discussion on doubly-robust procedures.

However, the treatment might have effects beyond the mean, for instance in the variance or skewness of the potential outcome. The average treatment effect will prove insufficient in this case. Consequently, one may be interested in testing whether the treatment has any effect in the distribution of the outcome. This question naturally arises in a variety of applications. For instance, one may want to check whether there is any difference between a brand-name drug and its generic counterpart, or understand whether a treatment simply shifts the distribution of the outcome (or, in turn, it also affects higher order moments).

̸

In this work, we revisit the problem of testing the null hypothesis H 0 : P Y ∗ 1 = P Y ∗ 0 against H 1 : P Y ∗ 1 = P Y ∗ 0 . We propose a distributional treatment effect test based on kernel mean embeddings and the asymptotic behaviour of the AIPW estimator. Our contributions are three-fold:

Up to our knowledge, we propose the first kernel-based distributional test to allow for doubly-robust estimators with provably valid type-I error.

The proposed distributional treatment effect test is permutation-free, which makes it computationally efficient.

We empirically test the power and size of the proposed test, showing the substantial benefits of the doubly robust approach.

2 Related work

Distributional treatment effects have been addressed from a variety of points of view. Abadie (2002) was one of the first works to propose to test distributional hypothesis attempting to estimate the counterfactual cumulative distribution function (cdf) of the outcome of the treated and untreated. Chernozhukov et al. (2013) proposed to regress the cdf after splitting the outcome in a grid. Further contributions followed with alike cdf-based approaches (Landmesser, 2016; Díaz, 2017).

Other approaches to the problem include focusing on the probability density function (pdf) instead of the cdf. Robins and Rotnitzky (2001) introduced a doubly robust kernel estimator for the counterfactual density, while Westling and Carone (2020) proposed to conduct density estimation under a monotone density assumption. Kim et al. (2018) and Kennedy et al. (2021) suggested to compute L p distances between the pdf of the outcome distribution for the different counterfactuals. Conditional distributional treatment effects have also been addressed, with Shen (2019) proposing to estimate the cdf of the counterfactuals for each value to be conditioned on.

On the other hand, kernel methods have recently gained more and more attention in the context of causal inference. Kernel-based two-stage instrumental variable regression was proposed in Singh et al. (2019), while Singh et al. (2020) presented estimators based on kernel ridge regression for nonparametric causal functions such as dose, heterogeneous, and incremental response curves. Furthermore, Singh et al. (2021) conducted mediation analysis and dynamic treatment effect using kernel-based regressors as nuisance functions. Causal inference with treatment measurement error i.e. when the cause is corrupted by error was addressed in Zhu et al. (2022) using kernel mean embeddings to learn the latent characteristic function.

However, kernel mean embeddings for distributional representation were not suggested in the causal inference literature until Muandet et al. (2021) proposed to use an IPW estimator to estimate the average treatment effect on the embedding, which leads to a kernel-based distributional treatment effect test based on this embedding and the MMD. With a very similar motivation, Park et al. (2021) proposed a test for conditional distributional treatment effects based on kernel conditional mean embeddings. In a preprint, Fawkes et al. (2022) extended the work to AIPW estimators, however no theoretical guarantees regarding type-1 error control of the proposed tests were provided.

Finally, the kernel-based tests used in the aforementioned cases involve test statistics that are degenerate U-statistics under the null, hence obtaining theoretical p-values of the statistic is not possible. In turn, Kim and Ramdas (2023) proposed the idea of cross U-statistics, which is based on splitting the data for achieving a normal asymptotic distribution after studentization. Similarly, Shekhar et al. (2022) exploited sample splitting for proposing a permutation-free kernel two sample test.

3 Preliminaries

In this section, we introduce the concepts that the proposed test for distributional treatment effects is mainly based on: Maximum Mean Discrepancy (Gretton et al., 2012), Conditional Mean Embeddings (Song et al., 2009), Kernel Treatment Effects (Muandet et al., 2021), dimension-agnostic inference using cross U-statistic (Kim and Ramdas, 2023; Shekhar et al., 2022), and the normal asymptotic behaviour of the doubly robust estimator (Funk et al., 2011).

3.1 Maximum Mean Discrepancy (MMD) and Conditional Mean Embeddings

Let X be a non-empty set and let H be a Hilbert space of functions f : X → R with inner product ⟨· , ·⟩ H . A function k : X ×X → R is called a reproducing kernel of H if (i) k ( · , x ) ∈ H , for all x ∈ X , (ii) ⟨ f, k ( · , x ) ⟩ H = f ( x ) for all x ∈ X , f ∈ H . If H has a reproducing kernel, then it is called a Reproducing Kernel Hilbert Space (RKHS).

Building on a reproducing kernel k and a set of probability measures P , the Kernel Mean Embedding (KME) maps distributions to elements in the corresponding Hilbert space as follows:

If the kernel k is ' characteristic ' (which is the case for frequently used kernels such as the RBF or Matern kernels), then µ is injective. Conditional mean embeddings (Song et al., 2009) extend the concept of kernel mean embeddings to conditional distributions. Given two RKHS H k x , H k y , the conditional mean embedding operator is a Hilbert-Schimdt operator C Y | X : H k x → H k y satisfying µ Y | X = x = C Y | X k x ( · , x ) , where C Y | X := C Y X C -1 XX , C Y X := E Y,X [ k y ( · , Y ) ⊗ k x ( · , X )] and C XX := E X,X [ k x ( · , X ) ⊗ k x ( · , X )] . Given a dataset { x , y } , a sample estimator may be defined as

where Φ y := [ k y ( · , y 1 ) , . . . , k y ( · , y n )] T , Φ x := [ k x ( · , x 1 ) , . . . , k x ( · , x n )] T , K xx := Φ x Φ T x denotes the Gram matrix, and λ is a regularization parameter.

Building on the KME, Gretton et al. (2012) introduced the kernel maximum mean discrepancy (MMD). Given two distributions P and Q and a kernel k , the MMD is defined as the largest difference in expectations over functions in the unit ball of the respective RKHS:

where

Alternatively, we may also define an unbiased version of it. Again, under the null, these are degenerate two-sample U-statistics. Hence, Muandet et al. (2021) proposed a permutation-based approach for thresholding.

The KTE considers the MMD between the mean embeddings of the counterfactual distributions. This is the cornerstone of the proposed distributional test, as will be exhibited in Section 4, which we extend in a doubly-robust manner to observational settings.

3.3 Permutation-free inference using cross U-statistics

The previously mentioned permutation-based approach for obtaining the threshold for the MMD statistic (and hence the KTE statistic) comes with finite-sample validity (Gretton et al., 2012). The number of permutations B used to find the empirical p-values generally varies from 100 to 1000. Consequently, the computational cost of finding a suitable threshold for the statistic is at least O ( Bn 2 ) . Such computational cost reduces the applicability of the approach, especially when time or computational resources are limited.

̸

Driven by developing dimension-agnostic inference tools, Kim and Ramdas (2023) presented a permutation-free approach to test null hypotheses of the form H 0 : µ = 0 against H 1 : µ = 0 , where µ is the mean embedding of a distribution P , based on the idea of sample splitting. If X 1 , ..., X 2 n ∼ P , the usual degenerate V-statistic considers

where ˆ µ 2 n = 1 2 n ∑ 2 n i =1 k ( · , X i ) (or the similar unbiased version). Kim and Ramdas (2023) proposed to split the data in two and study

where ˆ µ A n = 1 n ∑ n i =1 k ( · , X i ) , ˆ µ B n = 1 n ∑ 2 n j = n +1 k ( · , X j ) . Denoting U i = ⟨ k ( · , X i ) , ˆ µ B n ⟩ , we have that

Under the null and some mild assumptions on the embeddings, Kim and Ramdas (2023, Theorem 4.2) proved that

̸

It can be shown (Gretton et al., 2012) that MMD 2 ( P , Q ) = || µ P -µ Q || 2 H . If k is characteristic, then MMD ( P , Q ) = 0 if and only if P = Q . Given two samples drawn from P and Q , the MMD between the empirical distributions may be used to test the null hypothesis H 0 : P = Q against H 1 : P = Q . However, this statistic is a degenerate two-sample U-statistic under the null, thus one cannot analytically calculate the critical values. Consequently, a permutation-based resampling approach is widely used in practice (Gretton et al., 2012).

3.2 Kernel Treatment Effect: A distributional kernel-based treatment effect test

Based on the MMD and the potential outcomes framework, Kernel Treatment Effects (KTE) were introduced in Muandet et al. (2021) for testing distributional treatment effects in experimental settings (i.e. with known propensity scores).

̸

Let ( X i , A i , Y i ) n i =1 ∼ ( X,A,Y ) such that (i) (consistency) Y = AY ∗ 1 -(1 -A ) Y ∗ 0 , (ii) no unmeasured confounding and overlap assumptions hold. The KTE considers the MMD between Y ∗ 0 and Y ∗ 1 to test H 0 : P Y ∗ 0 = P Y ∗ 1 against H 0 : P Y ∗ 0 = P Y ∗ 1 . They define

where ¯ U = 1 n ∑ n i =1 U i , ˆ S 2 u = 1 n ∑ n i =1 ( U i -¯ U ) 2 . Consequently, ¯ x ̂ MMDis the statistic considered and the null is rejected when ¯ x ̂ MMD > z 1 -α , where z 1 -α is the (1 -α ) -quantile of N (0 , 1) . Such test avoids the need for computing the threshold so it reduces the computational cost by a factor of 1 B , and it is minimax rate optimal in the L 2 distance and hence its power cannot be improved beyond a constant factor (Kim and Ramdas, 2023).

The permutation-free nature of cross U-statistic is key in the proposed distributional test. It will allow us to circumvent the need for training regressors and propensity scores repeatedly, while preserving theoretical guarantees.

3.4 Empirical mean asymptotic behavior of AIPW

The main property from AIPW estimators that will be exploited in our proposed distributional treatment effect is the asymptotic empirical mean behaviour of the estimator. We present sufficient conditions in the next theorem.

Theorem 3.1. Let f ( x, a, y ) = { a π ( x ) -1 -a 1 -π ( x ) }{ y -θ a ( x ) } + θ 1 ( x ) -θ 0 ( x ) , so that ψ = E { f ( X,A,Y ) } is the average treatment effect. Suppose that

ˆ f is constructed from an independent sample or f and ˆ f are contained in a Donsker class.

|| ˆ f -f || = o P (1) .

Suppose also that (by clipping) P (ˆ π ∈ [ ϵ, 1 -ϵ ]) = 1 . If || ˆ π -π || ∑ a || ˆ θ a -θ a || = o P ( 1 √ n ) , then it follows that

so it is root-n consistent and and asymptotically normal.

Note that the IPW estimator can be seen as an AIPW with ˆ θ 0 ( X ) = ˆ θ 1 ( X ) = 0 almost surely. The IPW estimator is also asymptotically normal if ∥ ˆ π -π ∥ = o P ( 1 √ n ) . In experimental settings ∥ ˆ π -π ∥ = 0 , hence the root-n rate is achieved. Under certain conditions (e.g., consistent nuisance estimation at n -1 / 4 rates), the asymptotic variance of the AIPW estimator is minimized for ˆ θ 1 = θ 1 , ˆ θ 0 = θ 0 , thus the IPW estimator is generally dominated by the AIPW if ˆ θ 1 , ˆ θ 0 are consistent.

The idea exhibited in Theorem 3.1 will allow for using cross U-statistics in estimated mean embeddings, rather than the actual embeddings. Nonetheless, Theorem 3.1 applies to finite-dimensional outcomes Y . We state and prove the extension of Theorem 3.1 to Hilbert spaces in Appendix C, which will be needed to prove the main result of this work.

4 Main results

We are now ready to introduce the main result of the paper. Let Z ≡ ( X,A,Y ) ∼ P be such that Y = AY ∗ 1 +(1 -A ) Y ∗ 0 and that both no unmeasured confounding and overlap assumptions hold. We denote the space of observations by Z = X×A×Y . Weare given Z i ≡ ( X i , A i , Y i ) 2 n i =1 ∼ ( X,A,Y ) and we wish to test H 0 : P Y ∗ 1 = P Y ∗ 0 against H 1 : P Y ∗ 1 = P Y ∗ 0 . Given characteristic kernel k i.e. k ( y, ˜ y ) = ⟨ k ( · , y ) , k ( · , ˜ y ) ⟩ with induced RKHS H k , we equivalently test H 0 : E [ k ( · , Y ∗ 1 ) -k ( · , Y ∗ 0 )] = 0 .

̸

Under consistency, no unmeasured confounding, and overlap, we have

where

Note the change in notation, from θ a to β a , to emphasize that such regression functions are now H k -valued. Thus, we can equivalently test for H 0 : E [ ϕ ( Z )] = 0 . With this goal in mind, we denote D 1 = ( X i , A i , Y i ) n i =1 , D 2 = ( X j , A j , Y j ) 2 n j = n +1 and define

Above, ˆ ϕ ( r ) ( z ) is the plug-in estimate of ϕ ( z ) for r ∈ { 1 , 2 } using ˆ π ( r ) and ˆ β ( r ) a , which approximate π and β a respectively. Further, ¯ f † h and S † h denote the empirical mean and standard error of f † h :

The next theorem, which is the main result of the paper, establishes sufficient conditions for T † h to present Gaussian asymptotic behavior. While the main idea relies on combining cross U-statistics and the asymptotic empirical mean-like behaviour of AIPW estimators, we highlight a number of technical challenges underpinning this result. The proof combines the central idea presented in Kim and Ramdas (2023) with a variety of techniques including causal inference results, functional data analysis, and kernel method concepts. Furthermore, additional work is needed to extend Theorem 3.1 to H k -valued outcomes. Donsker classes are only defined for finite dimensional outcomes; in the H k -valued scenario, we ought to refer to asymptotically equicontinuous empirical processes (Park and Muandet, 2023) and Glivenko-Cantelli classes. We refer the reader to Appendix C for a presentation of such concepts, clarification of the norms used, and the proof of the theorem.

Theorem 4.1. Let k be a kernel that induces a separable RKHS and P Y ∗ 0 , P Y ∗ 1 be two distributions. Suppose that (i) E [ ϕ ( Z )] = 0 , (ii) E [ ⟨ ϕ ( Z 1 ) , ϕ ( Z 2 ) ⟩ 2 ] > 0 , (iii) E [ ∥ ϕ ( Z ) ∥ 4 H ] is finite. For r ∈ { 1 , 2 } , suppose that (iv) ˆ ϕ ( r ) is constructed independently from D r or (v) the empirical process of ˆ ϕ ( r ) is asymptotically equicontinuous at ϕ and ∥ ˆ ϕ ( r ) ∥ 2 H belongs to a Glivenko-Cantelli class. If it also holds that (vi) || ˆ ϕ ( r ) -ϕ || = o P (1) , (vii) P (ˆ π ( r ) ∈ [ ϵ, 1 -ϵ ]) = 1 , and

for r ∈ { 1 , 2 } , then it follows that where

We would like to highlight the mildness of the assumptions of Theorem 4.1. The separability of the RKHS is achieved for any continuous kernel on separable Y (Hein and Bousquet, 2004). Assumption (i) is always attained under the null hypothesis (it is precisely the null hypothesis). Assumption (ii) prevents ϕ ( Z ) from being constant. In such degenerate case, S † h = 0 thus T † h is not even well-defined. Assumption (iii) is the more restrictive out of the first three assumptions, inherited from the use of Lyapunov's CLT in the proof. However, we note that this condition is immediately satisfied under frequently used kernels. For instance, under bounded kernels (for example the common Gaussian and Laplace kernels) such that ∥ k ( · , Y ) ∥ H ≤ M , we have that

hence

which is upper bounded by 2( ϵ -1 +1) M . Consequently, E [ ∥ ϕ ( Z ) ∥ 4 H ] ≤ [ 2( ϵ -1 +1) M ] .

Furthermore, conditions (iv), (vi), (vii) and (3) deal with the proper behaviour of the AIPW estimator; they are standard in the causal inference scenario. Condition (iv) is equivalent to two-fold cross-fitting i.e., training ˆ ϕ ( r ) on only half of the data and evaluating such an estimator on the remaining half. Condition (v) replaces the Donsker class condition from the finite dimensional setting.

We emphasize the importance of double-robustness in the test; normality of the statistic is achieved due to the o P (1 / √ n ) rate, which is possible in view of the doubly robust nature of the estimators. We also highlight the fact that IPW estimators of the form ˆ ϕ ( r ) ( z ) = { a ˆ π ( x ) -1 -a 1 -ˆ π ( x ) } k ( · , y ) can be embedded in the framework considering β 0 , β 1 = 0 . In fact, the doubly robust kernel mean embedding estimator may be viewed as an augmented version of the KTE (which is a kernelized IPW) using regression approaches to kernel mean embeddings (Singh et al., 2020), just as AIPW augments IPW with regression approaches. Furthermore, (3) is always attained when the propensity scores π are known (i.e. experimental setting), given that ∥ ˆ π ( r ) -π ∥ = 0 .

̸

Based on the normal asymptotic behaviour of T † h , we propose to test the null hypothesis H 0 : P Y ∗ 1 = P Y ∗ 0 given the p-value p = 1 -Φ( T † h ) , where Φ is the cdf of a standard normal. For an α -level test, the test rejects the null if p ≤ α . We consider a one-sided test, rather than studying the two-sided p-value 1 -Φ( | T † h | ) , given that positive values of T † h are expected for E [ ϕ ( Z )] = 0 . The next algorithm illustrates the full procedure of the test, which we call AIPW-xKTE (Augmented Inverse Propensity Weighted cross Kernel Treatment Effect).

Algorithm 1 AIPW-xKTE

1: input Data D = ( X i , A i , Y i ) 2 n i =1 .

2: output The p-value of the test.

3: Choose kernel k and estimators ˆ β ( r ) a , ˆ π ( r ) for r ∈ { 1 , 2 } .

4: Split data in two sets D 1 = ( X i , A i , Y i ) n i =1 , D 2 = ( X i , A i , Y i ) 2 n i = n +1 .

5: If ˆ β ( r ) a , ˆ π ( r ) are such that condition (v) from Theorem 4.1 is attained, train them on D . Otherwise, train them on D 1 -r .

6: Define ˆ ϕ ( r ) ( z ) = { a ˆ π ( r ) ( x ) -1 -a 1 -ˆ π ( r ) ( x ) }{ k ( · , y ) -ˆ β ( r ) a ( x ) } + ˆ β ( r ) 1 ( x ) -ˆ β ( r ) 0 ( x ) .

7: Define f † h ( Z i ) = 1 n ∑ 2 n j = n +1 ⟨ ˆ ϕ (1) ( Z i ) , ˆ ϕ (2) ( Z j ) ⟩ for i = 1 , . . . , n . √

8: Calculate T † h := n ¯ f † h S † h , where ¯ f † h = 1 n ∑ n i =1 f † h ( Z i ) and S † h = √ 1 n ∑ n i =1 ( f † h ( Z i ) -¯ f † h ) 2 .

9: return p-value p = 1 -Φ( T † h ) .

Note that the proposed statistic is, at heart, a two sample test (with a nontrivial causal twist); in contrast to Shekhar et al. (2022), the two samples are not independent and are potentially confounded.

Extensive literature focuses on designing estimator ˆ π ( r ) , logistic regression being the most common choice. At this time, not so many choices exist for estimators ˆ β ( r ) a , given that it involves a regression task in a Hilbert space. Conditional mean embeddings are the most popular regressor, although other choices exist ( ´ Cevid et al., 2022).

Note that we have motivated the proposed procedure for testing distributional treatment effects with characteristic kernels. However, the actual null hypothesis being tested is H 0 : E [ k ( · , Y ∗ 1 )] = E [ k ( · , Y ∗ 0 )] . If the kernel chosen is not characteristic, the test would continue to be valid for H 0 , although it would not be valid to test equality between P Y ∗ 0 and P Y ∗ 1 . For instance, AIPW-xKTE with a linear kernel could be used to test equality in means of counterfactuals.

Furthermore, the proposed test is permutation-free, as the statistic T † h ought to be computed only once.

This permutation-free nature is crucial, as it avoids the repeated estimation of ˆ π ( r ) , ˆ β ( r ) a . For instance, conditional mean embeddings involve the inversion of a matrix, which scales at least at O ( n ω ) , with practical values being ω = 2 . 87 by Strassen's algorithm (Strassen et al., 1969). Calculating the conditional mean embedding for every permutation would imply O ( Bn ω ) , where B is the number of permutations. Furthermore, regressors for mean embeddings of different nature might involve a higher complexity, hence avoiding permutations becomes even more important in the approach.

If the actual embedding ϕ was known, the power of AIPW-xKTE could not be improved beyond a constant factor (by minimax optimality of cross U-statistics in L 2 distance). Further, every procedure will suffer from the error in estimation of ϕ . This means that we are potentially incurring in a loss of power by avoiding a permutation-based approach, however such a loss is controlled by a small factor. Nonetheless, this potential loss is inherited from splitting the data in our estimator (only half of the data is used on each side of the inner product). We highlight that sample splitting is needed when using flexible doubly-robust estimators, hence we expect no loss in power compared to other potential doubly-robust approaches in that case.

5 Experiments

In this section, we explore the empirical calibration and power of the proposed test AIPW-xKTE. For this, we assume that we observe ( x i , a i , y i ) n i =1 ∼ ( X,A,Y ) and that (causal inference assumptions) consistency, no unmeasured confounding, and overlap hold. Both synthetic data and real data are evaluated. All the tests are considered at a 0.05 level. For an exhaustive description of the simulations and outcomes, including additional experiments, we direct the reader to Appendix B.

Synthetic data. All data (covariates, treatments and responses) are artificially generated. We define four scenarios:

Scenario I: There is no treatment effect; thus, P Y ∗ 0 = P Y ∗ 1 .

Scenario II: There exists a treatment effect that only affects the means of P Y ∗ 0 , P Y ∗ 1 .

Scenario III and Scenario IV: There exists a treatment effect that does not affect the means but only affects the higher moments of P Y ∗ 0 and P Y ∗ 1 , differently for each scenario.

For all four scenarios, we consider the usual observational study setting, where the propensity scores π ( X ) are treated as unknown and hence they must be estimated. We define the proposed AIPW-xKTE test with the mean embedding regressions fitted as conditional mean embeddings and the propensity scores estimated by logistic regression.

We first study the empirical calibration of AIPW-xKTE and the Gaussian behaviour of T † h under the null. Figure 1 exhibits the performance of AIPW-xKTE in Scenario I. Both a standard normal behaviour and proper calibration are empirically attained in the simulations.

Due to the fact that the KTE (Muandet et al., 2021) may not be used in the observational setting, where the propensity scores are not known, there is no natural benchmark for the proposed test. In particular, we were unable to control the type-1 error of the test presented in Fawkes et al. (2022), and hence omitted from our simulations. Consequently, we compare the power of the proposed AIPW-xKTE and IPW-xKTE with respect to three methods that are widely used while conducting inference on the average treatment effect: Causal Forests (Wager and Athey, 2018), Bayesian Additive Regression Trees (BART) (Hahn et al., 2020), and a linear regression based AIPW estimator (Baseline-AIPW).

(A)

(B)

(C)

3

Draws from AIPW-xKTE

0.14

0.4

Standardnormalpdf

2

0.12

rate

Values

1

0.10

Density

positive

Ordered

0

0.08

0.2

-1

alse

0.06

0.04

0.1

2

0.02

0.0

0.00

-2

0

2

-2

0

2

200

400

600

800

1000

AIPW-xKTE

Theoreticalquantiles

n

(I)

(II)

(IV)

1.0

1.0

1.0

rate

0.8

T

0.8

0.8

positive

0.6

0.6

0.6

0.4

0.4

0.4

AIPW-xKTE (our proposal)

AIPW-xKTE (our proposal)

AIPW-xKTE (our proposal)

Baseline-AIPW

Baseline-AIPW

Baseline-AIPW

0.2

0.2

0.2

BART

BART

BART

CausalForest

Causalforest

Causalforest

0.0

0.0

0.0

100

150

200

250

300

350

100

150

200

250

300

350

100

150

200

250

300

350

n

n

n

Figure 2 exhibits the performance of such tests in Scenario II, Scenario III, and Scenario IV . The three methods dominate AIPW-xKTE in Scenario II, where there exists a mean shift in counterfactuals. However, and as expected, such methods show no power if the distributions differ but have equal means. In contrast, AIPW-xKTE detects distributional changes beyond the mean, exhibiting power in all scenarios.

Remark: While this work focuses on the observational setting, where double robustness is crucial, we highlight that the proposed AIPW-xKTE test may also be used in experiments (where propensity scores are known) for computational gains. The proposed test avoids permutations, which makes it more computationally efficient than the KTE (Muandet et al., 2021). We refer the reader to Appendix A for a comparison between the proposed AIPW-xKTE and the KTE.

Real data. We use data obtained from the Infant Health and Development Program (IHDP) and compiled by Hill (2011), in which the covariates come from a randomized experiment studying the effects of specialist home visits on cognitive test scores. This data has seen extensive use in causal inference (Johansson et al., 2016; Louizos et al., 2017; Shalit et al., 2017). We work with 18 variables of the covariate set and unknown propensity scores. We highlight that the propensity score model is likely misspecified in this real life scenario.

We consider six scenarios with the IHDP data. For Scenarios I, II, III and IV , we generate the response variables similarly to the previous experiments. In Scenario V, we take the IQ test (Stanford Binet) score measured at the end of the intervention (age 3) as our response variable. For Scenario VI, we calculate the average treatment effect in Scenario V using Causal Forests (obtaining 0.003 i.e. a positive shift) and subtract it from the IQ test score of those who are treated, thus obtaining a distribution of the IQ test scores with zero average treatment effect.

For each of the scenarios, we consider the tests AIPW-xKTE, Causal Forests, BART and BaselineAIPW on 500 bootstrapped subsets to estimate the rejection rates. All of them showed expected levels of rejection under the null i.e. Scenario I. The results for the remaining scenarios can be found in Table 1. We note that the performance of the tests on Scenarios I, II, III, and IV is similar to the analogous scenarios on synthetic data. While AIPW-xKTE exhibits a loss in power with respect to

the other tests when the average treatment effect is non zero, it detects distributional changes beyond the mean.

Besides BART, all of the tests reject the null in almost every simulation of Scenario V. As expected, Causal Forest, BART and Baseline-AIPW barely reject the null in Scenario VI, which considers the data of Scenario V with zero average treatment effect. Interestingly, the proposed distributional test has a rejection rate below 0.05, which supports the fact that specialist home visits have no effect on the distribution of cognitive test scores beyond an increase in the mean.

6 Conclusion and future work

We have developed a computationally efficient kernel-based test for distributional treatment effects. It is, to our knowledge, the first kernel-based test for distributional treatment effects that allows for a doubly-robust approach with provably valid type-I error. Furthermore, it does not suffer from the computational costs inherent to permutation-based tests. The proposed test empirically proves valid in the observational setting, where its predecessor KTE may not be used: the test is well calibrated and shows power in a variety of scenarios. Procedures designed to test for average treatment effects only outperform the proposed test if there is a mean shift between counterfactuals.

There are several possible avenues for future work. We highlighted that our procedure holds if consistency, no unmeasured confounding, and overlap hold. However, this may often not be the case in observational studies. Generalizing the work to other causal inference frameworks, for example by considering instrumental variables, would be of interest. Exploring the extension of this work to test for conditional treatment effects could also be a natural direction to follow. Lastly, we expect state-of-the-art regressors for kernel mean embeddings, such as distributional random forests ( ´ Cevid et al., 2022), to find outstanding use in our estimator. We envisage that this may motivate the development of flexible mean embedding estimators designed for complex data, such as image or text.

References

Abadie, A. (2002). Bootstrap tests for distributional treatment effects in instrumental variable models. Journal of the American Statistical Association , 97(457):284-292.

Bosq, D. (2000). Linear processes in function spaces: theory and applications , volume 149. Springer Science & Business Media.

Chernozhukov, V., Fernández-Val, I., and Melly, B. (2013). Inference on counterfactual distributions. Econometrica , 81(6):2205-2268.

Díaz, I. (2017). Efficient estimation of quantiles in missing data models. Journal of Statistical Planning and Inference , 190:39-51.

Fawkes, J., Hu, R., Evans, R. J., and Sejdinovic, D. (2022). Doubly robust kernel statistics for testing distributional treatment effects even under one sided overlap. arXiv preprint arXiv:2212.04922 .

Funk, M. J., Westreich, D., Wiesen, C., Stürmer, T., Brookhart, M. A., and Davidian, M. (2011). Doubly robust estimation of causal effects. American Journal of Epidemiology , 173(7):761-767.

Gretton, A., Borgwardt, K. M., Rasch, M. J., Schölkopf, B., and Smola, A. (2012). A kernel two-sample test. The Journal of Machine Learning Research , 13(1):723-773.

Hahn, P. R., Murray, J. S., and Carvalho, C. M. (2020). Bayesian regression tree models for causal inference: Regularization, confounding, and heterogeneous effects (with discussion). Bayesian Analysis , 15(3):965-1056.

Hein, M. and Bousquet, O. (2004). Kernels, associated structures and generalizations.

Hernan, M. and Robins, J. (2020). Causal inference: What if. boca raton: Chapman & hall/crc.

Hill, J. L. (2011). Bayesian nonparametric modeling for causal inference. Journal of Computational and Graphical Statistics , 20(1):217-240.

Horváth, L. and Kokoszka, P. (2012). Inference for functional data with applications , volume 200. Springer Science & Business Media.

Hsing, T. and Eubank, R. (2015). Theoretical foundations of functional data analysis, with an introduction to linear operators , volume 997. John Wiley & Sons.

Imbens, G. W. (2004). Nonparametric estimation of average treatment effects under exogeneity: A review. Review of Economics and Statistics , 86(1):4-29.

Johansson, F., Shalit, U., and Sontag, D. (2016). Learning representations for counterfactual inference. In International Conference on Machine Learning , pages 3020-3029. PMLR.

Kang, J. D. Y. and Schafer, J. L. (2007). Demystifying Double Robustness: A Comparison of Alternative Strategies for Estimating a Population Mean from Incomplete Data. Statistical Science , 22(4):523 - 539.

Kennedy, E. H. (2019). Nonparametric causal effects based on incremental propensity score interventions. Journal of the American Statistical Association , 114(526):645-656.

Kennedy, E. H. (2022). Semiparametric doubly robust targeted double machine learning: a review. arXiv preprint arXiv:2203.06469 .

Kennedy, E. H., Balakrishnan, S., and G'Sell, M. (2020). Sharp instruments for classifying compliers and generalizing causal effects. The Annals of Statistics , 48(4):2008-2030.

Kennedy, E. H., Balakrishnan, S., and Wasserman, L. (2021). Semiparametric counterfactual density estimation. arXiv preprint arXiv:2102.12034 .

Kim, I. and Ramdas, A. (2023). Dimension-agnostic inference using cross U-statistics. Bernoulli .

Kim, K., Kim, J., and Kennedy, E. H. (2018). Causal effects based on distributional distances. arXiv preprint arXiv:1806.02935 .

Landmesser, J. M. (2016). Decomposition of differences in income distributions using quantile regression. Statistics in Transition new series , 17(2):331-348.

Li, L., Tchetgen, E. T., van der Vaart, A., and Robins, J. M. (2011). Higher order inference on a treatment effect under low regularity conditions. Statistics & Probability Letters , 81(7):821-828.

Louizos, C., Shalit, U., Mooij, J. M., Sontag, D., Zemel, R., and Welling, M. (2017). Causal effect inference with deep latent-variable models. Advances in Neural Information Processing Systems , 30.

Luedtke, A. R., Diaz, I., and van der Laan, M. J. (2015). The statistics of sensitivity analyses.

Muandet, K., Kanagawa, M., Saengkyongam, S., and Marukatat, S. (2021). Counterfactual mean embeddings. The Journal of Machine Learning Research , 22(1):7322-7392.

Muñoz, I. D. and Van Der Laan, M. (2012). Population intervention causal effects based on stochastic interventions. Biometrics , 68(2):541-549.

Park, J. and Muandet, K. (2023). Towards empirical process theory for vector-valued functions: Metric entropy of smooth function classes. In International Conference on Algorithmic Learning Theory , pages 1216-1260. PMLR.

Park, J., Shalit, U., Schölkopf, B., and Muandet, K. (2021). Conditional distributional treatment effect with kernel conditional mean embeddings and U-statistic regression. In International Conference on Machine Learning , pages 8401-8412. PMLR.

Richardson, A., Hudgens, M. G., Gilbert, P. B., and Fine, J. P. (2014). Nonparametric bounds and sensitivity analysis of treatment effects. Statistical Science: A Review Journal of the Institute of Mathematical Statistics , 29(4):596.

Robins, J., Li, L., Tchetgen, E., van der Vaart, A., et al. (2008). Higher order influence functions and minimax estimation of nonlinear functionals. Probability and Statistics: Essays in Honor of David A. Freedman , 2:335-421.

Robins, J. M. and Rotnitzky, A. (2001). Comments on: Inference for semiparametric models: Some questions and an answer. Statistica Sinica .

Rubin, D. B. (2005). Causal inference using potential outcomes: Design, modeling, decisions. Journal of the American Statistical Association , 100(469):322-331.

Shalit, U., Johansson, F. D., and Sontag, D. (2017). Estimating individual treatment effect: generalization bounds and algorithms. In International Conference on Machine Learning , pages 3076-3085. PMLR.

Shekhar, S., Kim, I., and Ramdas, A. (2022). A permutation-free kernel two-sample test. In Advances in Neural Information Processing Systems .

Shen, S. (2019). Estimation and inference of distributional partial effects: theory and application. Journal of Business & Economic Statistics , 37(1):54-66.

Singh, R., Sahani, M., and Gretton, A. (2019). Kernel instrumental variable regression. Advances in Neural Information Processing Systems , 32.

Singh, R., Xu, L., and Gretton, A. (2020). Kernel methods for causal functions: Dose, heterogeneous, and incremental response curves. arXiv preprint arXiv:2010.04855 .

Singh, R., Xu, L., and Gretton, A. (2021). Kernel methods for multistage causal inference: Mediation analysis and dynamic treatment effects. arXiv preprint arXiv:2111.03950 .

Song, L., Huang, J., Smola, A., and Fukumizu, K. (2009). Hilbert space embeddings of conditional distributions with applications to dynamical systems. In Proceedings of the 26th Annual International Conference on Machine Learning , pages 961-968.

Strassen, V. et al. (1969). Gaussian elimination is not optimal. Numerische mathematik , 13(4):354356.

Van der Vaart, A. W. (2000). Asymptotic statistics , volume 3. Cambridge University Press.

Wager, S. and Athey, S. (2018). Estimation and inference of heterogeneous treatment effects using random forests. Journal of the American Statistical Association , 113(523):1228-1242.

Westling, T. and Carone, M. (2020). A unified study of nonparametric inference for monotone functions. The Annals of Statistics , 48(2):1001.

Zhu, Y., Gultchin, L., Gretton, A., Kusner, M. J., and Silva, R. (2022). Causal inference with treatment measurement error: a nonparametric instrumental variable approach. volume 180 of Proceedings of Machine Learning Research , pages 2414-2424. PMLR.

´ Cevid, D., Michel, L., Näf, J., Meinshausen, N., and Bühlmann, P. (2022). Distributional random forests: Heterogeneity adjustment and multivariate distributional regression. Journal of Machine Learning Research , 23(333):1-79.

A Comparison between KTE and AIPW-xKTE in the experimental setting

In Section 5, we exhibited the performance of the proposed AIPW-xKTE test in the observational setting. We emphasize that the proposed AIPW-xKTE test is designed to target the observational setting, where the KTE may not be used given that the propensity scores are unknown. In such a setting, the double-robustness of the proposed test is fully exploited. However, the proposed test may also be used in the experimental setting for its computational efficiency: the AIPW-xKTE test avoids permutations, which makes it more computationally efficient than the KTE.

Consequently, we explore in this appendix the performance of the AIPW-xKTE in the experimental setting. We consider a usual experimental design where n 2 units are treated and n 2 units are not treated so that π ( X ) = 1 2 . We compare the power of the proposed AIPW-xKTE with respect to the KTE test. Further, we include an IPW version of the proposed test (by taking β 0 = β 1 = 0 , see comments in Section 4) in order to elucidate the interest in considering the doubly robust version. We refer the reader to Appendix B for an exhaustive description of the experiments.

Figure 4 exhibits the performance of the tests in Scenario II, Scenario III, and Scenario IV. We note that IPW-xKTE has less power than the KTE, which is its analogous permutation-based version. This is due to the controlled loss in power of the cross U-statistic approach following the comments in Section 4. However, we see that the AIPW-xKTE competes with the KTE, despite showing slightly less power. The loss in power inherited by the permutation-free approach is compensated by the gain in power due to using an AIPW estimator. Furthermore, AIPW-xKTE shows a drastic improvement in computational costs with respect to KTE, as exhibited in Table 2. Lastly, we see that AIPW-xKTE clearly outperforms IPW-xKTE, which illustrates the significant benefits of the doubly robust approach.

(A)

(B)

(C)

0.5

Draws from AIPW-xKTE

3

0.14

AIPW-xKTE Experimental setting

Standard normal pdf

IPW-xKTE Experimental setting

0.4

2

rate

0.12

Values

Density

0.3

1

positive

0.10

Ordered

0

0.08

T

0.2

-1

False

0.06

0.04

VIM

0.1

-2

0.02

IIIIII'YI

3

0.0

0.00

-2

0

2

-2

0

2

200

400

600

800

1000

AIPW-XKTE

Theoreticalquantiles

n

(I1)

(II)

(IV)

1.0

1.0

1.0

rate

0.8

0.8

0.8

positive

0.6

0.6

★

0.6

True

0.4

0.4

0.4

AIPW-XKTE

AIPW-xKTE

AIPW-xKTE

0.2

IPW-xKTE

0.2

IPW-xKTE

0.2

IPW-xKTE

KTE

KTE

KTE

0.0

0.0

0.0

100

150

200

250

300

350

100

150

200

250

300

350

100

150

200

250

300

350

n

n

n

B Experiments

We present in this appendix a comprehensive display of the simulations conducted. Subsection B.1 includes an exhaustive description of the experiments introduced in Section 5 and Appendix A. Additional simulations and results may be found in Subsection B.2.

B.1 Exhaustive description of experiments

We assume that we observe ( x i , a i , y i ) n i =1 ∼ ( X,A,Y ) and that (causal inference assumptions) consistency, no unmeasured confounding, and overlap hold. All the tests are considered at a 0.05 level. Both synthetic data and real data are evaluated.

Synthetic data. All the data (covariates, treatments and responses) is artificially generated. We define

such that ϵ 0 , ϵ 1 ∼ N (0 , 0 . 5) are independent noises, X ∼ N (0 , I 5 ) and β = [0 . 1 , 0 . 2 , 0 . 3 , 0 . 4 , 0 . 5] T . We set b = 0 and b = 2 for Scenario I and Scenario II respectively. For Scenario III, we set b = 2 Z -1 , where Z is an independent Bernoulli random variable Z ∼ Bernoulli (0 . 5) . In Scenario IV, b ∼ Uniform ( -2 , 2) .

In the experimental setting, we consider a usual experimental design where n 2 units are treated and n 2 units are not treated such that π ( X ) = 1 2 almost surely. In the observational setting, we define π ( X ) = s ( α T X + α 0 ) , where s ( z ) = 1 / (1 + exp ( -z )) (sigmoid function), α 0 = 0 . 05 and α = [0 . 05 , 0 . 04 , 0 . 03 , 0 . 02 , 0 . 01] T . In such setting, we estimate ˆ π using an L2 regularized logistic regression with the regularization term set to 1e-6. For AIPW-xKTE, the mean embedding regressions are fitted as conditional mean embeddings.

In the experimental setting, we consider the tests IPW-xKTE, AIPW-xKTE, and KTE (Muandet et al., 2021). In the observational setting, we consider IPW-xKTE, AIPW-xKTE and permutationbased tests based on Causal Forests (Wager and Athey, 2018), Bayesian Additive Regression Trees (BART) (Hahn et al., 2020), and a linear regression based AIPW estimator (Baseline-AIPW). For the latter three, we recalculate the respective statistics for every permutation and reject the null if the original statistic is above the 0.95 empirical quantile. For all scenarios and settings, we consider 500

Response against covariates

(t)

(I)

(II)

(IV)

10.0

T=0

0=1

T=0

=0

[=1

7.5

5.0

2.5

0.0

2

2.5

5.0

-7.5

0

2

0

2

0

2

10.0

0

2

X(1)

X(1)

X(1)

X(1)

Histograms

(1)

(11)

(I1)

(IV)

T=0

35

35

T=0

T=0

40

T=1

30

T=1

OE

T=1

25

T=1

25

25

20

Count

20

20

20

15

15

10

10

10

0

2

5

-10

-5

0

5

10

Y

simulations for each n ∈ { 100 , 150 , 200 , 250 , 300 , 350 } . We exhibit an illustration of the synthetic data from one the simulations carried in Figure 5.

Real data. We use data obtained from the Infant Health and Development Program (IHDP) and compiled by Hill (2011), in which the covariates come from a randomized experiment studying the effects of specialist home visits on cognitive test scores. The propensity scores are unknown and hence they must be estimated. We work with the following 18 variables of the covariate set: ['bw','momage','nnhealth','birth.o','parity','moreprem','cigs','alcohol','ppvt.imp', 'bwg','female','mlt.birt','b.marry','livwho','language','whenpren','drugs','othstudy'], where the first nine are continuous and the last nine are discrete; we refer to Hill (2011) for a detailed presentation of the data set. We eliminate the rows in which there is a NaN for any of the variables considered, finishing with 908 observations, out of which 347 were treated. Further, we standarize the data such that every continuous variable has mean 0 and variance 1. We illustrate the data obtained after preprocessing in Figure 6.

We consider five scenarios with the IHDP data. For Scenarios I, II, III and IV, we generate the response variables similarly to the previous experiments:

where ϵ 0 , ϵ 1 ∼ N (0 , 0 . 5) are independent noises, and β = [1 , . . . , 1] T . We set b = 0 and b = 1 for Scenario I and Scenario II respectively. For scenario III, we set b = 2(2 Z -1) , where Z is an independent Bernoulli random variable Z ∼ Bernoulli (0 . 5) . In scenario IV, b ∼ Uniform ( -4 , 4) . In Scenario V, we take the IQ test (Stanford Binet) score, variable 'iqsb.36' of the data set, measured at the end of the intervention (age 3) as our response variable. For Scenario VI, we calculate the average treatment effect in Scenario V using Causal Forests (obtaining 0 . 0035 i.e. a positive shift) and subtract it to the IQ test score of those who are treated, thus obtaining a distribution of the IQ test scores with zero average treatment effect. We illustrate the data of Scenarios V and VI in Figure 6. For each of the scenarios, we consider the tests AIPW-xKTE, IPW-xKTE, Causal Forests, BART and Baseline-AIPW on 500 bootstrapped subsets to estimate the rejection rates. For AIPW-xKTE and IPW-xKTE, we estimate ˆ π using a regularized logistic regression with the regularization term set to 1e-6. For AIPW-xKTE, the mean embedding regressions are fitted as conditional mean embeddings.

We sample split for estimating the conditional mean embeddings, but for the propensity scores, we use the whole training data. While we stated Theorem 4.1 imposing condition (iv) or condition (v) on ˆ ϕ ( r ) (common practice in causal inference for ease of presentation, where one usually imposes

Response againstfirst principal component

(V)

(VI)

0.055

=0

T=0

0.050

0.05

0.045

0.040

0.04

Y

0.035

00

0.030

0.025

0.02

0.020

0.015

6

8

PC1

PC1

Histograms

(V)

(VI)

100

T=0

100

T=0

T=1

T=1

80

80

Count

60

60

40

40

20

20

(A)

(B)

(C)

0.40

Standard normal pdf

Draws from AIPW-xKTE

3

0.14

0.35

2

0.12

0.30

Values

0.10

1

positive

Densit

0.25

0.20

Ordered

0

0.08

0.15

-1

False

0.06

0.10

LL

0.04

2

0.05

0.02

0.00

3

0.00

-2

0

2

-2

0

2

200

400

600

800

1000

AIPW-XKTE

Theoreticalquantiles

n

sample splitting or a Donsker condition), Theorem 4.1 holds as long as sample splitting is conducted for those estimators ˆ π ( r ) , ˆ β ( r ) a which may overfit (analogous case in the standard doubly robust estimator). In our case, the propensity is modeled by L2 regularized logistic regression, which is simple and cannot overfit when the dimension is fixed and n →∞ .

All the kernels considered on Y for AIPW-xKTE, IPW-xKTE, and KTE are RBF i.e. k y ( y 1 , y 2 ) = exp ( ν y | y 1 -y 2 | 2 ) , with scale parameter ν y chosen by the median heuristic. For AIPW-xKTE, the mean embedding regressions are fitted as conditional mean embeddings, and we conduct sample splitting in order to train them. The kernel considered for such conditional mean embeddings on X is also RBF k x ( x 1 , x 2 ) = exp ( ν x | x 1 -x 2 | 2 ) with scale parameter ν x chosen by the median heuristic as well. The regularization parameter λ was taken to be equal to ν x .

B.2 Further experiments

In this subsection, we investigate the performance of the tests in non-linear settings. We extend the simulations with synthetic data presented in Subsection B.1, defining the potential outcomes

where β , b , ϵ 0 and ϵ 1 are defined as described in Subsection B.1. The different scenarios and the remaining parameters are also defined as in Subsection B.1.

Figure 7 displays the behaviour of AIPW-xKTE in Scenario I (under the null). Analogously to the linear setting, we note that AIPW-xKTE presents a Gaussian behaviour under the null, which leads to a well-calibrated test. The performance of the tests in Scenarios II, III and IV is displayed in Figure 8. In this non-linear case, AIPW-xKTE proves even more competitive in Scenario II, while retaining power in Scenarios III and IV. In short, the test proves valid (as expected) in the non-linear case, and the comments from Section 5 equally apply.

(I1)

(I)

(IV)

1.0

1.0

1.0

rate

0.8

0.8

0.8

positive

0.6

0.6

-

0.6

0.4

0.4

0.4

True

AIPW-xKTE

AIPW-xKTE

AIPW-xKTE

Baseline-AIPW

Baseline-AIPW

Baseline-AIPW

0.2

0.2

0.2

BART

BART

BART

CausalForest

CausalForest

CausalForest

0.0

0.0

0.0

100

150

200

250

300

350

100

150

200

250

300

350

100

150

200

250

300

350

n

n

n

C Proofs

Wepresent in this appendix the proof of the main result of the paper, Theorem 4.1. For this purpose, we introduce the notation used in Subsection C.1 followed by the exposition of previously known results in Subsection C.2. We present a sequence of extensions of such results to the vector-valued scenario in Subsection C.3, which will be used to prove the main theorem of the paper in Subsection C.4.

C.1 Notation

We use standard big-oh and little-oh notation, i.e., X n = O P ( r n ) means X n /r n is bounded in probability and X n = o P ( r n ) means X n /r n p → 0 . We use P n for sample averages i.e. P n ( f ) = P n { f ( Z ) } = 1 n ∑ i f ( Z i ) . For a potentially random function ̂ f , we denote P ( ̂ f ) = P { ̂ f ( Z ) } = ∫ ̂ f ( z ) d P ( z ) , and ∥ ̂ f ∥ 2 = ∫ ̂ f ( z ) 2 d P ( z ) represents the squared L 2 ( P ) norm.

Given a Hilbert space H and a H -valued ω ( z ) ∈ H , we denote its norm in the Hilbert space by ∥ ω ( z ) ∥ H . We let P n ( ω ) = P n { ω ( Z ) } = 1 n ∑ i ω ( Z i ) and P ( ̂ ω ) = P { ̂ ω ( Z ) } = ∫ ̂ ω ( z ) d P ( z ) (Bochner integral). Furthermore, ∥ ̂ ω ∥ 2 = ∫ ∥ ̂ ω ( z ) ∥ 2 H d P ( z ) denotes the squared L 2 ( P ) norm of the H -valued function norm, and ∥ · ∥ HS the Hilbert-Schmidt norm of an operator.

C.2 Previously known results

We start by introducing a generalization of the formula that relates the mean, variance and second moment of a real-valued random variable to H -valued random variables.

Lemma C.1 (Hsing and Eubank (2015, Theorem 7.2.2)) . Let ω ∈ H be a random variable such that E ∥ ω ∥ 2 H < ∞ . Then

Furthermore, one may retrieve the expectation of the product of projections of the H -valued random variables through the covariance operator, as exhibited in the following lemma.

Lemma C.2. Let ω ∈ H be a random variable such that E [ ω ] = 0 and E ∥ ω ∥ 2 H < ∞ . Then

where C = E [ ω ⊗ ω ] is the covariance operator.

Next, we present the generalization of the central limit theorem to H -valued random variables.

Theorem C.3 (Bosq (2000, Theorem 2.7)) . Let ( ω i , i ≥ 1) be a sequence of iid random variables in a separable Hilbert space H . If m = E [ ω 1 ] and E [ ∥ ω 1 ∥ 2 ] < ∞ , then

where ξ is a Gaussian random element with mean zero and covariance operator C = E [ ω 1 ⊗ ω 1 ] i.e.

Moreover, Gaussian H -valued random variables admit the following expansion in the orthonormal system defined by its covariance operator.

Lemma C.4 (Horváth and Kokoszka (2012, Equation 2.8)) . A normally distributed function ξ in a separable Hilbert space with a covariance operator C admits the expansion

where N j are independent standard normal distributions, ( λ j , j ≥ 1) is the sequence of eigenvalues of the covariance operator C , and ( v j , j ≥ 1) is a sequence of orthonormal eigenfunctions of the covariance operator C .

We now present a result concerning the asymptotic behaviour of an empirical process given that the considered random functions belong to a Donsker class. For a full discussion on Donsker classes and empirical processes, we refer the reader to Chapter 18 and Chapter 19 of Van der Vaart (2000).

Lemma C.5 (Van der Vaart (2000, Lemma 19.24)) . Suppose that F is a P -Donsker class of measurable (real-valued) functions and ˆ f n is a sequence of random functions that take their values in F such that ∥ ˆ f n -f ∥ 2 p → 0 for some f ∈ L 2 ( P ) . Then

However, the concept of Donsker class is fairly restrictive. In fact, the intuition behind the nice behaviour exhibited by Donsker classes in empirical process is their inability to overfit, due to their constricted flexibility. For instance, commonly used estimators such as random forests do not belong to a Donsker class. In order to circumvent this problem, sample splitting proves very useful. The following lemma exhibits the power of sample splitting regardless of the flexibility of the estimators.

Lemma C.6 (Kennedy et al. (2020, Lemma 2)) . Let ̂ f ( z ) be a function estimated from a sample Z N = ( Z n +1 , . . . , Z N ) , and let P n denote the empirical measure over ( Z 1 , . . . , Z n ) , which is independent of Z N . Then

Proof. First note that, conditional on Z N , the term in question has mean zero since

The conditional variance is

Therefore by iterated expectation and Chebyshev's inequality we have

Thus for any ϵ > 0 we can pick t = 1 / √ ϵ so that the probability above is no more than ϵ , which yields the result.

These two lemmas allow for proving Theorem 3.1. Recall that this theorem exhibits the empirical mean-like behaviour of doubly robust estimators. This will be key when proving the main result of the paper.

Proof of Theorem 3.1. We have that ˆ ψ DR -ψ = Z ∗ + T 1 + T 2 , where

The consistency of f in L 2 norm, together with either the Donsker condition (Lemma C.5) or sample splitting (Lemma C.6), ensures that T 1 = o P (1 / √ n ) . For T 2 = P ( ˆ f -f ) and f = f 1 -f 0 , we have

Same logic applies for | P ( ˆ f 0 -f 0 ) | ≤ 1 1 -ϵ ∥ π -ˆ π ∥∥ θ 0 -ˆ θ 0 ∥ . Therefore T 2 = o P (1 / √ n ) and the result follows.

We have exhibited the proofs of Lemma C.6 and Theorem 3.1 for sake of completeness. We will extend and prove the analogous results in the H -valued setting in Subsection C.3, which will be needed in demonstrating the main result of the paper. We refer the reader to Kennedy (2022) for a full discussion and exhibition of the aforementioned causal inference results. Lastly, we exhibit the definition of a Glivenko-Cantelli class, which is used in Theorem 4.1.

Definition C.7 (Glivenko-Cantelli) . We say that a class of integrable real-valued functions F is a Glivenko-Cantelli class for P if

converges to zero in probability as n →∞ .

C.3 Extension of the results to (infinite-dimensional) vector-valued outcomes

We introduce the extension of Theorem 3.1 to the H -valued outcome setting. For such goal, we also generalize Lemma C.6 to the H -valued scenario and we comment on asymptotically equicontinuous empirical processes Park and Muandet (2023), which will substitute the Donsker class condition in the generalized version of Theorem 3.1.

We start by presenting the generalization of Lemma C.6 to the functional setting.

Lemma C.8 (Extension of Lemma C.6 to H -valued outcomes) . Let H be a Hilbert space. Let ̂ ω ( z ) be a function estimated from a sample Z N = ( Z n +1 , . . . , Z N ) such that ̂ ω ( z ) ∈ H for all z , and let P n denote the empirical measure over ( Z 1 , . . . , Z n ) , which is independent of Z N . Then

Proof. First note that, conditional on Z N , the term in question has mean zero since

The conditional expectation of the squared norm is

̸

where (i) is obtained given independence of Z i , Z j when i = j and E { ( P n -P )( ̂ ω -ω ) ∣ ∣ ∣ Z N } = 0 , and (ii) by Lemma C.1. Therefore by iterated expectation and Markov's inequality we have

Thus for any ϵ > 0 we can pick t = 1 / √ ϵ so that the probability above is no more than ϵ , which yields the result.

Note that the former result implies that, if ∥ ̂ ω -ω ∥ p → 0 , then

given that we conduct sample splitting. However, sample splitting might not be necessary if our estimators are not flexible enough. Splitting the data might result in a loss in power, hence we are interested in stating another sufficient condition that will lead the empirical term ( P n -P )( ̂ ω -ω ) to be asymptotically negligible without conducting sample splitting. A direct extension of the Donsker class condition stated in Theorem 3.1 to H -valued outcomes is not possible, given that bounding entropies usually make explicit use of the fact that R is totally ordered (Park and Muandet, 2023). However, Park and Muandet (2023) defines asymptotic equicontinuity, which is precisely the condition which we will require in the extension of Theorem 3.1 to the functional setting.

Definition C.9 (Asymptotic equicontinuity) . We say that the empirical process { ν n ( ω ) = √ n ( P n -P ) ω : ω ∈ G} with values in H and indexed by G is asymptotic equicontinuous at ω 0 ∈ G if, for every sequence { ˆ ω n } ⊂ G with ∥ ˆ ω n -ω 0 ∥ p → 0 , we have

Note that (7) is equivalent to ( P n -P )( ̂ ω -ω ) = o P ( 1 √ n ) . Park and Muandet (2023) gives sufficient conditions for asymptotic equicontinuity to hold, as well as stating examples of classes that attain asymptotic equicontinuity. We are now ready to present the extension of Theorem 3.1 to the H -valued setting.

Theorem C.10 (Extension of Theorem 3.1 to H -valued outcomes) . Let ϕ ( z ) = { a π ( x ) -1 -a 1 -π ( x ) }{ y -β a ( x ) } + β 1 ( x ) -β 0 ( x ) , where y and β are H -valued, so that Ψ = E { ϕ ( Z ) } is the average treatment effect. Suppose that

ˆ ϕ is constructed from an independent sample or the respective empirical process is asymptotically equicontinuous at ϕ .

|| ˆ ϕ -ϕ || = o P (1) .

Also assume that P (ˆ π ∈ [ ϵ, 1 -ϵ ]) = 1 . Then if || ˆ π -π || ∑ a || ˆ β a -β a || = o P ( 1 √ n ) it follows that

where ∥ ξ n ∥ H = o P ( 1 √ n ) . Hence, it is root-n consistent.

Proof. We have that ˆ Ψ DR -Ψ = Z ∗ + T 1 + T 2 , where

The consistency of ϕ in L 2 norm, together with either asymptotic equicontinuity (Definition C.9) or sample splitting (Lemma C.8), ensures that ∥ T 1 ∥ H = o P (1 / √ n ) . For T 2 = P ( ˆ ϕ -ϕ ) and

ϕ = ϕ 1 -ϕ 0 , we have

Same logic applies for P ( ˆ ϕ 0 -ϕ 0 ) ≤ 1 1 -ϵ ∥ π -ˆ π ∥∥ β 0 -ˆ β 0 ∥ . Therefore ∥ T 2 ∥ H = o P (1 / √ n ) and the result follows.

C.4 Proof of Theorem 4.1

Assume that we have access to a sample ( X i , A i , Y i ) 2 n i =1 ∼ ( X,A,Y ) . We denote D 1 = ( X i , A i , Y i ) n i =1 , D 2 = ( X j , A j , Y j ) 2 n j = n +1 . Further,

where ¯ f h and S 2 h are the empirical mean and variance respectively:

In particular, note the indices 1 to n in the definition of ¯ f h , and the indices n +1 to 2 n when defining f h . Similarly, we denote the analogous version for estimated embeddings ˆ ϕ as follows:

For of ease of notation, we also define

Lastly, note that the following equalities hold by definition:

Proof of Theorem 4.1. We will prove the theorem in four steps:

We first prove that n ¯ f † h = n ¯ f h + o P (1) .

We then show that nS † h 2 = nS 2 h + o P (1) .

Further, we prove that 1 E [ n { f h ( Z ) } 2 |D 2 ] = O P (1) .

We conclude that T † h d → N (0 , 1) .

It now remains to prove each of the four steps.

Details of step 1. Theorem C.10 may be applied for both ˆ ϕ (0) , ˆ ϕ (1) because its conditions are fulfilled. We obtain that

where ∥ √ nξ (1) n ∥ H , ∥ √ nξ (2) n ∥ H = o P (1) . Rewriting the expression, we note that

where ∥ χ (1) n ∥ H , ∥ χ (2) n ∥ H = o P (1) . Based on Theorem C.3,

where C = E [ ϕ ( Z ) ⊗ ϕ ( Z )] , which implies max( ∥ τ 1 ∥ 2 H , ∥ τ 2 ∥ 2 H ) = O P (1) . Consequently,

Given that |⟨ τ 1 , χ (2) n ⟩ + ⟨ χ (1) n , τ 2 ⟩ + ⟨ χ (1) n , χ (2) n ⟩| is upper bounded by ∥ τ 1 ∥ H ∥ χ (2) n ∥ H + ∥ χ (1) n ∥ H ∥ τ 2 ∥ H + ∥ χ (1) n ∥ H ∥ χ (2) n ∥ H (by the triangle inequality and Cauchy-Schwarz inequality), which is O P (1) o P (1) + o P (1) O P (1) + o P (1) o P (1) = o P (1) , we deduce that

Details of step 2. We claim that

as well as

If both (9) and (10) hold, we obtain

hence thus concluding step 2.

In order to prove (9), we denote ϵ i = ˆ ϕ (1) ( Z i ) -ϕ ( Z i ) and consider

where (i) is obtained by the triangle inequality, and (ii), (iii), (iv) are obtained by Cauchy-Schwarz inequality. We recall that ∥ τ 2 ∥ 2 H = O P (1) . Moreover, E ∥ ϕ ( Z ) ∥ 2 H ≤ E ∥ ϕ ( Z ) ∥ 4 H < ∞ implies 1 n ∑ n i =1 ∥ ϕ ( Z i ) ∥ 2 H p → E ∥ ϕ ( Z ) ∥ 2 H by the law of large numbers, so 1 n ∑ n i =1 ∥ ϕ ( Z i ) ∥ 2 H = O P (1) .

If ˆ ϕ was constructed independently from D 1 , then

so by Markov's inequality

If ∥ ˆ ϕ (1) ∥ 2 H belongs to a Glivenko-Cantelli class, then ∥ ˆ ϕ (1) -ϕ ∥ 2 H belongs to a Glivenko-Cantelli class. From this, we conclude that ( P -P n ) ∥ ̂ ϕ (1) -ϕ ∥ 2 H = o P (1) , hence

In either case,

Further,

where we obtain (i) by the triangle inequality, (ii) by Cauchy-Schwarz inequality, and (iii) by CauchySchwarz inequality and grouping terms. By the triangle inequality, | 1 n ∑ n i =1 ⟨ ˆ ϕ (1) ( Z i ) , ˆ τ 2 ⟩ 2 -1 n ∑ n i =1 ⟨ ϕ ( Z i ) , τ 2 ⟩ 2 | is upper bounded by

which is = o P (1) + o P (1) = o P (1) , thus (9) holds.

In order to prove (10), we note that (8) implies ( n ¯ f † h ) 2 = ( n ¯ f h + o P (1)) 2 . Expanding the second term we obtain

given that | n ¯ f h | = |⟨ τ 1 , τ 2 ⟩| ≤ ∥ τ 1 ∥ H ∥ τ 2 ∥ H = O P (1) O P (1) = O P (1) . Consequently, ( n ¯ f † h ) 2 = ( n ¯ f h ) 2 + o P (1) , which implies (10) dividing by n in both sides.

Details of step 3. We claim that

To show this, we note that

where

C = E [ ϕ ( Z ) ⊗ ϕ ( Z )] , ( λ j , j ≥ 1) is the sequence of non-negative eigenvalues in decreasing order of the covariance operator C ,

( v j , j ≥ 1) is the sequence of orthonormal eigenfunctions of the covariance operator C (and basis of H as well),

τ 2 = ∑ ∞ i = j β j v j , being ( β j , j ≥ 1) the random coefficients of τ 2 with respect to the basis ( v j , j ≥ 1) ,

and (i) is obtained from Lemma C.2 and the fact that E [ ϕ ( Z )] = 0 , E [ ∥ ϕ ( Z ) ∥ 2 H ] ≤ E [ ∥ ϕ ( Z ) ∥ 4 H ] < ∞ . Based on Lemma C.4,

so by the continuous mapping theorem for random variables in separable Banach spaces (Bosq, 2000, Equation 2.11)

That is, the sum of non-negative eigenvalues is strictly greater than zero, so at least one of them has to be strictly positive. By the non-increasing order of the eigenvalues, this implies that the first one is strictly positive i.e., λ 1 > 0 . For any M > 0 it holds that

Consequently,

Given λ 2 1 > 0 and (13), we also obtain

where F is the cdf of a χ 2 1 distribution. Hence for every ϵ > 0 , there exists m ∈ N such that

Further, lim w → 0 F ( w ) = 0 , so there exists M ∗ > 0 such that F ( 1 M ∗ λ 2 1 ) < ϵ 2 . Consequently, for every ϵ > 0 , there exist M ∗ > 0 and m ∈ N such that

for n ≥ m , which implies for n ≥ m , so

thus concluding (12).

Details of step 4. Given Conditions (i), (ii), and (iii), Kim and Ramdas (2023, Proof of Theorem 4.2) proved that

Combining (8) and (12), we obtain

and based on (14) and Slutsky's theorem,

Combining (11) and (12), we obtain

as well as

Thus based on (15), we obtain

and by the continuous mapping theorem

Combining (16) and (17) and based on Slutsky's theorem, we obtain

Hence

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes Augmented Inverse Propensity Weighted cross Kernel Treatment Test (AIPW-xKTE), which is a doubly robust test with provably valid type-I error based on kernel mean embeddings to test for distributional treatment effect. The paper has one result, Theorem 4.1, showing the asymptotic normality of the proposed test statistic, and demonstrates its performance on synthetic and real datasets.\n\nSTRENGTHS:\nThe paper has one clear goal, i.e. to provide a testing procedure for distributional treatment effect. As the authors mention, distributional treatment effect has been an important topic of research for a while, and their proposal, AIPW-xKTE, has several advantages over the previous methods, most prominently that it has an analytical asymptotic null distribution, circumventing the need for permutation to get the null distribution, as well as being doubly robust. \n\nThe paper is very clearly written, setting out its goal in the backdrop of previous works and carrying out that goal with minimal fuss. The paper only offers one result, Theorem 4.1, but I think its conciseness is its value. I haven't gone through every detail of the proof in the appendix but a scan convinced me of its soundness. The paper was a pleasure to read.\n\nWEAKNESSES:\nPerhaps one thing to count against this paper is its lack of novelty, in that its proposal is not something completely novel, rather it combines two ideas, namely cross U-statistics and augmented inverse probability weighting. However, it combines them to good effect and conducts thorough analysis of it, both theoretically and empirically, and I do not think this should count heavily against the paper.\n\nQUESTIONS:\nThe bottleneck of this procedure would probably be the estimation of kernel conditional mean embeddings, which has $n^3$ complexity? Perhaps it would be worth looking at speeding this up, through approximate kernel ridge regression methods (e.g. Nystrom method proposed in [Grunewalder et al., 2012] or FALKON in [Rudi et al., 2017]).\n\nLIMITATIONS:\nThe conclusion section has listed a few of its limitations and interesting possible directions for future research.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a test of the null hypothesis that a binary treatment has no effect on the the potential outcome distribution. The test combines ideas of kernel mean embedding, double robustness, and cross U-statistics.\n\nSTRENGTHS:\nOriginality\n\n-The connection between kernel embeddings of effect distributions and the cross U-statistic appears to be new. \n\n-The main difference from Shekhar et al. (2022), who combine kernel mean embedding and cross U-statistics, appears to be the connection to double robustness.\n\n-The main difference from Fawkes et al (2022), who combine kernel mean embedding and double robustness, appears to be the connection to cross U-statistics.\n\n-For the kernel embedding of the potential outcome distribution, Muandet et al. (2021) use an IPW-style estimator while Singh et al. (2020) use a regression-style estimator. Similar to Fawkes et al. (2022), this paper combines IPW and regression estimators into a doubly robust AIPW estimator.\n\nQuality - The results are clear and appear to be correct, with some minor comments given below.\n\nClarity - The paper is well written, especially its appendix.\n\nSignificance - Ultimately, this is a paper that combines building blocks that have been partially combined before. The combination is well executed, and contributes to the literature.\n\nWEAKNESSES:\nI will raise the score if these items are addressed.\n\n\n\nStatistical concepts\n\n-The paper advertises efficiency, which when discussing tests, refers to certain statistical properties. However, the efficiency being described is computational by avoiding permutations. The framing should clarify this.\n\n-Asymptotic equicontinuity and Glivenko-Cantelli class are not well explained in the main text. The former is well explained in the appendix, so a pointer would suffice. The latter is not; please provide more explanation of what this condition means and why it is reasonable in this context.\n\n-Some statements about the asymptotic variance are too strong or poorly worded: on line 19 “the asymptotic variance…” and line  164 “the asymptotic variance…” \n\nComparisons\n\n-The references given for average treatment effect are actually for the local average treatment effect on line 21. Please update here and elsewhere.\n\n-The doubly robust kernel mean embedding estimator can be viewed as augmenting IPW (Muandet et al. 2021) with regression (Singh et al. 2020) approaches to kernel mean embeddings of potential outcome distributions, just as AIPW augments IPW with regression approaches to treatment effects. It would be worthwhile to point this out.\n\n-It would be good to see brief comparisons to Shekhar et al. (2022) and Fawkes et al (2022) following Theorem 4.1.\n\nNotation\n\n-Sometimes notation is overloaded, which is unnecessary and a bit confusing. For example, mu refers to a kernel mean embedding, a regression, and something else in the appendix. \n\n-Another notation issue is that the norm for beta is not defined in Theorem 4.1, and the cross fitting is poorly explained compared to Algorithm 1. \n\n-It is not a good notation choice to write k(w,y) when x and y are variables with specific meanings in the paper.\n\n-Finally, replace O(100n^2) with O(Bn^2).\n\nQUESTIONS:\nThe authors write “We were unable to control the type 1 error of the test presented in Fawkes (2022)…” What does this mean? Why not include Fawkes (2022), the most closely related work, in the simulations?\n\nThe authors show computational efficiency, but how about statistical efficiency?\n\nIn inequality (iii) of line 656, shouldn’t there be a 2 on the last term?\n\nI had some issues with the proof of Step 3. Shouldn’t the final expression have lambda_1^2 in (13)?  This correction would continue throughout the proof. At the bottom of page 27, how does the previous display imply lambda_1>0?\n\nLIMITATIONS:\nThere are no issues.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper focuses on studying Augmented Inverse Probability Weighting (IPW) for distributions instead of means. The outline of the paper is as follows:\n\n1. The authors provide motivation for the problem.\n1. They review several tools used to solve the problem, including Maximum Mean Discrepancy, Conditional Mean Embeddings, Kernel Treatment Effect, xMMD, and the asymptotics of AIPW.\n1. The main results for AIPW in Hilbert spaces are presented, and the authors discuss practical details of the proposed test.\n1. Discussion of the experiments.\n\nSTRENGTHS:\nThe paper is technical but clearly written. Authors prove non-trivial (to me) technical results, which convinces me that if I were to utilize AIPW for distributions, I would choose this particular implementation. More broadly, if I needed to test treatment effects on distributions and lacked access to propensity scores, I would opt for this test.\n\nWEAKNESSES:\nThe main limitation of this paper is that I struggle to think of a practical scenario in which I would have an interest in testing differences in the distribution of treatment effects. While the authors briefly mention that this question arises in various applications, none of those applications are utilized in the experiments. I'm uncertain whether investigating the effect of specialist home visits on cognitive test scores, beyond an increase in the mean, is a particularly relevant question to explore.\n\nIt might be more appropriate to compare this method with conditional average treatment effect tests. I can imagine situations where there are heterogeneous treatment effects that, on average, cancel each other out but work in opposing directions within two populations.\n\nedit: see the list attached by authors.\n\nQUESTIONS:\nI'd ask authors to discuss applications in which this test would be useful.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 1, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a test for the treatment effect which also takes distributional changes into account. The test strongly builds upon the recent works ( Kim and Ramdas, 2023) and ( Muandet et al. (2021)). The main novelty arises from extending the test in Kim and Ramdas, 2023 to the setting of treatment effect estimation. In comparison with previous tests (Figure 4) in the literature, the test proposed in this paper is computationally more efficient at a moderate price in power.\n\nSTRENGTHS:\nTesting for treatment effects is an important problem. The proposed test is computationally efficient and appears to be practically useful.\n\nWEAKNESSES:\n1) Figure 2 is misleading to some degree. BART and Causal Forests are estimating the mean of the treatment effect and therefore necessarily fail in scenarios III and IV.\n\n2) I am expecting a comparison against KTE ( Muandet et al. (2021)) and the AIPW extension from  Fawkes et al. (2022)  in the main text. This should come instead of the current  Figure 2. As far as I understand from Figure 4, the main contribution of this paper is not a better test in terms of power but in terms of computational run-time. This does not sufficiently come across in the main text. \n\n3) The contribution of the paper feels fairly limited and more like a straightforward extension of Kim and Ramdas, 2023. The theoretical results appear to me to follow from standard well-established arguments. Can the authors comment more on the technical challenges behind the proofs/ similarities to prior works? \n\n4) While 3) itself is not a reason for a low grade, based on this limitation I would expect a more exhaustive experimental analysis of the method demonstrating the practical usefulness and its limitations on real-world data sets.\n\nQUESTIONS:\n- How does the corresponding plot for Scenario (I) look for the setting in Figure 2 and 4?\n\n- I am willing to increase my score if the authors can address the shortcomings above.\n\nLIMITATIONS:\nSee Weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a statistical test to determine whether the distributions of the two counterfactuals are the same. This goes beyond the well-known average treatment effect, which only tries to understand whether the means of the distributions are the same. The first work in this direction was by (Muandet et al., 2021), who introduced the Kernel Treatment Effect (KTE). However, their test statistic is degenerate, which means that they cannot use the CLT to derive an asymptotic threshold, and they need to resort to a permutations approach in order to compute the threshold. This paper introduces the AIPW-xKTE test, which generalizes KTE by including a plug-in estimator in analogy with AIPW vs IPW, and most importantly by using the same approach as the Cross MMD test from (Kim and Ramdas, 2023), which does yield a statistic with asymptotically normal distribution.\n\nSTRENGTHS:\nThe contribution of the paper is clear, and the authors do a reasonable job at placing it within the literature.\n\nWEAKNESSES:\nThe main weakness is that the contribution is not highly novel, in that the test proposed is basically a combination of the KTE test from (Muandet et al., 2021) and the Cross MMD technique from (Kim and Ramdas, 2023). \n\nThe explanation would be more transparent if some important concepts were clearly defined. See more details in the questions section. There are some more detailed weaknesses that I also point out in the questions section.\n\nQUESTIONS:\n- What is double robustness? It is a relevant concept in the paper, as it is mentioned twice in the contributions part of the introduction, and many more times later on. However, it is never defined.\n\n- Line 23: The plug in estimator is not well defined: what does it actually look like? In lines 222-225, the estimator appears again under a different notation (\\hat{\\beta} instead of \\hat{\\theta}). The authors currently say that “At this time, not so many choices exist for estimators…“ and they cite a work on this. It would be good to give a more detailed explanation of how the estimator \\hat{\\beta} is computed. Similarly, it would be helpful to give more insight on how \\hat{\\pi} is computed.\n\n- Theorem 3.1: What is \\hat{\\pi}? What is \\hat{\\psi}_{DR}? These quantities have not been defined before as far as I can tell.\n\n- Figure 1: Show error bars in subfigure c, to show that the discrepancy from 0.05 can be attributed to a statistical error. The key question that needs to be answered here is: how large does n need to be for the CLT to kick in and for the Type I error guarantee to hold. Without error bars, the current figure does not provide an answer to this question.\n\n- Figure 2: Although not as critical, error bars in this figure would be appreciated too.\n\n- Table 1: Show standard error.\n\n- Comparison with (Muandet et al., 2021): The authors do not show any experimental comparison with the KTE test proposed by (Muandet et al., 2021). They argue that “Due to the fact that the KTE (Muandet et al., 2021) may not be used in the observational setting, where the propensity scores are not known, there is no natural benchmark for the proposed test.” Since AIPW-xKTE makes use of estimators of propensity scores, it seems natural to me to compare it with KTE using the same propensity score estimators. Is there a reason not to do this?\n\nLIMITATIONS:\nNo limitations", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-14"}