An Efficient Doubly-Robust Test for the Kernel Treatment Effect
Authors: Diego Martinez-Taboada
Aaditya Ramdas
Edward Kennedy
Venue: NeurIPS 2023 poster (2023)
Keywords: kernel treatment effect
causal inference
maximum mean discrepancy
TL;DR: We propose a new kernel-based test for distributional effects of the treatment.

Abstract: The average treatment effect, which is the difference in expectation of the counterfactuals, is probably the most popular target effect in causal inference with binary treatments. However, treatments may have effects beyond the mean, for instance decreasing or increasing the variance. We propose a new kernel-based test for distributional effects of the treatment. It is, to the best of our knowledge, the first kernel-based, doubly-robust test with provably valid type-I error. Furthermore, our proposed algorithm is computationally efficient, avoiding the use of permutations.

An Efficient Doubly-Robust Test for the Kernel Treatment Effect
Diego Martinez-Taboada
Aaditya Ramdas
Department of Statistics and Data Science Carnegie Mellon University Pittsburgh, PA 15213
Department of Statistics and Data Science
Machine Learning Department Carnegie Mellon University Pittsburgh, PA 15213 aramdas@stat. cmu. edu
diegomar@andrev. cmu. edu
Edward H. Kennedy
Department of Statistics and Data Science Camegie Mellon University Pittsburgh, PA 15213 edvard@stat.c cmu. edu
Abstract
The average treatment effect, which the difference in expectation of the counter- factuals. is probably the most popular target effect in causal inference with binary treatments. However, treatments may have effects beyond the mean, for instance decreasing increasing the variance. We propose new kernel-based test for distributional effects of the treatment. It is, the best of our knowledge, the first kernel- based, doubly-robust test with provably valid type-I error. Furthermore, our proposed algorithm computationally efficient, avoiding the use permutations.
Introduction
In the context of causal inference, potential outcomes (Rubin) 2005) are widely used to address counterfactual questions (e.g. what would have happened had some intervention been performed?). This framework considers
(X,A,Y)~P,
where X and Y represent the covariates and outcome respectively, and A {0,1) is binary treatment. Furthermore, the following three conditions are assumed:
i) (Consistency) Y AYY + (1 A)Y," (where Yf Yo are the potential outcomes).
ii) (No unmeasured confounding) Y, Yf A X.
iii) (Overlap) For some 0, we have π (X) == PAI
< PA
almost surely.
Such assumptions allow for identification of causal target parameters. For instance, may be interested in the average variance- variance-weighted treatment effects (Robins et al. 2008 [i et al.) (011). stochastic intervention effects (Muñoz and Van Der Laan 2012 Kennedy] 2019). treatment effect bounds (Richardson et al. Luedtke et al. 2015). However, most of the literature focuses on estimation and inference the average treatment effect (Imbens 2004 Hernan and Robins 2020).
defined as the difference in expectation of the potential outcomes
= Y Y.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Given (i) AY), there are three widespread estimators of vb. First, the plug-in in (PI) estimator:
1 N
-
-
where 01 (x),00(X) estimate E[ 1o X]. Second, the inverse propensity weighting (IPW)
estimator:
Y;A; Y:(1 x.)1-A)))
V1Pw
N
A(X;) 1 - *(X,)
where # (X) estimates the propensity scores P[A 1|x]. Third, the so called Augmented Inverse Propensity Weighted (AIPW) estimator:
APPPW
1 N
Σ
N
((())))-o(xx) (x())) +
A
1 --A,,)) 1- # (X.)
((,)
Under certain conditions (e.g., consistent nuisance estimation at n 11/4 rates), the asymptotic mean squared error of the AIPW estimator is smaller than that of the IPW and PI estimator, and minimax optimal in a local asymptotic sense (Kennedy) 2022). hence has become increasingly popular in the last decade. The AIPW estimator is often referred to as the doubly-robust estimator. We highlight that double-robustness is an intriguing property of an estimator that makes use of two models, in which the estimator is consistent even if only one of the two models is well-specified and the other may be misspecified; we refer the reader to Kang and Schafer (2007) for a discussion on doubly-robust procedures.
However, the treatment might have effects beyond the mean, for instance in the variance skewness of the potential outcome. The average treatment effect will prove insufficient in this case. Consequently, one may be interested in testing whether the treatment has any effect in the distribution of the outcome. This question naturally arises in a variety of applications. For instance, one may want check whether there is any difference between brand-name drug and its generic counterpart, or understand whether a treatment simply shifts the distribution of the outcome (or, in turn, it also affects higher order moments).
In this work, we revisit the problem of testing the null hypothesis Ho Py- = Py; against H1 Pyr # Pyz. We propose distributional treatment effect test based on kernel mean embeddings and the asymptotic behaviour of the AIPW estimator. Our contributions are three-fold:
Up our knowledge, propose the first kernel-based distributional test to allow for doubly-robust estimators with provably valid type-I error.
The proposed distributional treatment effect test is permutation-free, which makes it compu- tationally efficient.
We empirically test the power and size of the proposed test, showing the substantial benefits of the doubly robust approach.
2 Related work
Distributional treatment effects have been addressed from a variety of points of view. Abadie (2002) was one of the first works to propose to test distributional hypothesis attempting to estimate the counterfactual cumulative distribution function (cdf) of the outcome of the treated and untreated.
Chernozhukov et al. (2013) proposed to regress the cdf after splitting the outcome in grid. Further contributions followed with alike edf-based approaches (Landmesser 2016 Diaz 2017).
Other approaches to the problem include focusing on the probability density function (pdf) instead of the edf. Robins and Rotnitzky| (200T) introduced a doubly robust kernel estimator for the counter- factual density, while Westling and Carone (220) proposed to conduct density estimation under monotone density assumption. Kim et al. (2015 8) and Kennedy et al. (2021) suggested to compute IP distances between the pdf of the outcome distribution for the different counterfactuals. Conditional distributional treatment effects have also been addressed, with [Shen (2019) proposing to estimate the edf of the counterfactuals for each value to be conditioned on.

---

On the other hand, kernel methods have recently gained more and more attention in the context of causal inference. Kernel-based two-stage instrumental variable regression was proposed in Singh et al. (2019), while |ingh et al. (220)) presented estimators based on kernel ridge regression for nonparametric causal functions such as dose, heterogeneous, and incremental response curves. Furthermore, (Singh et al. (2021) conducted mediation analysis and dynamic treatment effect using kernel-based regressors as nuisance functiins. Causal inference with treatment measurement error i.c. when the cause is corrupted by error was addressed in Zhu et al. (2022) using kernel mean embeddings to learn the latent characteristic function.
However, kernel mean embeddings for distributional representation were not suggested in the causal inference literature until Muandet et al. (2021) proposed to use an IPW estimator to estimate the average treatment effect on the embedding, which leads to kernel-based distributional treatment effect test based on this embedding and the MMD. With a very similar motivation, Park et al. (2021) proposed a test for conditional distributional treatment effects based on kernel conditional mean embeddings. In a preprint, FFawess et al. (2022) extended the work to AIPW estimators, however no theoretical guarantees regarding type-1 I enror control of the proposed tests were provided.
Finally, the kernel-based tests used in the aforementioned cases involve test statistics that are degen- erate U-statistics under the null, hence obtaining theoretical p-values of the statistic is not possible. In turn, Kim and Ramdas (2023) proposed the idea of cross U-statistics, which is based on splitting the data for achieving a normal asymptotic distribution after studentization. Similarly, (Shekhar et al. (202)) exploited sample splitting for proposing a permutation-free kernel two sample test.
3 Preliminaries
In this section, we introduce the concepts that the proposed test for distributional treatment effects is mainly based on: Maximum Mean Discrepancy (Gretton et al., 2012). Conditional Mean Embeddings
(Song et al. 2009), Kernel Treatment Effects (Muandet et al. 2021) 1), dimension-agnostie inference
using cross U-statistic (Kim and Ramdas 2023) Shhkhhrr et al. 2022), and the normal asymptotic
behaviour of the doubly robust estimator (Funk et al. 2011).
3.1 Maximum Mean Discrepancy (MMD) and Conditional Mean Embeddings
Let X' be a non-empty set and let H be Hilbert space of functions f: + R with inner product (; 1). function X x X' - IR. is called a reproducing kernel of H if (i) k(;,x) E H. for all e fxx for all x E f f E H. If 7 has reproducing kernel, then it is called a Reproducing Kernel Hilbert Space (RKHS).
X. (ii) (f, (f.,k(())))
Building on a reproducing kernel k and a set of probability measures P, the Kernel Mean Embedding (KME) maps distributions to elements in the corresponding Hilbert space as follows:
f / z))P((..
P:P - H. P P-1:
If the kernel l is "characteristic" (which is the case for frequently used kernels such as the RBF Matern kernels), then fa is injective. Conditional mean embeddings (Song et al., 2009) extend the concept of kernel mean embeddings to conditional distributions. Given two RKHS He, Hay the conditional mean embedding operator is Hilbert-Schimdt operator Cyxx Hr. + Hi,
satisfying PY|x== = (r, where Cy := CyxCxxx Cyx := Ey.x (ky Y) )) @ (x) and Cxx = Ex,x (,, @ l,, X)). Given dataset (x, y), a sample estimator may be defined as
Cvix = 1y (Kxx -11
where by [ky 81 .................... (. yn IT. 4x := (. ..... k. TT, Kxx := DT denotes the Gram matrix, and X is regularization parameter.
Building the KME, Gretton et al. (2012) introduced the kernel maximum mean discrepancy (MMD). Given two distributions IP and ! and a kernel k, the MMD is defined as the largest difference
in expectations over functions in the unit ball of the respective RKHS:
MMD(P, ?
sup E [x-rIf(X) Ex (x')) = || - Mell
fEW
3

---

U. It can be shown Gretton et al. 2012 that MMD Q llp -glli If l: is characteristic, then (IP Q) 0 if and only IP Q Given two samples drawn from and Q. the MMD between the empirical distributions may be used to test the null hypothesis Ho P Q against H1 P + Q. However, this statistic degenerate two-sample U-statistic under the null, thus one cannot analytically calculate the critical values. Consequently, permutation- based resampling approach is widely used practice (Gretton et al. 2012).
3.2 Kernel Treatment Effect: A distributional Bernel-based treatment effect test
Based on the MMD and the potential outcomes framework, Kernel Treatment Effects (KTE) were introduced in Muandet et al. (2021) for testing distributional treatment effects experimental settings (i.e. with known propensity scores).
Let (X,, A,, YJ)== (X, A, Y) such that (i) (consistency) Y AYT (1 A)Y (ii) no unmeasured confounding and overlap assumptions hold. The KTE considers the MMD between Yū and Yf to test Hg Py;-- = Py, against Ho PY; + Py; They define
KTE
11
By, 11:
where
A,k(, Y.) T( ((x))
(1 (,,,() n (X,)
by.
Nog
n1
Alternatively, we may also define an unbiased version it. Again, under the null, these are degenerate two-sample U-statistics. Hence, Muandet et al. (2021) proposed permutation- based approach for thresholding.
The KTE considers the MMD between the mean embeddings of the counterfactual distributions. This is the cornerstone of the proposed distributional test, will be exhibited in Section 4 which extend in a doubly-robust manner to observational settings.
3.3 Permutation-free inference using cross U-statistics
The previously mentioned permutation-basee approach for obtaining the threshold for the MMD statistic (and hence the KTE statistic) comes with finite-sample validity (Gretton et al., 2012). The number of permutations used to find the empirical p p-values generally varies from 100 1000. Consequently, the computational cost of finding a suitable threshold for the statistic is at least O(Br²). Such computational cost reduces the applicability of the approach, especially when time or computational resources are limited.
Driven by developing dimension aagnossii inference tools, Kim and Ramdas (2023) presented permutation-fin approach to test null hypotheses the form Hp / = O against H, / F O. where μ is the mean embedding of distribution IP, based on the idea of sample splitting. If X1, X2n IP. the usual degenerate V-statistic considers
MMD = 1|
where fu2n 2n, Eis k(,,X,) X;) (or the similar unbiased version). Kim and Ramdas (2023) proposed to split the data in two and study
2n
xMMD (p/t A8)
where An = En k(, X.) = 1 Dm Ky, Denoting U2 (k(+, Xx), A.), we have that
xMMD = Eu
1=1
Under the null and some mild assumptions on the embeddings, Kim and Ramdas (2023 Theorem 4.2) proved that
VnU 4 N(,,
MMD
S,

---

Consequently, FMMD is the statistic considered and the null rejected when 7MMD > 21-a- where 21-a the (1 - az) -uaantile of (0, 1). Such test avoids the need for computing the threshold so it reduces the computational cost by factor of B and it is minimax rate optimal in the 7,2 distance and hence its power cannot be improved beyond constant factor (Kim and Ramdas 2023).
where U
The permutation-free nature of cross U-statistic is key in the proposed distributional test. It will allow us to circumvent the need for training regressors and propensity scores repeatedly, while preserving theoretical guarantees.
3.4 Empirical mean asymptotic behavior of AIPW
The main property from AIPW estimators that will be exploited in proposed distributional treatment effect is the asymptotic empirical mean behaviour the estimator. We present sufficient conditions in the next theorem.
Theorem 3.1. Let (2,0,
-- (x)) + (1() (o(x). that eb
EU((X, Y)) is the average treatment effect. Suppose that
. f is constructed from an independent sample or I and f are contained in a Donsker class.
f = ff=
Suppose also that (by clipping) IP( E [e,1 e]) = 1. If m|| E. 110, -aall =oe( (t)) then it follows that
hAIPW - = (P + or( 1 )
so it is root-n consistent and and usymptotically normal.
Note that the IPW estimator can be seen as an AIPW with (X) (x) = 0 almost surely. The IPW estimator also asymptotically normal if ş| ক|| do (.))
). In experimental settings || - т| = 0, hence the root-n rate is achieved. Under certain conditions (e.g., consistent nuisance estimation at 1/4 rates), the asymptotic variance of the AIPW estimator is minimized for 0, 0,. 8 = thus the IPW estimator is generally dominated by the AIPW if 01, 0g are consistent.
The idea exhibited Theorem 3.1 will allow for using cross U-statistics in estimated mean embed- dings, rather than the actual embeddings. Nonetheless, Theorem 3.1 applies to finite-dimensional outcomes Y. We state and prove the extension of Theorem 3.1 Hilbert spaces in Appendix C which will be needed to prove the main result of this work.
Main results
We are now ready to introduce the main result of the paper. Let = (X, be such that
Y = AYT + (1 A)), and that both no unmeasured confounding and overlap assumptions hold. We denote the space of observations by = x'xxx)) We are given Z, (X,, A,, Y)) (X, A. Y) and we wish to test Hg Py: Pro against H1 Pyr * Pyς. Given characteristic kernel
i.c. k(y. W) = (k(s v), k(,, 7)) with induced RKHS Hg. we equivalently test Ho [[k(( Yf)
k(, Y7) =
Under consistency, no unmeasured confounding, and overlap, we have
Elk(, YT" k(-, X2) [[)(
where
1-a |k Ba(x)) (z -Bo(x), ((2) T(x) =(x)
* (Φx) E[A X = 2), (x(z) = Elk(, 14
4= a z].
5

---

Note the change in notation, from 9n to 3u, to emphasize that such regression functions are now H..-valued. Thus, we can equivalently test for Ho [[](2)] 0. With this goal in mind, we denote D1 (X,, As, 1=) D2 = (X,: Ag. =++= and define
T VnfM S
(2)
where
(2,) (3(2) (Zi), ((2)) ie [n]
Above, (z) is the plug-in estimate of ((2) for E {1,2} using *(r) and (8r), which approximate and 8n respectively. Further, fh and S), denote the empirical mean and standard error of f/:
n f1. = n Σ flZ.).
n n
5
-1)22.
The next theorem, which is the main result of the paper, establishes sufficient conditions for 77, to present Gaussian asymptotic behavior. While the main idea relies on combining cross U-statistics and the asymptotic empirical mean-like behaviour of AIPW estimators, we highlight number of technical challenges underpinning this result. The proof combines the central idea presented in Kim and Ramdas (2023) with a variety of techniques including causal inference results, functional data analysis, and kernel method concepts. Furthermore, additional work needed extend Theorem [.]] to Ha-valued outcomes. Donsker classes are only defined for finite dimensional outcomes; in the Ha-valued scenario, we ought to refer to asymptotically equicontinuous empirical processes (Park and Muandet 2023) and Cllivenko-Cantel classes. We refer the reader to Appendix C for presentation of such concepts, clarification of the norms used, and the proof of the theorem.
Theorem 4.1. Let k be a kernel that induces a separable RKHS and Py; Py, be two distributions. Suppose that (i) E(ő(Z) [Z]] = O, (ûi E [(((z)) < (Z)))² > O, (iii) E [l is finite. For E (1, suppose that (iv) g(r) is constructed independently from DD (v) the empirical process of (())) is asymptotically equicontinuous at & and | belongs to Glivenko- Glivenko-Cantelli class. If it
also holds that (vi) [|ā(+) all = or),), (vii) (((((r) E (c, 1 c)) = 1, and
[|+(() T|| > (1 Ball oF Ad
(3)
for (1,2), then it follows that
T 4 1).
We would like to highlight the mildness of the assumptions of Theorem 4.1 The separability of the RKHS is achieved for any continuous kernel on separable y (Hein and Bousquet 2004). Assumption (i) is always attained under the null hypothesis (it is precisely the null hypothesis). Assumption (ii) prevents d(Z) from being constant. such degenerate case, S thus TÍ. is not even well-defined. Assumption (iii) is the more restrictive out of the first three assumptions. inherited from the use of Lyapunov's ' CLT the proof. However, we note that this condition is immediately satisfied under frequently used kernels. For instance, under bounded kernels (for example the common Gaussian and Laplace kernels) such that [|A((, Y) I M, we have that
[B[K( Y)JA a, X 20]1 S E(II(, 1) a, X 20) s M, [1Ba())
hence
(1))()) Y)I + 6 max (x)m))=)1111)0 + 118 [1) + 11,3 (1) [0) S --
which is upper bounded by 2 + 1)M. Consequently, IE [[]((Z) [1]= < [2(e-1 - 1)

---

Furthermore, conditions (iv), (vi), (vii) and (3) deal with the proper behaviour of the AIPW estimator; they are standard in the causal inference scenario. Condition (iv) is equivalent to two-fold cross- booss-fitting ie., training ((() on only half of the data and evaluating such an estimator on the remaining half. Condition (v) replaces the Donsker class condition from the finite dimensional setting.
We emphasize the importance of double-robustmes in the test; normality of the statistic is achieved due to the or (1/ vn) rate, which is possible in view of the doubly robust nature of the estimators. We also highlight the fact that IPW estimators of the form (2) [ - }i(:,)) y) can be embedded in the framework considering Bu 31 O. In fact, the doubly robust kernel mean embedding estimator may be viewed as an augmented version of the KTE (which is a kernelized IPW) using regression approaches to kernel mean embeddings (Singh al., 2020), just as AIPW augments IPW with regression approaches. Furthermore, (3) is always attained when the propensity scores π are known (i.e. experimental setting), given that ||
- - 0.
Based on the normal asymptotic behaviour of h. we propose to test the null hypothesis Ho: Py; =
Py; given the p-value p = (TT)), where & is the edf of standard normal. For an a-level test, the test rejects the null if I a. We consider a one-sided test, rather than studying the two-sided p-value 1 - (. given that positive values of TV are expected for [l((Z) o O. The next algorithm illustrates the full procedure the test, which we call AIPW-xKTE (Augmented Inverse Propensity Weighted cross Kernel Treatment Effect).
Algorithm 1 AIPW-xKTE
1: input Data D = (X,, A., Yi)
2: output The p-value of the test.
3: Choose kernel k and estimators 36r), ((r) for E {1,2].
4: Split data in two sets D (X,, A,, 7 D2 (X,, A,, 10
5: If 3,r)) (*)) are such that condition (v) from Theorem 4.1 is attained, train them D. Otherwise,
train them on D1-r-
6: Define dir) (z)
{ [,)() I-
1-a HM( v) - 30 (x)) + (z)
7: Define (2())) 1 "n
a Li=n+1 (Z.) (22) (Zz)) for 1, I,..., n2.
8: Calculate T/ vTlg where Ph = n Em f/((% (7+) and S (T fi
5%
9: return p-value p = (T)))
Note that the proposed statistic is, at heart, two sample test (with a nontrivial causal twist); in contrast to Shekhar et al. (2022), the two samples are not independent and are potentially confounded.
Extensive literature focuses on designing estimator (1)), logistic regression being the most common choice. At this time, not many choices exist for estimators ((r) given that it involves regression task in a Hilbert space. Conditional mean embeddings are the most popular regressor, although other choices exist (Cevid et al. 2022).
Note that we have motivated the proposed procedure for testing distributional treatment effects with characteristic kemels. However, the actual null hypothesis being tested is Ho : BB[k(- Yf) = Ejk(-, ). If the kernel chosen is not characteristic, the test would continue to be valid for Ho. although it would not be valid to test equality between Fy, and Fyr For instance, AIPW-xKTE with a linear kernel could be used to test equality in means of counterfactuals.
Furthermore, the proposed test is permutation-free, as the statistic T/द ought to be computed only once.
This permutation free nature is crucial, as it avoids the repeated estimation of (r)) A((r) For instance, conditional mean embeddings involve the inversion of a matrix, which scales at least at O(z²), with practical values being w = 2.87 by Strassen's 's algorithm (Strassen et al.) 1969). Calculating the conditional mean embedding for every permutation would imply O(Bn where B is the number of permutations. Furthermore, regressors for mean embeddings of different nature might involve a higher complexity, hence avoiding permutations becomes even more important in the approach.
7

---

If the actual embedding é was known, the power of AIPW-xKTE could not be improved beyond constant factor (by minimax optimality of cross U-statisties in L distance). Further, every procedure will suffer from the error in estimation of Φb This means that we are potentially incurring in loss of power by avoiding a permutation-basee approach, however such a loss is controlled by a small factor. Nonetheless, this potential loss is inherited from splitting the data in our estimator (only half of the data is used on each side of the inner product). We highlight that sample splitting is needed when using flexible doubly-robust estimators, hence expect no loss in power compared to other potential doubly-robust approaches in that case.
5 Experiments
In this section, explore the empirical calibration and power of the proposed test AIPW-xKTE. For this, we assume that we observe (24, a, y:)== ~ (X, A, Y) and that (causal inference assumptions) consistency, no unmeasured confounding, and overlap hold. Both synthetic data and real data evaluated. All the tests are considered at a 0.05 level. For an exhaustive description of the simulations and outcomes, including additional experiments, direct the reader Appendix [
Synthetic data, All data (covariates, treatments and responses) are artificially generated. We define four scenarios:
Scenario I: There is no treatment effect: thus, Py; = Py;
Scenario II: There exists treatment effect that only affects the means of Py; Pyy.
Scenario III and Scenario IV: There exists treatment effect that does not affect the means but only affects the higher moments of Py; and Py,, differently for each scenario.
For all four scenarios, we consider the usual observational study setting, where the propensity scores T (X) are treated unknown and hence they must be estimated. We define the proposed AIPW-xKTE test with the mean embedding regressions fitted as conditional mean embeddings and the propensity scores estimated by logistic regression.
We first study the empirical calibration of AIPW-xKTE and the Gaussian behaviour of T under the null. Figure exhibits exhibitts the performance of AIPW-xKTE in Scenario I. Both standard normal behaviour and proper calibration are empirically attained in the simulations.
Due to the fact that the KTE (Muandet et al., 2021) may not be used in the observational setting, where the propensity scores not known, there is natural benchmark for the proposed test. In partieular, we were unable to control the type-1 error of the test presented in Fawkes et al. (2022). and hence omitted from simulations. Consequently, and IPW-xKTE with respect to three methods that are widely used while conducting inference on the average treatment effect: Causal Forests (Wager and Athey 2018). Bayesian Additive Regression Trees (BART) (Hahn et al. 2020), and linear regression based AIPW estimator (Baseline-AIPW),
compare the power of the proposed AIPW-xKTE
(8)
Drews from APW WKTC
041
Stancard pdf
03
Dersity
Talls positive
Ordered
0.08
0.06
0.1
0
200
400
600
800
1000
quantiles
Figure 1: Illustration of 500 simulations of the AIPW-xKTE under the null: (A) Histogram of AIPW- xКТЕ alongside the pdf of a standard normal for n = 500, (B) Normal Q-Q plot of AIPW-xKTE for 500, (C) Empirical size of AIPW-xKTE against different sample sizes. The figures show the Gaussian behaviour of the statistic under the null, which leads to a well calibrated test.
8

---

True positive rate
0.0
0.0
0.6
0.6
AIPW-OKTF four propesal)
AIPW-OKTTE jour procosal)
PWWJKCT smppssa)
62-
Baseline AIPW
02-
Bsseline-APPY
02-
Basaline-APPW
BART
BART
BART
FausslForest
CausalFarest
Cauraloorest
100
150
200
250
300
350
100
150
200
300
350
100
150
200
250
300
350
Figure 2: True positive rates of 500 simulations of the tests in Scenarios II, III, and IV. AIPW-XKTE shows notable true positive rates in every scenario, unlike competitors.
Figure 2 exhibits the performance of such tests in Scenario II, Scenario III, and Scenario IV. The three methods dominate AIPW-xKTE in Scenario II, where there exists mean shift in counterfactuals. However, and as expected, such methods show no power if the distributions differ but have equal means. contrast, AIPW-xKTE detects distributional changes beyond the mean, exhibiting power in all scenarios.
Remark: While this work focuses on the observational setting, where double robustness is crucial, we highlight that the proposed AIPW-xKTE test may also be used in experiments (where propensity scores are known) for computational gains. The proposed test avoids permutations, which makes more computationally efficient than the KTE (Muandet et al. (021). We refer the reader to Appendix ffror a comparison between the proposed AIPW-xKTE and the KTE.
Real data. We use data obtained from the Infant Health and Development Program (IHDP) and compiled by Hill (20TT), in which the covariates come from randomized experiment studying the effects of specialist home visits on cognitive test scores. This data has seen extensive use in causal inference (Johansson et al. 2016 Louizos al., 2017 Shalit et al.) (017)). We work with 18 variables of the covariate set and unknown propensity scores. We highlight that the propensity score model is likely misspecified in this real life scenario.
We consider six scenarios with the IHDP data. For Scenarios I, II, III and IV, we generate the response variables similarly the previous experiments. In Scenario V, we take the IQ test (Stanford Binet) score measured at the end of the intervention (age 3) as our response variable. For Scenario VI, calculate the average treatment effect in Scenario V using Causal Forests (obtaining 0.003 i.e. a positive shift) and subtract from the IQ test score of those who are treated, thus obtaining a distribution of the IQ test scores with zero average treatment effect.
For each of the scenarios, we consider the tests AIPW-xKTE, Causal Forests, BART and Bascline- AIPW on 500 bootstrapped subsets estimate the rejection rates. All of them showed expected levels of rejection under the null i.e. Scenario I. The results for the remaining scenarios can be found in Table We note that the performance of the tests on Scenarios I, II, III, and IV is similar the analogous scenarios on synthetic data. While AIPW-xKTE exhibits a loss in power with respect to
Table True positive rates (+ std) for the different scenarios and tests using the IHDP data. While AIPW-xKTE shows less power in Scenario II, it outperforms its competitors in Scenarios III and IV.
Test
Scenario
II
III
IV
V
VI
AIPW-KKTE
0.44 + 0.05
0.34 + 0.05
0.53 + 0.05
0.99 = 0.01
0.03 + 0.02
Baseline-AIPW
0.95 + 0.02
0.00 0.00
0.00
0.00
1.00
- 0.00
0.00 4 0.00
BART
0.77 +: 0.04
0.15 + 0.04
0.24
+ 0.04
0.10
- 0.03
0.04 0.02
CausalForest
+.0000000
0.04 0.02
0.05
0.02
1.00 0.00
0.00 0.00

---

the other tests when the average treatment effect is non zero, it detects distributional changes beyond the mean.
Besides BART, all of the tests reject the null in almost every simulation of Scenario V. As expected, Causal Forest, BART and Baseline-AIPW barely reject the null in Scenario VI, which considers the data of Scenario V with zero average treatment effect. Interestingly. the proposed distributional test has rejection rate below 0.05, which supports the fact that specialist home visits have no effect on the distribution of cognitive test scores beyond an increase in the mean.
6 Conclusion and future work
We have developed a computationally efficient kernel- test for distributional treatment effects. It is, to knowledge, the first kerne--based test for distributional treatment effects that allows for doubly-robust approach with provably valid type-I error. Furthermore, it does not suffer from the computational costs inherent permutation-basee tests. The proposed test empirically proves valid in the observational setting, where its predecessor KTE may not be used: the test is well calibrated and shows power in variety of scenarios. Procedures designed test for average treatment effects only outperform the proposed test if there is mean shift between counterfactuals.
There are several possible avenues for future work. We highlighted that our procedure holds if consistency, no unmeasured confounding, and overlap hold. However, this may often not be the case in observational studies. Generalizing the work to other causal inference frameworks, for example by considering instrumental variables, would be of interest. Exploring the extension of this work to test for conditional treatment effects could also be natural direction to follow. Lastly, expect state-of-the-art regressors for kernel mean embeddings, such as distributional random forests (Cevid et al. 2022), to find outstanding use in our estimator. We envisage that this may motivate the development of flexible mean embedding estimators designed for complex data, such image text.
Acknowledgements
The project that gave rise to these results received the support of a fellowship from 'la Caixa" Foundation (ID 100010434). The fellowship code is LCF/BB/EUU2//1130005
References
Abadie, A. (2002). Bootstrap tests for distributional treatment effects in instrumental variable models. Journal of the American Statistical Association, 97(457):224-22
Bosq, D. (2000). Linear processes in function spaces: theory and applications, volume 149, Springer Science & Business Media.
Chernozhukov, V., Fernández-Val Val, I., and Melly, B. (2013), Inference on counterfactual distributions. Econometrica, 2166522205
Díaz, (2017). Efficient estimation of quantiles in missing data models. Journal of Statistical Planning and Inference, 190:39-51.
Fawkes, J., Hu, R., Evans, J., and Sejdinovic, D. (2022). Doubly robust kernel statistics for testing distributional treatment effects even under one sided overlap. arXiv preprint arXiv:2212.00 02422
Funk, M. J., Westreich, D., Wiesen, C.. Stürmer, T., Brookhart, M. A., and Davidian, M. (2011). Doubly robust estimation of causal effects. American Journal of Epidemiology, 173(7):761-767.
Gretton, A., Borgwardt, K. M., Rasch, M. J., Schölkopf, B., and Smola, A. (2012). A kernel two-sample test. The Journal of Machine Learning Research, 13(1):723-773.
Hahn, P. R., Murray, J. S., and Carvalho, C. M. (2020). Bayesian regression tree models for causal inference: Regularization, confounding, and heterogeneous effects (with discussion). Bayesian Analysis, 15(3):955-1056
10

---

Hein, M. and Bousquet, O. (2004). Kernels, associated structures and generalizations.
Heran,, M. and Robins, J. (2020). Causal inference: What if. boca raton: Chapman & hall/erc.
Hill, J. L. (2011). Bayesian nonparametric modeling for causal inference. Journal of Computational and Graphical Statistics, 20( (0))2217.
Horváth, L. and Kokoszka, P. (2012). Inference functional data with applications, volume 200. Springer Science Business Media.
Hsing, and Eubank, R. (2015). Theoretical foundations of functional data analysis, with an introduction to linear operators, volume 997. John Wiley & Sons.
Imbens, G. W. (2004). Nonparametric estimation of average treatment effects under exogeneity: A review. Review of Economics and Statistics, 86(1):4-29
Johansson, F., Shalit, U., and Sontag, D. (2016). Learning representations for counterfactual inference. In International Conference on Machine Learning, pages 3020-3029. PMLR.
Kang, D. and Schafer, J. L. (2007). Demystifying Double Robustness: A Comparison of Alternative Strategies for Estimating Population Mean from Incomplete Data. Statistical Science, 22(4):523 539.
Kennedy, E. H. (2019). Nonparametric causal effects based on incremental propensity score interven- tions. Journal of the American Statistical Association, 114(526):645-66 656.
Kennedy, E. H. (2022). Semiparametric doubly robust targeted double machine learning: review. arXiv preprint arXiv:2203.0644:
Kennedy, E. H., Balakrishnan, S., and Sell, M. (2020), Sharp instruments for classifying compliers and generalizing causal effects. The Annals of Statistics, (5(4):2008-2033)
Kennedy, H., Balakrishnan, S., and Wasserman, L. (2021). Semiparametric counterfactual density estimation. arXiv preprint arXiv:2102.12034,
Kim, I. and Ramdas, A. (2023). Dimension-agnont aagnostic inference using cross U-statistics. Bernoulli.
Kim, K., Kim, J., and Kennedy, E. H. (2018). Causal effects based distributional distances. arXiv preprint arXiv:/806.02993.
Landmesser, J. M. (2016). Decomposition of differences in income distributions using quantile regression. Statistics Transition new series, 17(2):33-334.
Li, L., Tchetgen, E. T., van der Vaart, A., and Robins, J. M. (2011). Higher order inference on a treatment effect under low regularity conditions. Statistics & Probability Letters, 81(7):821-828,
Louizos, C., Shalit, U., Mooij, M., Sontag, D., Zemel, R., and Welling, M. (2017). Causal effect inference with deep latent-variable models. Advances in Neural Information Processing Systems, 30.
Luedtke, A. R., Diaz, I., and van der Laan, M. (2015). The statistics of sensitivity analyses.
Muandet, K., Kanagawa, M., Saengkyongam, S., and Marukatat, S. (2021). Counterfactual mean embeddings. The Journal of Machine Learning Research, 2211)7732--332.
Muñoz, I. D. and Van Der Laan, M. (2012). Population intervention causal effects based stochastic interventions. Biometrics, 68(2):541-549.
Park, J. and Muandet, K. (2023). Towards empirical process theory for vector-valued functions: Metric entropy of smooth function classes. In International Conference on Algorithmic Learning Theory, pages 1216-1260. PMLR.
Park, J., Shalit, U., Schölkopf, B., and Muandet, K. (2021). Conditional distributional treatment effect with kernel conditional mean embeddings and U-statistic regression. In International Conference on Machine Learning, pages 8401-8412. PMLR.

---

Richardson, A., Hudgens, M. G., Gilbert, P. B., and Fine, J. P. (2014). Nonparametric bounds and sensitivity analysis of treatment effects. Statistical Science: A Review Journal of the Institute of Mathematical Statistics, 29(4):996.
Robins, J., Li, L., Tchetgen, E., van der Vaart, A., et al. (2008). Higher order influence functions and minimax estimation of nonlinear functionals. Probability and Statistics: Essays in Honor of David A. Freedman, 2:335-422 -21.
Robins, J. M. and Rotmitzky, A. (2001). Comments on: Inference for semiparametric models: Some questions and an answer. Statistica Sinica.
Rubin, D. B. (2005), Causal inference using potential outcomes: Design, modeling, decisions. Journal of the American Statistical Association, 100(469):322-331.
Shalit, U., Johansson, D., and Sontag, D. (2017). Estimating individual treatment effect: generaliza- tion bounds and algorithms. In International Conference on Machine Learning, pages 3076-3085. PMLR.
Shekhar, S., Kim, I., and Ramdas, A. (2022). A permutation kernel two-sample test. In Advances in Neural Information Processing Systems.
Shen, S. (2019). Estimation and inference of distributional partial effects: theory and application. Journal of Business & Economic Statistics, 37(1):54-66.
Singh, R., Sahani, M., and Gretton, A. (2019), Kernel instrumental variable regression. Advances in Neural Information Processing Systems, 32.
Singh, R., Xu, L., and Gretton, A. (2020). Kernel methods for causal functions: Dose, heterogeneous, and incremental response curves. arXiv preprint arXiv:201O. 048555
Singh, R., Xu, L., and Gretton, A. (2021). Kernel methods for multistage causal inference: Mediation analysis and dynamic treatment effects. arXiv preprint vrXiv:2111.0399
Song, L., Huang, J., Smola, A., and Fukumizu, K. (2009). Hilbert space embeddings of condi- tional distributions with applications to dynamical systems. In Proceedings of the 26th Annual International Conference on Machine Learning, pages 961-968.
Strassen, V. et al. (1969). Gaussian elimination is not optimal. Numerische mathematik, 13(4):354- 356.
Van der Vaart, A. W. (2000). Asymptotic statistics, volume 3. Cambridge University Press.
Wager, and Athey, S. (2018). Estimation and inference of heterogeneous treatment effects using random forests. Journal of the American Statistical Association, 113(523):1228-1242
Westling, T. and Carone, M. (2020). A unified study of nonparametric inference for monotone functions. The Annals of Statistics, 4802):1001.
Zhu, Y., Gultchin, L., Gretton, A., Kusner, M. J., and Silva, R. (2022). Causal inference with treatment measurement error: nonparametric instrumental variable approach. volume 180 Proceedings of Machine Learning Research, pages 2414-24444 24244. PMLR.
Čevid, D., Michel, L., Näf, J., Meinshausen, N., and Bühlmann, (2022). Distributional random forests: Heterogeneity adjustment and multivariate distributional regression. Journal of Machine Learning Research, 23(333):1779.
12

---

A Comparison between KTE and AIPW-xKTE in the experimental setting
In Section we exhibited the performance of the proposed AIPW-xKTE in the observational setting. We emphasize that the proposed AIPW-xKTE test is designed to target the observational setting, where the KTE may not be used given that the propensity scores are unknown. In such setting, the double-robustness of the proposed test is fully exploited. However, the proposed test may also be used in the experimental setting for its computational efficiency: the AIPW-xKTE test avoids permutations, which makes it more computationally efficient than the KTE.
Consequently, explore in this appendix the performance of the AIPW-XKTE in the experimental setting. We consider usual experimental design where 2 units are treated and units are not treated so that T (X) Z. We compare the power of the proposed AIPW-xKTE with respect to the KTE test. Further, we include an IPW version of the proposed test (by taking 3n = 31 = O, see comments in Section 4 in order to elucidate the interest in considering the doubly robust version. We refer the B for an exhaustive description of the experiments.
reader to Appendix
Figure exhibits the performance of the tests in Scenario II, Scenario III, and Scenario IV. We note that IPW-xKTE has less power than the KTE, which its analogous permutation- This is due to the controlled loss in power of the cross U-statistic approach following the comments in Section 4 However, we see that the AIPW-xKTE competes with the KTE, despite showing slightly less power. The loss in power inherited by the permutation-fiee free approach is compensated by the gain in power due to using an AIPW estimator. Furthermore, AIPW-xKTE shows a drastic improvement in computational costs with respect to KTE, as exhibited in Table 2 Lastly, we see that AIPW-xKTE clearly outperforms IPW-xKTE, which illustrates the significant benefits of the doubly robust approach.
version.
Table 2: Average times (in milliseconds) for different tests and sample sizes. AIPW-xKTE is almost 20 times faster than KTE for n 350 with only 100 permutations used.
Test
150
250
350
AIPW-xKTE IPW-xKTE
1.770
2.994
4.940
1.496
2.297
2.331
KTE
16.903
44.483
89.875
(8)
03-
Drows from KIP Jo0e
0.11
APV WIT Exporimonal sett
Stanclrdd narmal pol
3TW ABL Expermental actling
04
Ordered Va luees
Ealse gosltort FS2
0.10
0.12
Dersit
0.4
0-
0.00
0.06
0.04
0.02
66
0.00
200
400
600
800
1000
APPW-KK XITE
Theoretical quantiles
Figure Illustration of 500 simulations of the AIPW-xKTE under the null in the experimental setting: (A) Histogram of AIPW-xKTE alongside the pdf of a standard normal for = 500, (B) Normal Q-Q plot of AIPW-xKTE for n = 500, (C) Empirical size of AIPW-xKTE and IPW-xKTE against different sample sizes. The figures show the Gaussian behaviour of the statistic under the null, which leads to a well calibrated test.
13

---

00
010
(M)
10-
10-
1.5
True positive rate
64
0.8
0.4
AIWA-KTTE
APPW-KTE PWW-KTT
AIPLY-KTT
82-
02-
KTE
KTE
KTE
100
150
200
250
300
350
100
150
200
250
300
350
100
150
200
250
300
350
Figure 4: True positive rates of 500 simulations of the tests in Scenarios II, III, and IV in the exper- imental setting. AIPW-xKTE demonstrates comparable power to KTE, and improves appreciably over IPW.xKTE.
14

---

B Experiments
We present this appendix a comprehensive display of the simulations conducted. Subsection includes an exhaustive description of the experiments introduced in Section B and Appendix A
Additional simulations and results may be found in Subsection B.2
B.1 Exhaustive description of experiments
We assume that we observe (I; 0z M/) 1= ~ A, and that (causal inference assumptions) consistency, no unmeasured confounding, and overlap hold. All the tests are considered a 0.05 level. Both synthetic data and real data are evaluated.
Synthetic data. All the data (covariates, treatments and responses) is artificially generated. We define
%8 = 3d x + Yi = 3 X 61,
such that Co, 61 N0,00. 0.5) are independent noises, X ~ (0. Is) and 5 = [0.1, 0.,, 0.3, 0.4, 0.5] for Scenario I and Scenario respectively. For Scenario III, we set b = 27 - 1, where Z is independent Bernoulli random variable Z ~ Bernoulli(O.5). In Scenario IV, Uniform( 2).
We set
and
In the experimental setting, we consider a usual experimental design where units are treated and units are not treated such that (X) 1 almost surely. In the observational setting, we define (X) s(a"X X + aa), where s()) 1/(1 exp(-2)) (sigmoid function), ogg 0.05 and a = [0.05, 0.04, 0.03, 0.02, 0.01]". In such setting, we estimate र using an L2 regularized logistic regression with the regularization term set to 1e-6. For AIPW-xKTE, the mean embedding regressions are fitted as conditional mean embeddings.
In the experimental setting, we consider the tests IPW-xKTE, AIPW-xKTE, and KTE (Muandet al., 2.21). In the observational setting, we consider IPW-xKTE, AIPW-xKTE and permutation- based tests based on Causal Forests (Wager and Athey 2018), Bayesian Additive Regression Trees (BART) (Hahn et al. 2020), and a linear regression based AIPW estimator (BaselineAIIPW), For the latter three, recalculate the respective statistics for every permutation and reject the null if the original statistic is above the 0.95 empirical quantile. For all scenarios and settings, we consider 500
Response against covariates
(N
1-2
T-11
T-0
-15
2
X(1)
X(3)
XL11
X()
Histograms
COUrt
15
-5
100
-5
20
Figure 5: Illustration of one simulation of the simulated data in the experimental setting with 350 for Scenarios I, II, III, and IV. We display scatter plots of the outcome against the first covariate and histograms of the outcomes, both grouped by treatment.
15

---

simulations for each n E {100, 150, 200, 250, 300, 350). We exhibit an illustration of the synthetic data from one the simulations carried in Figure
Real data. We use data obtained from the Infant Health and Development Program (IHDP) and compiled by Hill (20TT), in which the covariates come from a randomized experiment study- ing the effects of specialist home visits on cognitive test scores. The propensity scores are unknown and hence they must be estimated. We work with the following 18 variables of the covariate set: ['bw", 'momage', 'nnhealth', 'birth..o,, "birth.o'.'parity" 'mmoreprmm' 'bwg'. 'female', 'ilt.birt', 'b.marry', 'livwho', 'language', 'whenpren' 'drugs', 'othstudy'], J. where the first nine are continuous and the last nine are discrete; we refer to Hill (2011) for detailed pre- sentation of the data set. We eliminate the rows in which there is a NaN for any of the variables considered, finishing with 908 observations, out of which 347 were treated. Further, we standarize the data such that every continuous variable has mean 0 and variance 1. We illustrate the data obtained after preprocessing in Figure G
"aalcooo"""o "ppcttip.''ppt..mmm" 'ppvt.imp',
We consider five scenarios with the IHDP data. For Scenarios I, II, III and IV, we generate the response variables similarly to the previous experiments:
Yo = 3" X + Yo =
X = + C1,
(5)
where 60,E1 (0,00.5) are independent noises, and B = [l...., 11"h We set b = 0 and b = 1 for Scenario and Scenario II respectively. For scenario III, we set b = 2(2Z 1), where Z is an independent Bernoulli random variable Z ~ Bernoull(O0.)). In scenario IV, b ~ Uniform 4). In Scenario V, we take the IQ test (Stanford Binet) score, variable 'iqsb.36' of the data set, measured at the end of the intervention (age 3) as our response variable. For Scenario VI, we calculate the average treatment effect in Scenario V using Causal Forests (obtaining 0.0035 i.e. a positive shift) and subtract it to the IQ test score of those who are treated, thus obtaining a distribution of the IQ test scores with zero average treatment effect. We illustrate the data of Scenarios V and VI in Figure 6 For each of the scenarios, we consider the tests AIPW-xKTE, IPW-xKTE, Causal Forests, BART and Baseline-AIPW on 500 bootstrapped subsets to estimate the rejection rates. For AIPW-xKTE and IPW-xKTE, we estimate á using a regularized logistic regression with the regularization term set to 1e-6. For AIPW-xKTE, the mean embedding regressions are fitted as conditional mean embeddings.
We sample split for estimating the conditional mean embeddings, but for the propensity scores, we use the whole training data. While we stated Theorem 4.1 imposing condition (iv) or condition (v) on ((r) (common practice in causal inference for ease of presentation, where one usually imposes
Response against first principal component
NE
0423
0845
0640
0.4
0137
010
0150
6173
0123
Histograms
(1)
090
T-D
Count
Figure 6: Illustration of the IHDP data for Scenarios V and VI. We display scatter plots of the outcome against the first principal component and histograms of the outcomes, both grouped by treatment.
16

---

JraNG XCTE
0.14
0.35
0.12
Orderad Values
Falber poollive
0.25-
0.00
025
0.00
0.13
2
200
400
600
800
1000
Theoretical quantiles
Figure 7: Illustration of 500 simulations of the AIPW-xKTE with non-linear effects under the null: (A) Histogram of AIPW-xKTE alongside the pdf standard normal for = 500, (B) Normal Q-Q plot of AIPW-xKTE for 500, (C) Empirical size of AIPW-xKTE. against different sample sizes. The figures show the same Gaussian behaviour of the statistic under the null as in the linear setting.
sample splitting or Donsker condition), Theorem 4.1 holds long sample splitting is conducted
for those estimators ((r), ((r) which may overfit (analogous case in the standard doubly robust estimator). In our case, the propensity is modeled by L2 regularized logistic regression, which is simple and cannot overfit when the dimension fixed and n OC.
All the kernels considered on y for AIPW-xKTE, IPW-xKTE, and KTE are RBF i.e. ky (2) = exp(p, /2 with scale parameter Vy chosen by the median heuristic. For AIPW-xKTE, the fitted conditional mean embeddings, and we conduct sample splitting in order to train them. The kernel considered for such conditional mean embeddings on X' also RBF 2 (x1, #2) erp(yz |2 22 ?) with scale parameter V2 chosen by the median heuristic as well. The regularization parameter was taken to equal to V/-
mean embedding regressions
B.2 Further experiments
In this subsection, we investigate the performance of the tests in non-linear settings. We extend the simulations with synthetic data presented Subsection B.I defining the potential outcomes
y cos( x) Ep. Y = cos X) + E1,
(6)
where B. b, co and 6 are defined as described in Subsection B.1 The different scenarios and the remaining parameters are also defined as in Subsection B.1
Figure displays the behaviour of AIPW-xKTE in Scenario I (under the null). Analogously the linear setting, we note that AIPW-xKTE presents a Gaussian behaviour under the null, which leads to a well-calibrated test. The performance of the tests Scenarios II, III and IV displayed Figure g In this non-linear case, AIPW-xKTE proves even more competitive in Scenario II. while retaining power in Scenarios III and IV, In short, the test proves valid (as expected) in the non-lime on-linnar case, and the comments from Section lequally apply.
17

---

True positive rate
0.8
0.6
6.6
0.6
APPA-XKTE
APMAYKTE
AIPW-OKTE
62-
Baseline-APP AIPW
02-
Baseline-APPY
02-
Basaline-APP
BART
BART
BART
FausalForest
CalisaFarres
CausalForest
150
250
350
100
150
200
300
350
150
250
300
Figure 8: True positive rates of 500 simulations of the tests in Scenarios II, III, and IV with non-linear effects. AIPW-xKTE shows notable true positive rates in every scenario, in the linear setting.
18

---

C
Proofs
We present in this appendix the proof the main result of the paper, Theorem 4.1 For this purpose, we introduce the notation used in Subsection C.I followed by the exposition of previously known results Subsection C.2 We present sequence of extensions of such results to the vector-valued scenario in Subsection
C.3 which will be used to prove the main theorem of the paper in Subsection C.4
C.1 Notation
We use standard big-oh and little-oh notation, i.c. Xn, = Op(rn) means Xn/rn is bounded in probability and X, (p(ra) means Xn/ra 4 O. We use Pn for sample averages i.c. Pn((f)
P.(f(Z)) E. fZ.). For potentially random function f, we denote P(f) P(f(Z))
J f() dP(2), and | f f()) dP(()) represents the squared L2 (IP) norm.
Given a Hilbert space H and H-valued w(z) H. denote its norm in the Hilbert space by [=)))) We let (w) = P, [Z) = E (Z,) and P()) P()(Z)} = ((2) d3(x) (Bochner integral). Furthermore, |ā|| [=) dP((z) denotes the squared (2()) norm of the H-valued function norm, and is the Hilbert-Schmidt norm of operator.
C.2 Previously known results
We start by introducing a generalization of the formula that relates the mean, variance and second moment of real-valued random variable to H-valued random variables.
Lemma C.1 (Hsing and Eubank (2015 Theorem 7.2.2)) Let H. be random variable such that
E Bl < 0c. Then
Bllw - [[w]]|
|B[]]
Furthermore, one may retrieve the expectation the product of projections of the H-valued random variables through the covariance operator, exhibited in the following lemma.
Lemma C.2. Let E H be random variable such that B[w] = 0 and Ellw < co. Then
ER,, (w, 9)1 = (Cf.g), f.ge e H. where C = Elw @ w| is the covariance operator.
Next, we present the generalization of the central limit theorem to H-valued random variables. Theorem C.3 (Bosq (2000) Theorem 2.7)). Let (w,, i 1) be sequence of iid random variables in a separable Hilbert space H. mf = E[wi] and [|[[[]]|| [| < co, then
(wi - m) $ c.
where { is a Gaussian random element with mean zero and covariance operator C' = [R[wy @ wi] ie.
E ~ N (O, E|w1 @ wil).
Moreover, Gaussian H-valued random variables admit the following expansion in the orthonormal system defined by its covariance operator.
Lemma C.4 (Horváth and Kokoszka 2002 Equation 2.8)). normally distributed function & in a separable Hilbert space with covariance operator C admits the expansion
E Σ N,N,,,
where N; are independent standard normal distributions, (Ajs. 2 1) is the sequence of eigenvalues of the covariance operator C, and (Vy. 2 1) is sequence of orthonormal eigenfunctions of the covariance operator C.
19

---

We now present result concerning the asymptotic behaviour of an empirical process given that the considered random functions belong Donsker class. For a full discussion on Donsker classes and empirical processes, we refer the reader to Chapter 18 and Chapter 19 of Van der Vaart (2000).
Lemma C.5 (Van der Vaart (2000) Lemma 19.24)). Suppose that F is a P- D-Dnske class of measur- able (real-valued) functci functions and fn is a sequence of random functions that take their values in F. such that I - fl 2n 0 for some f E L2()). Then
1 Vn
(P,, P))
or
However, the concept of Donsker class is fairly restrictive. In fact, the intuition behind the nice behaviour exhibited by Donsker classes in empirical process is their inability to overfit, due to their constricted flexibilityy. For instance, commonly used estimators such as random forests do not belong to a Donsker class. In order to circumvent this problem, sample splitting proves very useful. The following lemma exhibits the power of sample splitting regardless of the flexibility of the estimators.
Lemma C.6 (Kennedy et al. (020 Lemma 2)). Let f(2) be a function estimated from a sample NN (Zu+1,... 7,), and let In denote the empirical measure over (Zi,..., 7n,), which is
independent of N Then
[1 -11 (m)(
(P, -) f) = - =r
v'n
Proof. First note that, conditional on zN, the term in question has mean zero since
E -P)C- f) 2 E BP zM P(F - f)
= Ef ) -P( - f)
PP( f) -P(ff
=0.
The conditional variance is var (P,, 1l.-P)J P)(( f) |
= var Pr f)
zN} var f - f 2 S 115 f11//n.
Therefore by iterated expectation and Chebyshev's inequality we have
{f [Pp, PP - f)l
21)=
(Pn PF f) fl//á
2 =P
21 z
If fll/vn
llf
Thus for any 0 can pick t 1/ve so that the probability above is more than e, which yields the result.
These two lemmas allow for proving Theorem 3.1 Recall that this theorem exhibits the empirical mean-like behaviour of doubly robust estimators. This will be key when proving the main result of the paper.
Proof of Theorem
3.1 We have that ÉDR à 2" + T1 T2, where
z" = (P - P)f. Ti = (Pr, - Pf - f), T- = TP=P((- f)-
The consistency of f in T.2 norm, together with either the Donsker condition (Lemma C.5 or sample
splitting (Lemma C.6). ensures that Ti op(l/vn). For T2 IP( f) and f fi fo. we have
[PP(f fi)l
IP|
_ (Y 0,) 8, 0 }1
A
PP( _m 1)(0, - b,H
- Pf| - * a110; 6 < || #|||| -lll-
Same logic applies for [P(fo - foll -- #|]]|0 bull. Therefore T- (1(1//)) and the result follows.
D
20

---

We have exhibited the proofs of Lemma C.6 and Theorem 3.1 for sake of completeness. We will extend and prove the analogous results in the H-valued setting in Subsection C. which will be needed in demonstrating the main result of the paper. We refer the reader to | (2022) for
full discussion and exhibition of the aforementioned causal inference results. Lastly, we exhibit the definition of a Gllivenk--Canter class, which is used in Theorem 4.1
Definition C.7 Glivenko-Cantell class for P if
We say that a class of integrable real-valued functions F is
sup Σ f(x)) FEF 1=1
converges to zero in probability +
C.3 Extension of the results to (infnite-dimensionaal) vector-valued outcomes
We introduce the extension of Theorem 3.1 to the N-valued outcome setting. For such goal, we also generalize Lemma C.6 to the H/valued scenario and we comment on asympiotically equicontinuous empirical processes Park and Muandet (2023), which will substitute the Donsker class condition in the generalized version of Theorem
We start by presenting the generalization of Lemma C.6 to the functional setting.
Lemma C.8 (Extension of Lemma C.6 to H-valued outcomes). Let H be a Hilbert space. Let ((z) be a function estimated from sample z 1,....... ZN) such that (+2z) E H for all z, and let F, denote the empirical measure over (Z,.... Zn), which is independent of 2N, Then
1 w|
[I - ( -) = Or
Proof. First note that, conditional on zN, the term in question has mean zero since
B (IP. (2 -w) z E
[m w) z } P( w)
E(# w zzN P( w)
P(@ - w) - Pl@ w)
0.
The conditional expectation of the squared norm is
B B[IICP.. (W w)ll
2 = E ((P, - -W) -P,, P) w))H
[ (()) P( -w] 1 n Σ 110 () - P(@ w]] 2
R
Σ
B ((@ (Z,) P(O - w) (3 - (Z,) P(O -w)) z
n² 1,=1
B ((0 - ( P(O P))-w), (0 - w) P(O w)) 2
Σ
B B|1100 -) P(O wi)l NN
nZ
= E {[1a - (Z)) PP(@ - w)1l EN
()) 1
[B/1IG (2))l |>N} { || - -w) 2m1]]
n
1 // [ w)ll z
110 w/l
< |10 w
21

---

where (i) obtained given independence of Z, Zy when * j and ( -P) P) -w)
2
and (ii) by Lemma C.I Therefore by iterated expectation and Markov's inequality we have
Pn, (a - W)
[P,, - P) - w)l
z
P
2 t
= P
E
2
| w /1l/
[ wwllvv
[[ P) - w)
t² z
P
E
w|13/1
8
Thus for any > 0 we can pick & = 1/v/ so that the probability above is more than which yields the result.
Note that the former result implies that, if a w|l 3 0, then
[I - P) w)| =
given that we conduct sample splitting. However, sample splitting might not be necessary if our estimators are not flexible enough. Splitting the data might result in loss in power, hence interested in stating another sufficient condition that will lead the empirical term (P, - P) w) to be asymptotically negligible without conducting sample splitting. A direct extension of the Donsker class condition stated in Theorem 3.1 to H-valued outcomes is not possible, given that bounding entropies usually make explicit use of the fact that IR is totally ordered (Park and Muandet 2023). However, Park and Muandet (2023) defines asymptotic equicontinuity. which is precisely the condition which we will require the extension of Theorem to the functional setting.
Definition C.9 (Asymptotic equicontinuity). We say that the empirical process (w) = vn(P, - IPW: w E g} with values in H. and indexed by 9 is asymprotic equicontimuuus at wn E G if, for every sequence { G with ||ā, woll 2 O, we have
(0,,) - vn, vn.(w))l 3 0.
Note that 7 is equivalent to (P,, - (2- oz
Park and Muandet (2023) gives sufficient
vn
conditions for asymptotic equicontinuity to hold, as well as stating examples of classes that attain
asymptotic equicontinuity. We are now ready to present the extension of Theorem 3.1 to the H-valued setting.
Theorem C.10 (Extension of Theorem 3.1 to H-valued outcomes). Lei ((z))
( - tv
(x(x)}+ + (x) - (o(x), where and are H-valued, so that Vi = E[o(Z)} is the average treatment effect. Suppose that
à is constructed from an independent sample or the respective empirical process is asymp- totically equicontinuous at Φ.
11 - o1)
Also assume that P(a E [e, - c)) = L. Then || - 11 113a - -Ball = oz (t)))) it follows that
VDR v = -))(() + +
where ||Ea 1 = orl (n) Hence, it is rootn consistent.
Proof. We have that V 1/- = - + Ti + TT. where
7' = (P, P)d, T (I.. P) Φ).
T, T2= T P(6--).
The consistency of Φ in L2 norm, together with either asymptotic equicontinuity (Definition C.9 or sample splitting (Lemma C.8) ensures that || (1/vnn). For T2 P(δ Φ) and
22

---

Ø Φ1 - ou, we have
(PP(2 - x1)lux= = |IP( (Y - BA) + 31 -B1Hl S P(ll( - 1) - -1
$
1 -1(T - -+1131 - -bill - -1-- #|||B3 - -Bll--
1
Same logic applies for P(òu- 6 S - | - =111,00 - -Bull- Therefore ||Tall = oe(1/ and the result follows.
C.4 Proof of Theorem
Assume that we have access to a sample (X;, A;, Y3) ~ (X) We denote Dj =
(X,, A,, Y.) D2 = (X,, A,. 3 -+1-- Further.
1 2n
Vnha
fz)
Σ ()()). j=n+1
T.
n
Sh.
where fn and S are the empirical mean and variance respectively:
fr - 1
- -h 1 - Z(n()) (f)
==1
In particular, note the indices I to in the definition of fr. and the indices n + to 2n when defining fu. Similarly, we denote the analogous version for estimated embeddings & as follows:
(1(=)) Σ (1))
T
vuSh S
1 Σ SL,)), n
= = Σ (1((Z))
j=++1
For of ease of notation, we also define
((Z,),
2n
(Zz)):
1
300 (Z,),
1
( (Zz). (Z))
vn
vn j=
==a+1
Lastly, note that the following equalities hold by definition:
2n
nf.. = nl
((1))
1
((,))
1
(Zz),
Σ
((2z)) = (T),
n
1=1
j=
j=
2n.
2n
nft = n 1
300 (Z,),
(12 =
(2)),
Σ
3 (2,)) = = (2)
i=1
j=n+1
j=ト+1
(VTSa))² n1
(0 Zi);
1
Σ ((z)))
-(h)²²
Σ
( (Zi); T2)² )² (ViJN
" j=n+1
n
(VTSSt)²
(1()) (Z+), 1
Σ
(2)))
- -(7))
1 (())(())) (2)), (Vrrft)².
j=nー1
Proof of Theorem
4.7 We will prove the theorem in four steps:
1. We first prove that n. = nn + op()).
2. We then show that ns! h
n + (1)).
3. Further, we prove that B B1rth = Op(1).
4. We conclude that T 4 1).
It now remains to prove each of the four steps.
23

---

Details of step 1. Theorem C.10 may be applied for both 200), 2()) because its conditions are fulfilled. We obtain that
t1 + Vnch f2 = + vngg
where VV 11, ll
11 - (1). Rewriting the expression, we note that
ř1 + x
f2 12 x!?
where [1xx I1,e, 11xxxx1xx 1lx = (1) Based on Theorem C.3
a 4 N(O,C), 72 S N(O, C),
where [E(((Z) @ (Z)), which implies max |1 m=x) = (+()). Consequently,
nfh (f1, F2)
(TI Xn , + x )
(11,7) + (m1 (2)) + [xn (2) + (xm)) xx
= =nfa + ) + (x (2) + (xn) xx))
(1)
Given that |[T, (2) + (xa (2) + (xm x is upper bounded by | 11 ||
Ixxx (Ixllll + ||x (1) 1x H (by the triangle inequality and Cauchy-Schwarz inequality), which is (1)))))())= oF (1)0p((1 + (1()))=(1) = o1)),, we deduce that
(2)
nf,, nfa + 01))
(8)
Details of step 2. We claim that
(30) (Z., - Σ
-
=
(9)
as well as
= + o1)).
(10)
both p and (0)) hold, we obtain
- 1
(Vnst)² (VTiS))=
300 (7)) 12) (VTTT)3)
Σ
(d(Z)), T2)² (vTTJn)
n1
n
Σ
(00) (Z,), F2)2 2
Σ
(()))
(
(r(1) + (1)
= or()),
hence
n.s 2 n + op
(11)
thus concluding step 2.
24

---

In order to prove (9) we denote - (z,) - (Z,) and consider
1
Σ (
R
(1), T2)²
n
(o(Z,), T2) = 1
Σ (0(2.) + +,,, T2)²
n
- 1 Σ
1 n
(o(Z)),
1=1
-
n
(e r2) 2 +
++2
2
(d(z), T2) (E) T2)1
11
"
IV@
1
(e (2)
Σ
| (z). T2 (eb T2)
n
i=1
1
n
c4 T2) 2 +2
Σ
(Zi), r2)*
Σ
cis T2)
n
i=1
n
(18
(E T2) 2 +2
1 n
Σ
(
(
Σ
n
(eis T2)
n2
|c 11 1
t=1
( Σ
(
1
+
| Zz)
Iee
11
1
where (i) is obtained by the triangle inequality, and (ii), (iii), (iv) are obtained by Cauchy-Schwarz inequality. We recall that ||r2l = (r()). Moreover, Øl 1 I E < co implies
2 (Z) P by the law of large numbers, ΣΣ (2, Op(1).
If was constructed independently from D1, then
PP
E
74
=P
||
[10300 011k
-11300
-012 = (1),
T
so by Markov's inequality
|e =1))
If 11 belongs Seiienko-Caantllee class, then 1| ~ol1% belongs Srivenko-Cantelll class, From this, we conclude that (P (P( 21 = og (1), hence
n
||: 2 = P..1(((1) -6
dblb
(P - PT)] - 211 + PP (P(((1)) 0
-P-- Pn) +1 + 11)0) - $112
= =op(1).
25

---

In either case,
Σ (B (2.),
1
Σ
(o)),,, T2) -
1
Σ
lez
11
122 112
71
71 mm1
$
+
1
n
(1
n
|ee |e
71
n Σ
S 11
||
11
+
12
$
+211T22133
1
Σ l
(+
1=1
or()).
Further,
Σ (5 (2)),
1
(30 (Z:), 72)²
1
(Z:), + +x 1
(Zi),
n
i=1
(Z.) x2))
(Z:), T2) (Z)). (2.)
IA3
2 n
(Z)) (2)))x²))) /
+
((2 (Z;), (2) xx)))
(Z)), x10
n
(311) (Zi), (2)) [ (2.), x(asp
+2
11
(8))
31) (2))
(Ixx²)B + 21 11 11x
71
(+ (or + (1()op())) (r),
where we obtain (i) by the triangle inequality, (ii) by Schwhy-Schwarz inequality, and (iii) by Cauchy- Schwarz inequality and grouping terms. By the triangle incquality, 1 E.' (6()) (Z.). f2)
Σn (Z(z), T2) is upper bounded by
(1))) (Zx), 12
(Z).
+
1 n
(300) (2)),
(bb (z)) )2)
which is = (1) + (1) = (1), thus holds.
In order to prove
10 we note that 8 implies (nfh)² = (nf, + (1))) Expanding the second
term we obtain
(nh, + or())) = (nff))² + (or + (n
(nfx)²² + (nffa²+ (o(()))++
(nff))² o1))
given that [nJal = (T,, < | 12 Ix = (1())=+(1) = (1)). Consequently, (n.)))²
(nfr)²² + oz (1), which implies 10 dividing by in both sides.
26

---

Details of step 3. We claim that
(n)32(2)3212) [2] Op(1).
(12)
To show this, note that
(2)( = =()())) r2) _ (CT2, T2) = = >503
where
C = E[o(Z) (z( (Ay.j 2 1) is the sequence of non-negative eigenvalues in decreasing order of the covariance operator C,
(Ug= 2 1) is the sequence of orthonormal eigenfunctions of the covariance operator C' (and basis of H. as well),
T2 = Di ررUر. being (3.. 2 1) the random coefficients of T2 with respect to the basis (ny. 1),
and is obtained from Lemma C.2 and the fact that E[ø(Z)) = 0. E (Z) E OZ) co. Based on Lemma C.4
N
so by the continuous mapping theorem for random variables separable Banach spaces (Bosq 2000 Equation 2.1
B1 = (T2, $
e1) VNN1
==1
Consequently,
fr (Z) 1 D2 =
2 2 1 N7N7.
(13)
i=1
Further, by Sauchy-Schwarz inequality, 0 < E (((Z)). S [10022))1310002 =6² [1
Σ (z) 1z)1
(((()),01) 1z)²
2
8
Σ
Cv 1z))
A
That is, the sum of non-negative eigenvalues is strictly greater than zero, so at least one of them has be strictly positive. By the non- in-iicceassin order of the eigenvalues, this implies that the first one is strictly positive i.c., X O. For any M > 0 it holds that
< 1 ) = P Σ M
P (IIEA
A <
SP
x S M
M
27

---

Given A, > 0 and 13
we also obtain
TP
A1
N
M
MX
MX
where F is the edf of xi distribution. Hence for every there exists m N such that
1
P
S
F
M
MX
2
for 2 m. which implies
17 <
< +
F
M
MX
Further, lim F(w) = O, so there exists M such that every E 0, there exist M 0 and E N such that
< 4.
Consequently, for
A
P
1 S M
<
=6
2
for 2 m,
1 M+
PP
[ 2 M¹ =P (E) S
x S
<e,
M
thus concluding
12
Details of step 4. 4.2) proved that
Given Conditions (i), (i), and (iii), Kim and Ramdas 2023
Proof of Theorem
nf
Vah
NO 1).
(14)
E D2
( D2
as well
n(Sa)2
Sh
2 1.
(15)
(Z D2
Combining and 12), we obtain
nfT
nfn + (1)
1 Jr (Z)
nfh
01)00=(11
1 Sr, ( 1 D22
nthn
o1);
V P2)
and based on (14 and Slutsky's theorem,
nf
4 1).
(16)
EBi f. 22))
Combining and 12). we obtain
n
S + (1(1)
n (
Ennis (2)
n.S
+ O1)00ppp
E[nif D2]
nSR
+ o1).
28

---

Thus based on
15 we obtain
n.S 2
1,
(
and by the continuous mapping theorem
E B (2))² D2
$1.
(17)
Vnsst
Combining 16) and 17
and based on Slutsky's theorem, we obtain
n
n nft
E
(2) S NN 1).
8%
E E
VTS
1N(0.1
31
Hence
T vath 4 N(..
Sh
29


Reviews:
[{"review_id": "avTOd5XszM", "reviewer": "Reviewer_yX9h", "summary": "The paper introduces a statistical test to determine whether the distributions of the two counterfactuals are the same. This goes beyond the well-known average treatment effect, which only tries to understand whether the means of the distributions are the same. The first work in this direction was by (Muandet et al., 2021), who introduced the Kernel Treatment Effect (KTE). However, their test statistic is degenerate, which means that they cannot use the CLT to derive an asymptotic threshold, and they need to resort to a permutations approach in order to compute the threshold. This paper introduces the AIPW-xKTE test, which generalizes KTE by including a plug-in estimator in analogy with AIPW vs IPW, and most importantly by using the same approach as the Cross MMD test from (Kim and Ramdas, 2023), which does yield a statistic with asymptotically normal distribution.\n", "questions": "- What is double robustness? It is a relevant concept in the paper, as it is mentioned twice in the contributions part of the introduction, and many more times later on. However, it is never defined.\n\n- Line 23: The plug in estimator is not well defined: what does it actually look like? In lines 222-225, the estimator appears again under a different notation (\\hat{\\beta} instead of \\hat{\\theta}). The authors currently say that “At this time, not so many choices exist for estimators…“ and they cite a work on this. It would be good to give a more detailed explanation of how the estimator \\hat{\\beta} is computed. Similarly, it would be helpful to give more insight on how \\hat{\\pi} is computed.\n\n- Theorem 3.1: What is \\hat{\\pi}? What is \\hat{\\psi}_{DR}? These quantities have not been defined before as far as I can tell.\n\n- Figure 1: Show error bars in subfigure c, to show that the discrepancy from 0.05 can be attributed to a statistical error. The key question that needs to be answered here is: how large does n need to be for the CLT to kick in and for the Type I error guarantee to hold. Without error bars, the current figure does not provide an answer to this question.\n\n- Figure 2: Although not as critical, error bars in this figure would be appreciated too.\n\n- Table 1: Show standard error.\n\n- Comparison with (Muandet et al., 2021): The authors do not show any experimental comparison with the KTE test proposed by (Muandet et al., 2021). They argue that “Due to the fact that the KTE (Muandet et al., 2021) may not be used in the observational setting, where the propensity scores are not known, there is no natural benchmark for the proposed test.” Since AIPW-xKTE makes use of estimators of propensity scores, it seems natural to me to compare it with KTE using the same propensity score estimators. Is there a reason not to do this?\n", "limitations": "No limitations", "rating": 4, "confidence": 4, "soundness": 4, "presentation": 2, "contribution": 2, "strengths": "The contribution of the paper is clear, and the authors do a reasonable job at placing it within the literature.\n", "weaknesses": "The main weakness is that the contribution is not highly novel, in that the test proposed is basically a combination of the KTE test from (Muandet et al., 2021) and the Cross MMD technique from (Kim and Ramdas, 2023). \n\nThe explanation would be more transparent if some important concepts were clearly defined. See more details in the questions section. There are some more detailed weaknesses that I also point out in the questions section.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the comments. We would like to address the following weaknesses and questions raised by the reviewer.\n\n- The main weakness is that the contribution is not highly novel, in that the test proposed is basically a combination of the KTE test from (Muandet et al., 2021) and the Cross MMD technique from (Kim and Ramdas, 2023).\n\nAlthough we presented the contribution as if it was a natural extension of previous works, we would like to highlight the nontrivial technical challenges addressed in Appendix C. The proof of Theorem 4.1, which is around 6 pages long, is completely novel, and cannot reduce to invoking any existing theorems. It combines the main idea presented in Kim and Ramdas, 2023 with a variety of techniques including kernel ideas, functional data results and causal inference results (eg: the novel Lemma C.7 and Theorem C.9) .\n\nWe highlight that the proof heavily differs from the work presented in Kim and Ramdas (2023) and any other paper that we have read on the topic. In fact, the only relevant part from their paper is Equation 14 and Equation 15 (part of step 4). We thus emphasize that the theoretical contribution of the work is far from being straightforward. We have now added a paragraph in the main paper to highlight these advances.\n\n- What is double robustness? It is a relevant concept in the paper, as it is mentioned twice in the contributions part of the introduction, and many more times later on. However, it is never defined.\n\nWe have changed line 33 from \n\"... we highlight that double-robustness is a property that is shared with many other estimators.\"\nto \n\"... we highlight that double-robustness is an intriguing property of an estimator that makes use of two models, in which the estimator is consistent even if only one of the two models is well-specified and the other may be misspecified; we refer the reader to [1] for a discussion on doubly-robust procedures.\"\n[1] Kang, Joseph DY, and Joseph L. Schafer. \"Demystifying double robustness: A comparison of alternative strategies for estimating a population mean from incomplete data.\" (2007): 523-539.\n\n- Line 23: The plug in estimator is not well defined: what does it actually look like? In lines 222-225, the estimator appears again under a different notation ($\\hat\\beta$ instead of $\\hat\\theta$). The authors currently say that “At this time, not so many choices exist for estimators…“ and they cite a work on this. It would be good to give a more detailed explanation of how the estimator $\\hat\\beta$ is computed. Similarly, it would be helpful to give more insight on how $\\hat\\pi$ is computed.\n\nThe plug in estimator is well defined (line 24), but its form depends on the choice of the regression functions (denoted $\\hat\\theta$). Throughout the work, we denote univariate or multivariate regressors by $\\hat\\theta$, and infinite-dimensional regressors (such as conditional mean embeddings) by $\\hat\\beta$, which is introduced in line 183.  Thank you for pointing out that we have not explicitly mentioned this somewhere, we now do so in Section 4 after line 179.\n\nMany different options exist for $\\hat\\theta$, $\\hat\\beta$, and $\\hat\\pi$. Linear regression, logistic regression, random forests, or neural nets may be used for $\\hat{\\theta}$ and $\\hat{\\pi}$; conditional mean embeddings or the work cited in line 225 may be used for $\\hat\\beta$. \n\n- Theorem 3.1: What is $\\hat\\pi$? What is $\\hat\\psi_{DR}$?\n\n$\\hat{\\pi}$ is defined in line 27. $\\hat\\psi_{DR}$ should be $\\hat\\psi_{AIPW}$  instead, defined in line 28 (we thank the reviewer for spotting this typo!). \n\n- Figure 1: Show error bars in subfigure c, to show that the discrepancy from 0.05 can be attributed to a statistical error. The key question that needs to be answered here is: how large does n need to be for the CLT to kick in and for the Type I error guarantee to hold. Without error bars, the current figure does not provide an answer to this question\nFigure 2: Error bars in this figure would be appreciated too\nTable 1: Show standard error.\n\nThe outcome of the test is either a 0 or a 1, hence its distribution is fully specified by its mean, which is estimated by Monte Carlo (repeating the test B times for a large B). The only error is Monte Carlo error, and we have now added error bars to show that our observations are not a result of Monte Carlo error. We uploaded a PDF with some plots in the Author Rebuttal section.\n\n- Comparison with (Muandet et al., 2021): The authors do not show any experimental comparison with the KTE test proposed by (Muandet et al., 2021). They argue that “Due to the fact that the KTE (Muandet et al., 2021) may not be used in the observational setting, where the propensity scores are not known, there is no natural benchmark for the proposed test.” Since AIPW-xKTE makes use of estimators of propensity scores, it seems natural to me to compare it with KTE using the same propensity score estimators. Is there a reason not to do this?\n\nThe test proposed in (Muandet et al., 2021) is not valid if the propensity scores are misspecified (which is usually the case when the propensity scores are unknown). In causal inference, there is a huge difference between randomized experiments (where they are known) and observational settings (where they are unknown). Plugging in estimated propensity scores for the true ones doesn’t work: that would be too easy a solution to the latter problem — this is why the AIPW estimator is not simply the IPW estimator with plug-in estimate of the propensity score: one has to “debias” IPW in some sense and also reduce its variance. Muandet’s test only works in the randomized experiments setting, while ours is designed for the much harder observational setting. Consequently, we refrained from including it in the main body of the paper, where we discuss observational studies. We included a comparison between KTE and AIPW-xKTE in Appendix A, for the specific case of having known propensity scores (experimental studies)."}, {"review_id": "CQDKExTT77", "reviewer": "Reviewer_sucV", "summary": "The paper introduces a test for the treatment effect which also takes distributional changes into account. The test strongly builds upon the recent works ( Kim and Ramdas, 2023) and ( Muandet et al. (2021)). The main novelty arises from extending the test in Kim and Ramdas, 2023 to the setting of treatment effect estimation. In comparison with previous tests (Figure 4) in the literature, the test proposed in this paper is computationally more efficient at a moderate price in power.   ", "questions": "- How does the corresponding plot for Scenario (I) look for the setting in Figure 2 and 4?\n\n- I am willing to increase my score if the authors can address the shortcomings above. ", "limitations": "See Weaknesses. ", "rating": 7, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "Testing for treatment effects is an important problem. The proposed test is computationally efficient and appears to be practically useful. \n", "weaknesses": "\n1) Figure 2 is misleading to some degree. BART and Causal Forests are estimating the mean of the treatment effect and therefore necessarily fail in scenarios III and IV.\n\n2) I am expecting a comparison against KTE ( Muandet et al. (2021)) and the AIPW extension from  Fawkes et al. (2022)  in the main text. This should come instead of the current  Figure 2. As far as I understand from Figure 4, the main contribution of this paper is not a better test in terms of power but in terms of computational run-time. This does not sufficiently come across in the main text. \n\n3) The contribution of the paper feels fairly limited and more like a straightforward extension of Kim and Ramdas, 2023. The theoretical results appear to me to follow from standard well-established arguments. Can the authors comment more on the technical challenges behind the proofs/ similarities to prior works? \n\n4) While 3) itself is not a reason for a low grade, based on this limitation I would expect a more exhaustive experimental analysis of the method demonstrating the practical usefulness and its limitations on real-world data sets. \n\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the comments. We would like to address the following weaknesses and questions raised by the reviewer.\n\n- Figure 2 is misleading to some degree. BART and Causal Forests are estimating the mean of the treatment effect and therefore necessarily fail in scenarios III and IV.\n\nGiven that our work is the first to test the KTE in observational settings, there is no algorithm to use for benchmarking, as commented in line 266. We included algorithms designed for the average treatment effect, given that we saw no other choice. And we did say in line 274: \"However, and as expected, such methods show no power if the distributions differ but have equal means\". \n\n\n- I am expecting a comparison against KTE ( Muandet et al. (2021)) and the AIPW extension from Fawkes et al. (2022) in the main text. This should come instead of the current Figure 2. As far as I understand from Figure 4, the main contribution of this paper is not a better test in terms of power but in terms of computational run-time. This does not sufficiently come across in the main text.\n\nAs explained to Reviewer 1 as well, the test presented in (Muandet et al., 2021) cannot be used with unknown propensity scores, and hence it does not apply to our problem (observational studies). We only included a comparison between KTE and AIPW-xKTE in Appendix A to show that, in the specific case of having known propensity scores (experimental studies), our test loses a little power for a large computational gain. \n\nFurther, we implemented the test presented in Fawkes (2022) -still a preprint-, but we obtained rejection rates close to 35% under the null (i.e. we were unable to control the type 1 error). This is not surprising: if you read carefully, Fawkes (2022) does not present any form of theoretical guarantee of the test. They do not claim that their test controls type-1 error, and indeed we find that it does not. Consequently, we left such a test out of the discussion in order to not be overly critical about a preprint, but we can add a note about our experiments with it.\n\nOur test is the only test with theoretical type 1 error control in observational studies (where propensity scores are not known). \n\n\n- The contribution of the paper feels fairly limited and more like a straightforward extension of Kim and Ramdas, 2023. The theoretical results appear to me to follow from standard well-established arguments. Can the authors comment more on the technical challenges behind the proofs/ similarities to prior works?\n\nAlthough we presented the contribution as if it was a natural extension of previous works, we would like to highlight the nontrivial technical challenges addressed in Appendix C. The proof of Theorem 4.1, which is around 6 pages long, is completely novel, and cannot reduce to invoking any existing theorems. It combines the main idea presented in Kim and Ramdas, 2023 with a variety of techniques including kernel ideas, functional data results and causal inference results (eg: the novel Lemma C.7 and Theorem C.9) .\n\nWe highlight that the proof heavily differs from the work presented in Kim and Ramdas (2023) and any other paper that we have read on the topic. In fact, the only relevant part from their paper is Equation 14 and Equation 15 (part of step 4). We thus emphasize that the theoretical contribution of the work is far from being straightforward. We have now added a paragraph in the main paper to highlight these advances.\n\n\n\n- While 3) itself is not a reason for a low grade, based on this limitation I would expect a more exhaustive experimental analysis of the method demonstrating the practical usefulness and its limitations on real-world data sets.\n\nWe considered, in total, 8 different scenarios with synthetic data and 6 different scenarios with real data. Further, we highlight that it is nearly impossible to assess our test in real-life observational studies, given that we do not know the ground truth. The test may reject or accept the null hypothesis, but we have no information on whether the null hypothesis is actually true or not. These experiments vary different aspects of the considered method and competitors. But if there is a particular aspect that you would like to see better explored, we would be happy to add an experiment for it.\n\n\n- How does the corresponding plot for Scenario (I) look for the setting in Figure 2 and 4?\n\nThe Gaussian behavior (Subfigure A and Subfigure B)  is only expected with our proposed test, so those subfigures would not be interesting for the remaining tests. Subfigure C is very similar for all the tests studied (around 0.05 with some noise), which is expected given that they are all provably well calibrated (hence we refrained from including those in the paper). We can add a note about these to the paper.\n"}, {"review_id": "V3yba5wzPb", "reviewer": "Reviewer_gJy9", "summary": "The paper focuses on studying Augmented Inverse Probability Weighting (IPW) for distributions instead of means. The outline of the paper is as follows:\n\n1. The authors provide motivation for the problem.\n1. They review several tools used to solve the problem, including Maximum Mean Discrepancy, Conditional Mean Embeddings, Kernel Treatment Effect, xMMD, and the asymptotics of AIPW.\n1. The main results for AIPW in Hilbert spaces are presented, and the authors discuss practical details of the proposed test.\n1. Discussion of the experiments.", "questions": "I'd ask authors to discuss applications in which this test would be useful.", "limitations": "yes", "rating": 6, "confidence": 1, "soundness": 3, "presentation": 3, "contribution": 1, "strengths": "The paper is technical but clearly written. Authors prove non-trivial (to me) technical results, which convinces me that if I were to utilize AIPW for distributions, I would choose this particular implementation. More broadly, if I needed to test treatment effects on distributions and lacked access to propensity scores, I would opt for this test.", "weaknesses": "The main limitation of this paper is that I struggle to think of a practical scenario in which I would have an interest in testing differences in the distribution of treatment effects. While the authors briefly mention that this question arises in various applications, none of those applications are utilized in the experiments. I'm uncertain whether investigating the effect of specialist home visits on cognitive test scores, beyond an increase in the mean, is a particularly relevant question to explore.\n\nIt might be more appropriate to compare this method with conditional average treatment effect tests. I can imagine situations where there are heterogeneous treatment effects that, on average, cancel each other out but work in opposing directions within two populations.\n\nedit: see the list attached by authors.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the comments. We would like to address the following weaknesses and questions raised by the reviewer.\n\n- The main limitation of this paper is that I struggle to think of a practical scenario in which I would have an interest in testing differences in the distribution of treatment effects. I'd ask authors to discuss applications in which this test would be useful.\n\nOur work is the extension of kernel two sample tests [1] to the observational causal inference context. In non-causal settings, kernel distributional tests have had a very large impact in the ML community over the last decade (see for instance the citations of [1]). The main advantage of these tests is that they look beyond the mean effect. In causal inference, certainly testing if the average treatment effect equals zero is the most popular way to assess if a treatment is different from a placebo. But one may argue that this is not suitable in many settings, and we should call a treatment as different if (for example) the variance of the response is different from that for the placebo. Kernel treatment effects are a general way to answer these types of questions (with means and variances being special cases obtained with the linear or quadratic kernels).\n\n[1] Gretton, Arthur, et al. \"A kernel two-sample test.\" The Journal of Machine Learning Research 13.1 (2012): 723-773.\n\n- While the authors briefly mention that this question arises in various applications, none of those applications are utilized in the experiments.\n\nWe mention that the test may be used to understand whether a treatment simply shifts the distribution of the outcome or, in turn, it also affects higher order moments. This is exactly what we did when investigating the effect of specialist home visits on cognitive test scores.\n\n\n- I'm uncertain whether investigating the effect of specialist home visits on cognitive test scores, beyond an increase in the mean, is a particularly relevant question to explore.\n\nIf a treatment simply shifts the distribution of the outcome (in this case, specialist home visits increase cognitive test scores on average) but the higher order moments are unchanged, we can conclude that the treatment is always desired. Otherwise, the treatment may shift the mean but hugely increase the variance, for instance, which may be harmful for the children that are negatively affected by the treatment. \n\n\n- It might be more appropriate to compare this method with conditional average treatment effect tests. I can imagine situations where there are heterogeneous treatment effects that, on average, cancel each other out but work in opposing directions within two populations.\n\nOur test does not attempt to estimate conditional average treatment effects of any form, so we believe that the comparison would not be suitable. Tests for conditional average treatment effects are interesting, but complementary to the current paper.\n"}, {"review_id": "GiRQEG1EAS", "reviewer": "Reviewer_NqCE", "summary": "The paper proposes a test of the null hypothesis that a binary treatment has no effect on the the potential outcome distribution. The test combines ideas of kernel mean embedding, double robustness, and cross U-statistics.", "questions": "The authors write “We were unable to control the type 1 error of the test presented in Fawkes (2022)…” What does this mean? Why not include Fawkes (2022), the most closely related work, in the simulations?\n\nThe authors show computational efficiency, but how about statistical efficiency?\n\nIn inequality (iii) of line 656, shouldn’t there be a 2 on the last term?\n\nI had some issues with the proof of Step 3. Shouldn’t the final expression have lambda_1^2 in (13)?  This correction would continue throughout the proof. At the bottom of page 27, how does the previous display imply lambda_1>0?", "limitations": "There are no issues.", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "Originality\n\n-The connection between kernel embeddings of effect distributions and the cross U-statistic appears to be new. \n\n-The main difference from Shekhar et al. (2022), who combine kernel mean embedding and cross U-statistics, appears to be the connection to double robustness.\n\n-The main difference from Fawkes et al (2022), who combine kernel mean embedding and double robustness, appears to be the connection to cross U-statistics.\n\n-For the kernel embedding of the potential outcome distribution, Muandet et al. (2021) use an IPW-style estimator while Singh et al. (2020) use a regression-style estimator. Similar to Fawkes et al. (2022), this paper combines IPW and regression estimators into a doubly robust AIPW estimator.\n\nQuality - The results are clear and appear to be correct, with some minor comments given below.\n\nClarity - The paper is well written, especially its appendix.\n\nSignificance - Ultimately, this is a paper that combines building blocks that have been partially combined before. The combination is well executed, and contributes to the literature.", "weaknesses": "I will raise the score if these items are addressed.\n\n\n\nStatistical concepts\n\n-The paper advertises efficiency, which when discussing tests, refers to certain statistical properties. However, the efficiency being described is computational by avoiding permutations. The framing should clarify this.\n\n-Asymptotic equicontinuity and Glivenko-Cantelli class are not well explained in the main text. The former is well explained in the appendix, so a pointer would suffice. The latter is not; please provide more explanation of what this condition means and why it is reasonable in this context.\n\n-Some statements about the asymptotic variance are too strong or poorly worded: on line 19 “the asymptotic variance…” and line  164 “the asymptotic variance…” \n\nComparisons\n\n-The references given for average treatment effect are actually for the local average treatment effect on line 21. Please update here and elsewhere.\n\n-The doubly robust kernel mean embedding estimator can be viewed as augmenting IPW (Muandet et al. 2021) with regression (Singh et al. 2020) approaches to kernel mean embeddings of potential outcome distributions, just as AIPW augments IPW with regression approaches to treatment effects. It would be worthwhile to point this out.\n\n-It would be good to see brief comparisons to Shekhar et al. (2022) and Fawkes et al (2022) following Theorem 4.1.\n\nNotation\n\n-Sometimes notation is overloaded, which is unnecessary and a bit confusing. For example, mu refers to a kernel mean embedding, a regression, and something else in the appendix. \n\n-Another notation issue is that the norm for beta is not defined in Theorem 4.1, and the cross fitting is poorly explained compared to Algorithm 1. \n\n-It is not a good notation choice to write k(w,y) when x and y are variables with specific meanings in the paper.\n\n-Finally, replace O(100n^2) with O(Bn^2).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the comments. We would like to address the following weaknesses and questions raised by the reviewer.\n- 1\n\nWe have replaced the word \"efficient\" by \"computationally efficient\" in lines 7 and 307 to clarify that we are referring to computational efficiency. \n- 2\n\nWe have replaced lines 190-191 by \"…we ought to refer to asymptotically equicontinuous empirical processes (Park and Muandet, 2023) and Glivenko-Cantelli classes. We refer the reader to Appendix C for a presentation of such concepts, clarification of the norms used, and the proof of the following theorem.\"\nand we have included the following definition of a Glivenko-Cantelli class in the same appendix:\n\nDefinition (Glivenko-Cantelli). We say that a class of integrable real-valued functions \\mathcal{F} is a Glivenko-Cantelli class for \\mathbb{P} if\n\\begin{equation*}\nsup_{f \\in \\mathcal{F}} \\| \\frac{1}{n} \\sum_{i=1}^n f(X_i) - \\mathbb{E}[f(X)] \\|\n\\end{equation*}\nconverges to zero in probability as $n \\to \\infty$.\n\n\nWe highlight that the Glivenko-Cantelli condition is required on the squared norm of the estimator (line 195), which is a scalar, hence we are dealing with the classical Glivenko-Cantelli concept. This condition appears to be a light assumption, as it is frequently satisfied by parametric function classes in low dimensional settings, and we are imposing it on the norm of the estimator, hence retrieving a one-dimensional setting. It is used to control the asymptotic behavior of the residuals in line 654. \n- 3\n\nWe have replaced line 29 by \"Under certain conditions (e.g., consistent nuisance estimation at $n^{-1/4}$ rates), the asymptotic mean squared error of the AIPW estimator is smaller than that of the IPW and PI estimator, and minimax optimal in a local asymptotic sense (Kennedy, 2020), hence... \", with the following cite:\nKennedy, Edward H. \"Semiparametric doubly robust targeted double machine learning: a review.\" arXiv preprint arXiv:2203.06469 (2022).\n\nWe have replaced lines 164-166 by \"Under certain conditions (e.g., consistent nuisance estimation at n^{-1/4} rates), the asymptotic variance of the AIPW estimator is minimized for $μ^1 = μ_1, μ^0 = μ_0$, thus the IPW estimator is generally dominated by the AIPW if $μ^1, μ^0$ are consistent.\"\n- 4\n\nWe have replaced the Angrist and Imbens (1995) reference by \"Hernán MA, Robins JM (2020). Causal Inference: What If. Boca Raton: Chapman & Hall/CRC.\"\n\nWe understand that Imbens (2004) may still be appropriate, given that it is mainly about ATEs/ATTs, but we are happy to include a different reference if you believe there are better suited options. \n- 5\n\nThanks! We have included \"In fact, the doubly robust kernel mean embedding estimator may be viewed as an augmented version of the KTE (which is a kernelized IPW) using regression approaches to kernel mean embeddings (Singh et al. 2020), just as AIPW augments IPW with regression approaches\" in line 214.\n- 6\n\nWe have included \"Note that the proposed statistic is, at heart, a two sample test (with a nontrivial causal twist); in contrast to Shekhar et al. (2022), the two samples are not independent and are potentially confounded.\" in line 222. \n\nWe refrained from including Fawkes et al (2022) in the discussion given that we were unable to control the type 1 error of their test (see comment below). They did not present any theoretical guarantee of their current test, and it is still a preprint version -so we understand their method is subject to change-.\n- 7\n\nWe agree, sorry about that. We have changed the $\\mu$'s in section 3.4 to $\\theta$'s, in accordance with the introduction. We have changed the $\\mu$_1 and $\\mu$_2 (and $\\hat \\mu_1$ and $\\hat \\mu_2$) from Appendix C to $\\tau_1$ and $\\tau_2$  (and $\\hat \\tau_1$ and $\\hat \\tau_2$). Consequently, the $\\mu$ will solely refer to kernel mean embeddings. We have further changed the generic $\\phi$'s in Appendix C for $\\omega$'s so that $\\phi$ only refers to the AIPW estimator of the mean embedding.\n- 8\n\nWith the aforementioned change in lines 190-191, we now point the reader to the appendix where we clarify the notation used. \n\nFurther, we have included the sentence \"Condition (iv) is equivalent to two-fold cross-fitting i.e. training $\\hat \\phi ^{(r)}$ on only half of the data and evaluating such an estimator on the remaining half.\" \nin line 209 before \"Condition (v)...\".\n- 9\n\nThanks, we have replaced it by $k(y, \\tilde y) = \\langle k(\\cdot, y), k(\\cdot, \\tilde y) \\rangle$ in line 176 (given that the kernel is going to be evaluated in $\\mathcal Y$).\n- 10\n\nThanks, done.\n- 11\n\nWe implemented the test presented in Fawkes (2022), but we obtained rejection rates close to 35% under the null (i.e. we were unable to control the type 1 error at the targeted 5%). They did not prove that their test controls type 1 error, and indeed we found that it does not. Consequently, we left such a test out of the discussion.\n- 12\n\nIf the actual embedding was known, the test would be minimax rate optimal against L2 alternatives -inherited from the properties of cross U-statistics, see Proposition E.1 and subsequent comments in Kim and Ramdas (2023)-. Given that we have to estimate the embeddings, the analysis is not straightforward. We refer the reviewer to the paragraph that expands from line 238 to line 245, where we comment on the statistical efficiency.\n- 13\n\nYes, we have included the 2, thanks.\n- 14\n\nIndeed, the final expression in (13) should have \\lambda_1^2. The correction continues throughout the proof without further implications. We have replaced \\lambda_1 by \\lambda_1^2 in the remaining lines of step 3 and highlighted that \\lambda_1 > 0 implies \\lambda_1^2 > 0. Thank you for spotting the typo!\n- 15\n\nWe have that the square of the sum of non-negative eigenvalues is strictly greater than zero, hence one of them has to be strictly positive, so the first one is strictly positive -by decreasing order of the eigenvalues-. We have included this remark in the proof."}, {"review_id": "8Rksg4J1Ia", "reviewer": "Reviewer_DRRe", "summary": "This paper proposes Augmented Inverse Propensity Weighted cross Kernel Treatment Test (AIPW-xKTE), which is a doubly robust test with provably valid type-I error based on kernel mean embeddings to test for distributional treatment effect. The paper has one result, Theorem 4.1, showing the asymptotic normality of the proposed test statistic, and demonstrates its performance on synthetic and real datasets. ", "questions": "The bottleneck of this procedure would probably be the estimation of kernel conditional mean embeddings, which has $n^3$ complexity? Perhaps it would be worth looking at speeding this up, through approximate kernel ridge regression methods (e.g. Nystrom method proposed in [Grunewalder et al., 2012] or FALKON in [Rudi et al., 2017]). ", "limitations": "The conclusion section has listed a few of its limitations and interesting possible directions for future research. ", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "The paper has one clear goal, i.e. to provide a testing procedure for distributional treatment effect. As the authors mention, distributional treatment effect has been an important topic of research for a while, and their proposal, AIPW-xKTE, has several advantages over the previous methods, most prominently that it has an analytical asymptotic null distribution, circumventing the need for permutation to get the null distribution, as well as being doubly robust. \n\nThe paper is very clearly written, setting out its goal in the backdrop of previous works and carrying out that goal with minimal fuss. The paper only offers one result, Theorem 4.1, but I think its conciseness is its value. I haven't gone through every detail of the proof in the appendix but a scan convinced me of its soundness. The paper was a pleasure to read. ", "weaknesses": "Perhaps one thing to count against this paper is its lack of novelty, in that its proposal is not something completely novel, rather it combines two ideas, namely cross U-statistics and augmented inverse probability weighting. However, it combines them to good effect and conducts thorough analysis of it, both theoretically and empirically, and I do not think this should count heavily against the paper. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the comments.\n\n- The bottleneck of this procedure would probably be the estimation of kernel conditional mean embeddings, which has n3 complexity? Perhaps it would be worth looking at speeding this up, through approximate kernel ridge regression methods.\n\nWe agree that looking at approximate kernel ridge regression methods for faster approximation of kernel conditional mean embeddings would be an interesting point to consider in future work. \n"}]

Author rebuttal:
We upload a PDF containing the changes in Figure 1, Figure 2 and Table 1, now with error bars / standard errors, suggested by one of the reviewers. We have included error bars in the remaining figures of the paper, not shown in here for space constraints. 

Meta-review: The paper makes timely contribution by expanding the field of MMD based testing for usage of testing in causal inference problems and provides an elegant doubly robust solution without needing permutations. The reviewers agreed and had a consensus towards accept.
Decision: Accept (poster)