Doubly Robust Kernel Statistics for Testing Distributional Treatment Effects
Authors: Jake Fawkes
Robert Hu
Robin J. Evans
Dino Sejdinovic
Venue: Accepted by TMLR ()

Abstract: With the widespread application of causal inference, it is increasingly important to have tools which can test for the presence of causal effects in a diverse array of circumstances. In this vein we focus on the problem of testing for \emph{distributional} causal effects, where the treatment affects not just the mean, but also higher order moments of the distribution, as well as multidimensional or structured outcomes. We build upon a previously introduced framework, Counterfactual Mean Embeddings, for representing causal distributions within Reproducing Kernel Hilbert Spaces (RKHS) by proposing new, improved, estimators for the distributional embeddings.  These improved estimators are inspired by doubly robust estimators of the causal mean, using a similar form within the kernel space. We analyse these estimators, proving they retain the doubly robust property and have improved convergence rates compared to the original estimators. This leads to new permutation-based tests for distributional causal effects, by constructing the test statistics based on the estimators we propose. We experimentally and theoretically demonstrate the validity of our tests.

Doubly Robust Kernel Statistics for Testing Distributional Treatment Effects
Jake Fawkes
jake.ffoukeesstt-uggs.oo.o.o
Deportment of Statistics University of Orford
Robert Hu Amazon!
robyhuu@hrmazznncco.k
Robin J. Evans
cvans@ssaass.or wwt.ts............
Department of Statistics University of Oxford
Dino Sejdinovic
dino. beijoseediooviiccc seridinooiiiaddelaiddee Wadelaide. edu
School of Mathematical Sciences University of Adelaide
Abstract
With the widespread application of causal inference, is increasingly important to have tools which can test for the presence of causal effects a diverse array of circumstances. In this vein we focus on the problem of testing for distributional causal effects, where the treatment affects not just the mean, but also higher order moments of the distribution, as well as multidimensional or structured outcomes. We build upon previously introduced framework, Counterfactual Mcan Embeddings, for representing causal distributions within Reproducing Kernel Hilbert Spaces (RKHS) by proposing new, improved, estimators for the distributional embeddings. These improved estimators are inspired by doubly robust estimators of the causal mean, using a similar form within the kernel space. We analyse these estimators, proving they retain the doubly robust property and have improved convergence rates compared to the original estimators. We then the proposed estimators test statistics in permutation based test for distributional causal effects. Finally, experimentally and theoretically demonstrate the validity of these tests.
1 Introduction
In this work we focus on the problem of testing for distributional treatment effects (Bellot. van der Schaar 2021 Park et al., 2021; Chikahara al. 2022), where the aim is to test for causal effect which manifest something other than mcan shift. This can be especially useful when the target variable is high dimensional or utrdctured as a network, since in these cases there is no natural mean to compare. Our contributions are as follows:
We introduce new estimators to be used within the Counterfactual Mean Embeddings framework, based upon the doubly robust estimator the causal mcan from somi-parametric statistics (Bang & Robins 2005 Tsiatis 2006). These may be applied estimate kernelised versions of the average treatment effect and effect of treatment
the treated. We prove these estimators inherit the double robustness properties of well established semi-parame parametriii estimators of causal effects and that they converge to the correct value if either of the models underlying them converge to the true model. This shows that they have theoretically improved convergence results when compared to the previous counterfactual mean embedding ostimators.
Work mainly done while the authors were with the Department of Statistics, University of Oxford.

---

2. We apply these new estimators to permutation testing for distributional causal effects. We propose a new permulation approach which allows for doubly robust trainable statistics to be used within permutation testing and prove that these tests are valid.
3. We experimentally validate the performance of our test on synthetic, semi- synthetic and real data. An implementation of our approach can be found at: /ttp www.aa/wwwaews/////////// /DR distributional test.
1.1 Related work
Our work builds upon the counterfactual mean embeddings framework introduced in Muandet et al. (2021) which we detail further within Section 2.3 This falls under the general area of applying kernels to test for distributional causal effects of which Bellot & van der Schaar (2021) is an early example, whose test statistic arises as special case of our own for the distributional effect of treatment on the treated. In a concurrent work, Martinez-Tabonat
et al. (2023) develop a similar approach using doubly robust statistics (Robins & Rotnitzky 1995) within kernel spaces. However, by applying the work of [Shekhar et al. (2022) they are able to take a permutation free approach to testing distributional causal effects. Due to the concurrency and current lack of publicly available code we do not compare against this method. This work has also been built. to estimate counterfactual densities in Martinez- alartine--Taboada & Kennedy (2023). In more general applications of kernel spaces to testing distributional causal effects/Prk et al. (2021) ) develop a statistic targeting the conditional average treatment effect (CATE) using conditional mean embeddings to estimate the RKHS distance between the expected potential outcomes. Chikahara el al. (2022) use tests for distributional causal effects to find which features are relevant to difference in treatment effects. Testing for causal distributional effects falls within long tradition of using kernel spaces for hypothesis testing as they faithfully capture all features of distribution (Gretton et. al. 2005 2012). Kernel methods have also been widely applied within
the larger causality literature. For example, instrumental variables (Singh et al. 2019 Muandet et al.
2020), to causal learning with proxies and unmeasured confounders (Singh et al. 2020 Mastouri et al. 2021 and to the orientation of causal edges (Mitrovic et al. 2018
2
Background and Notation
2.1 Causal Set Up
Throughout, we will let Y denote the outcome, which is alfected by a binary treatment T in the presence of additional covariates X. We will use the the
potential outcome notation [Rbbin] 1997) so that (t) corresponds outcome observed when T is given the value t. We assume that the causal relationships between these variables are given by the basic confounding DAG in Figure
X
T
The Propensity Score and Overlap Throughout will make reference to the propensity score, which is the probability of treatment given a set of covariates: P(T== = X=
Figure 1: Assumed DAG
= x). For a flexible notation we will use e(x, to denote P(T = X = x)) however as we are using binary treatment we will have that e(x, t) = - e(x, t')
1-e(x,)) i' We will also make use of the inverse propensity odds which we will denote w(xx() e(x,1) Finally ((x, and (a, t) will be used to denote finite sample estimates of these quantitics.
An important assumption for causal inference from observational data is that of overlap. This ensures that there are treated and untreated individuals we can compare and it written formally as:
Definition 1. We say there is overlap for T overlap is two-sided if it holds for both t and one-sided otherwise. Any overlap is strict if there is some 8 >>0 0 such that 6 S e(X,t) again with probability 1.
we have 0 < e(X,1) with probability 1. We say the
Under the one-sided overlap assumption that e(z,, 6) > 0 and the causal DAG in Figure 1 the distribution of the interventional quantity (t) is uniquely identified PPaal 2009
Richardson & Robins 2013 from
We make of the SWIG framework combine causal graphical models with potential outcomes. More details can be found Richardson and Rabins Richardson & Robins 2013
2

---

observational data as follows:
((Y(1) y) Ex(x (P(Y = X,7 = =
This allows us to estimate causal quantities of interest from observational distributions alone, such as the average Trealment effect (ATE):
(1) Y
It important to note that both-sided overlap is required for the identification of the averago treatment effect, as each interventional distribution requires overlap at different T= for identification. In the case where we only likely to have one-sided overlap, such when experimental new medical treatment is only given to the most severe cases, practitioners instead focus on the effect of treatment the treated (ETT):
-
This quantity is identified from observational data whenever e(x, 0) > 0 for almost all æ, so all individuals have some probability of not being treated. This allows us to stiill get some estimate of treatment efficacy cases where both-sided overlap violated. It is also worth noting that overlap is testable assumption, with Lest developed for example in Lei et al. (2021).
2.2 Kernel Background
We use kernel embeddings of distributions to construct our test statistics and so now briefly introduce the relevant background material and notation. For a more thorough engagement with this material we refer the reader to Muandet et al. (2017).
Reproducing Kernel Hilbert Spaces Let X' be some non-empty space. A real-valued RKHS, (Hx, (1 xxx), is complete inner product space of functions f X + IR such that the evaluation function is continuous for all x E X. Due to the Riesz representer theorem we have that for all x E X there is a function kg E H.x which satisfies the reproducing property, that f(x) = (f. ka) for all f E H4. The function k(x, x²) (k2, ) known as the reproducing kernel for the space Hxx Conversely, via the Moore-A Aronszajn Theorem (Aronzzajn] 1950), any symmetric positive definite function K on X defines a unique RKHS. We will suppose access to RKHS on X and y denoted by H.x and Hy, with with kernels l and e respectively.
Further, for random variable X X with distribution Px we can define the mean embedding of X as:
px = Exx[px(2 [k.
Under the intergrability condition that fx k(z x)dPx (x) < oo we have that px E Hxx If we have two random variables, X, X', over X' with distributions Px, Px respoctively, with a slight abuse of notation we denote the marinum mean discrepuncy (MMD) by:
MMD[X, X", Hx] == ||rrx - 1r
The kernel i is known characteristic if MMD[X, X', Hx. = 0 if and only if Px A Px. Many popular kernels such the Gaussian and Matérn are charecteristic kernels. The MMD between two distributions can be estimated from finite samples by computing the MMD between the empirical distributions. This was shown in Gretton et al. (0012) to convorge to the true MMD at the paramteric convergence rate, Op 7). For this reason the MMD is often used for efficiently testing equality of distributions.
Conditional Mean Embedding RKHSs also allow us to represent conditional distributions through the Conditional Mean Embedding (CME) Song et al. 2009 Muandet et al., 2017) given by:
Ny|x== ET((Y.)) X x].
3

---

In order to estimate this Grünewälder et al. (2012 propose to take regression point of view, seeing the CME as the solution to the following regression problem:
C = arg Big H Porx |I((Y))) Ck(X,
MY|X== C² (z,)
where B2 Hy) is the space of Hilbert-Schmidt operators Hx - Hy. This interpretation leads to an estimate of the CME from a finite dataset, D = {Ti, Vi as:
C = arg min
[e(vs,)) Ck(a, -1 7/0
(Hx,
(1)
Wk
P Wk(a)
where l = ()( (ki(r;, Wi-1 and = (K, for K = (k(r;, xy)) We will often make use of the CME conditional on a particular value of T, so py|x=a,T=t In that case we let b,, W, and k, denote the respective matrices used in the estimation of (xxX==oT==-
2.3 Counterfactual Mean Embeddings
Our work builds upon the counterfactual mean embeddings framework of Muandet et al. (2021) J. They demonstrate that under the causal structure in Section 2.1 and with the one-sided overlap e( t) > 0, the mcan embodding of the potential outcome, (t), can be written as:
T(xY,) - e(X,1)
(Y = E
which may be seen as the kernel equivalent of the inverse probability weighting estimator for the causal mean. Given finite sample, ((4,, y=) and access to the true propensity score, this quantity can be estimated
Ely,, Alt, =t e(x;, 1)
(Y
Muandet et al. (2021] prove that this converges to the true embedding at rate Op(n If we have both-sided overlap both (r(1) and (oy(0) are therefore identified from data and we can use MMD[Y ( Hyl to measure any distributional effects of treatment. Furthermore, the kernel is characteristic we have that MMD[Y YO0), Hyy if and only if (()) L Y Finally, this quantity can be estimated from finite samples at rate Op(n à) by:
MMD (1), ((), Hy) = - ()(()))
This leads to a permutation test. for distributional causal effects following the conditional permutation scheme for testing for causal effects (Rosenbaum 1984). and using the estimated MMD between the potential outcomes as test statistic. Throughout we will refer to MMD(Y (O), Hly as the distributional average treatment effect (DATE), due to the similarity between this and the average treatment effect statistic Scction 211
Analogously to the average treatment effect, the distributional average treatment effect is only identified from data under both-sided overlap. Therefore we now introduce second target based of the effect of treatment the treated, the distributional effect of treatment the treated (DETT):
(1))
Hy] = || - ([00))=
Estimation this quantity requires estimation of the embeddings 1) P(Y T T= estimate the latter embedding using observed samples of Y for individuals with
Since P(Y T
4

---

T= 1. This means the only challenge is the estimation of ((Y(0)T-1) which Muandet esimtate using the fact that:
(0) = Ep(x/==)) [my|,,==)]
This means a finite sample estimate of the conditional mean embedding from Section 2.2 can be used to compute a finite sample estimate of the mean embedding.
Alternatively, Bellot & van der Schaar 20211 also target the distributional effect of treatment on the treated using the fact. that:
-- = [(e, (T = 0)w(X.0))
Both of these estimators lead to test statistics based on the distributional effect of trcatment on the treated.
3 Doubly Robust Counterfactual Mean Embeddings
In this section we build on previous methodology by considering estimators for the causal mean embeddings that are built on doubly robust (DR) estimators of causal effects (Robins & Rotnitzky 1995). We propose new estimators for the distributional average treatment effect and the distributional effect of treatment on the treated. We prove that both of these estimators have the doubly robust. properly and so they will converge to the true causal mean embedding if either of the two models underlying them converges to the true model. We call this set of approaches Doubly Robust Counterfactual Mean Embeddings.
3.1 Doubly robust estimator of the distributional average treatment effect (DATE)
Firstly, based the doubly robust estimator of the average treatment effect we note that the embedding of the potential outcome, (t), can also be written as:
Ar(O B[((Y - -YIx,T= +
1[T = (4(X,-) - -YIXX
E
+ +l
c(X,t)
(] (((Y))) -MYIXII)) + e(X,t)
E
This a kernelised version of the doubly robust estimator the treatment mean (Roiins & Rotnitzky 1995 it can be estimated from finite samples by fitting models ē(x, 1), N= on a training set and then averaging them over the test set as:
1
Llts = t] (e(u,)))) fuy Tx1 + ArIX=MNTEE
(2)
e(x;,t)
Take a constant train/test split and let (X,t) .) and (n)
Py|x,T= be the models that arise from a total sample of size n. Now under the assumption that the propensity and estimated propensity are uniformly bounded, the following theorem shows that (t) has the doubly robust property that it converges to the true embedding as long as either of the propensity model or the conditional mean embedding converges. Furthermore, it also doubly rate robust, in the sense that it achieves a convergence rate that is the product of the convergence rates of these working models.
Theorem 1. Assuming that e(x,t) is strongly bounded away from zero, well as additional overlop assumptions in Appendix A.1 we have that if the estimators en(X, and (mm) prjx,T== satisfy the following:
|| (X,t) e(X, = Op(Ten))
M MYXXT==
Op((tt))
Hy
with Je,n (() and Tn. = 0(1). Then:
| -vP()) HY
=p(max - Vonnt.
5

---

Therefore, if Ye, Onn=)) we obtain the parametric convergence rate.
These statistics can be used to form a doubly robust estimator of the distributional average treatment effect as:
MMDDR (1), (O), Hy] = 1|2P8,
DR Py(0)
We derive the closed form of this statistic squared in Appendix B.1 This will converge to the correct MMD if both (1) and (Y( converge to the true embedding at rate that is maximum of both their rates.
3.2 Doubly robust estimator of the Distributional Effect of Treatment on the Treated (DETT)
We now turn to estimating the distributional effect of treatment on the treated 2 again we form a kernel version the standard doubly robust estimator of the distributional effect of treatment on the treated MModiie et al. 2018). This leads from the fact that the mean embedding of the counterfactual distribution (tY(t) =') can be written as:
PY (t) -E) py X,T= TT +'I BluyiX,== T
((( =t)
(((X,))P(T= =t) (((Y,-)- -xixx,=)) TT= + ElprIx.T=: [T= P(T -) =(T w(X.t) - IT= +Elpyxx, | TT=t), E
where w(x, c(x
c(x,1)
By fitting models (xx t) and Ay|x=s,7=t on training samples we get a finite sample estimate of the distributional effect of treatment on the treated as:
Σ (1tt= (xx, (4(y)-) AVIx==,TT=) 1(t; t'} prix=
n
DR PYOIT==
Th
(a) where ni Att t}. Again letting én (2, t) and PrixT== be the respective models trained on of
size n we have that:
Theorem 2. Under overlap assumptions given in the appendir, we have that if the estimators (x(X, t) and P satisfy the following:
|1 (X,t)
Op(Ten):
- - -YIXT== = Op(rro))
where Te.n O1) and Tr. = (1) then:
DR | - PYOJT== Hy
Op(max(n - Ne..))
Applying this we can now estimate the relevant MMD for between treated and untreated for this population
DR PYOOT== 1 n E1tt ltt = t') ((m.)))))))
MMD (1) ((0) My]
np
Hy
Which has the same convergence rate as DR (0 Therefore, we only need strict one-sided overlap for convergence, ú DR PYinTT= just requires e(x, 1) > Again we derive the squared statistic and give it in closed
form in Appendix B.2
We leave the arbitrary, so if this would form effect of treatment the control. For simplicity do not distinguish
the cases.
3 Any model for wfxx t) leads model for e(x,t) directly.

---

4 Permutation testing for Distributional Treatment Effects
We now apply both test statisties to the problem of testing for distributional treatment effects, taking permutation based approach. As is standard in this context we test against Fisher's ' sharp null FFisher 1936 Rosenbaum 2002 [4]
Ho Y((1) ((())-
In order to apply a permutation algorithm, must choose permutations such that the treatment vector is ezchangeable under them. That is, if we let be the size of our dataset D = (X, Y) and Sym, be the permutation group of size n we want to restrict to permutations, E Sym,, such that:
Y.Y.T) & (X,Y,T,),
where T, = (To()) is the permuted treatment vector.
A standard way to ensure that the permutations have this property is to throngh matching (Stuart 2010 where we split the data into matched sets, such that within each set we have one treated individual. These matched sets are formed using some measure of 'similarity" of individuals, such as the Mahalanobis distance on covariates, through the propensity score. Rosenbaum (2002) argues that if this matching is exact, that there are differences in propensity between matched individuals, then the treatment is exchangeable under permutatioos that preserve the matched sets. That is if we let M C Sym, be the subset of permulations that only permute within matched sets and sample some a E M, then T is exchangeable under a. Therefore if we fix a statistic S, in our case either the DATE or DETT statistic, and randomly sample 0i,-.., Fm from M, we have that:
$
S,,YY are exchangeable.
This allows means that we can define the p-value of the test as:
1 1+ T 1(5 Y, 2 S(,
where this is now a valid p-value in the sense that P(p S a) S a under the null. However, permuting this way creates computational challenges when S is one of the proposed statistics. Namely, the calculation of these statistics requires the fitting of propensity model and conditional mean embedding, and this would need to be repeated for every permutation.
To resolve this first form the matched sets, and then randomly split into train/test sets, DT. Dre, such that each matching set is fully contained in one of the datascts. Now let MiD,y be the set of permutations on D which leave Dre fixed and preserve the matching on Dr,, and vice versa for Dr. We then sample N random permutations {a1....,
oN from MiDryl and form the set M permutations as:
MN (a (oon: {Id) U (oI,..,,oo),, MD
Then the following demonstrates by applying results from Ramdas et al. (2023) that we can form a valid permutation test by randomly sampling permutations from M:
Proposition 1. Under eract matching, if we sample .o..... Tm from My we have that the following is valid p-vauue:
- 1+ T 1+
(T))
p
+
for any statistic and number of sampled permutations N.
We note that while the test must be formulated against Fisher's sharp mull, it will generally have power against alternatives which show distributional causal effects, so Y(1) Y(O).
7

---

This alleviates the computational challenges associated with using the DATE DETT statistics as there is now a controllable parameter N which determines how many models must be trained. Increasing N will mean training more models and so will decrease the variance in p, however it will incur greater computational cost.
To our knowledge, this is the first time this permutation strategy has appeared in the literature. Moreover, this procedure could be applied to efficiently use other trainable test statistics within permutation testing, such as standard doubly robust estimators of the causal mean more general permutation tests for assessing the performance of predictive models in Ojala & Garriga (2010).
The result in Proposition 1 relies on exact matching, which is a common assumption in proving the validity of conditional permutation tests Rosenbaum (2002). However recent results in the area have shown that p-values arising from matched permulations are approximately valid under inexact matching (Berrelt et. al., Pimentel 2022). We expect and empirically observe that similar results hold in our case.
2020
5
Experiments
5.1
Fit tests for Doubly Robust Counterfactual Mean Embeddings
To demonstrate the convergence properties implied by our theoretical results in Section we fit, our statisties on simulated data from the following data generating processes:
) " -Ex (
27
X ~ To)
p ++ exp (a ca
(1+ xx)))
1
~ Ber(p)
== x b 3T
where E ~ N o2) and the values of a, b R2 and 3, a E R are given in Appendix
This is similar to the
sctting in Muandet et al. (2021). We also simulate from:
T T Ber
X ~~ (10 (1)
Y= fr)
é N~,
where for fi 10 + IR and a' are given in the Appendix. The setting here is the same as in Bellot & van der Schaar 2021 For both scttings we a lincar logistic regression for the propensity score so that the model incorrectly specified. In first data generating process we have both-sided overlap in the true propensity and so plots (a) and (b) in Figure 2 demonstrate that the doubly robust embeddings converge for both interventional values. Due to an incorrectly specified propensity model the counterfactual mean
embeddings of Muandet et al. (2021) do not converge.
For the second data generating process, increasing the value of b creates setting where the overlap is more one-sided. As such plots (c) and (d) in Figure demonstrate that only a (1
DR and DR (Y()]T= converge quickly
to the true embeddings.
Plots of the true propensity for both simulations can be found in Appendix
5.2 Testing For Distributional Effects on Simulated Data
We now apply these statistics to the testing of distributional causal effects where the data is simulated from:
xx (x )=(1+op(axx)
X N(O, To), p 1+ exp (a
(3)
T ~ Ber(p), Y b 3(22 -
e e~ c~(100
(4)
Where a, E R and a E R are as in Section 5.1 and the variable Z is 1, a Ber(s) Uni 1]). The second two settings capture when there is a distributional causal effect which does not shift the causal mcans.
In Figure 3 we plot the rejection rate of the tests based on four statistics, the DATE and DETT statistics estimated without conditional mean embeddings, and the DATE and DETT statistics estimated with conditional mcan embeddings denoted by DR-DATE and DR-DETT respectively. The matching for all statistics is done via logistic regression and we apply the permutation from Scction 4 We compare our methods against Double Machine Learning (Chernozhukov et al. 2018) and Targeted Maximum Likelihood
8

---

Assesing fit of Doubly Robust Counterfactual Mean Embeddings on Simulated Data
RKHS distance from Thue Embeddaings
0.034
0.07
Prron
0.05
P(w1)) T-
0.010
0.05
0.008
0.04
PeR P(W11 T=
0.006
0.004
0.01
0.002
500
1000
1500
2000
3000
3500
4000
1000
1500
2000
2500
3000
3500
4000
Number of Samples
Number of Samples
(a) DATE embeddings, Both sided overlap
(b) DETT embeddings, Both sided overlap
0.007
0.0225
Aunon
0.0200
0.006
0.005
17/00
0.0175
p m-
0.0150
0.004
0.0125
0.003
0,0100
0.0075
0.002
0.0050
0.0025
500
250C
Number of Samples DATE embeddings. One sided over ap
Number Samples
(d) DETT embeddings. One sided overlap
Figure 2: These plots demonstrate the convergence of the doubly robust estimators when the propensity model is misspecified. The left plots show the embeddings required for DATE, whilst the right plots show those needed for DETT. We vary between two data generating process given in Section 5.1 aiming to show both sided and one sided overlap of the propensity score.
Estimation (ann Der Laan & Rubin 2006) as baselines to effectively pick up a shift in causal means. We run these experiments with 2000 data points, rejecting at the 0.05 significance level. Power plots showing the rejection rates over 100 re-runs of this experiment are displayed in Figure 3
In this experiment we observe similar performance for all distributional test statistics. We believe this is due to the fact that both the true and estimated propensities are linear functions of the covariates, and so all statistics will correctly fit the causal mean embeddings. Plot (a) demonstrates that when the shift in the mcan, as expected our methods will perform worse than the basclines. This is due the cost of targeting distributional effects over simply mean shifts. However, plots (b) and (c) demonstrate that when the causal effect is only distributional, our statisties can correctly reject the mull, unlike these traditional methods.
5.3 Semi-Synthetic and Real World Data
Finally, to evaluate the performance of our tests within more realistic settings we evaluate on a selection of semi-synthetic data. We evaluate on two standard semi-synthetic tasks, the infant health and development program (IDHP) introduced in Hill 2011). the linked births and deaths data (LBIDD) (Shimoni al.) 2018
As both of these datasets are semi-synthetic, we have access to the counterfactuals and so can simulate realistic data under the null hypothesis. In order to prevent problems with extreme propensity scores we remove any data for which e(z, t) 0.08 for any value of We again use logistic regression matching and weights model.
9

---

Testing stributionna Causal Effects on Simulated Date
0.6
(ooecton
DA-MTTF
0.
DETT
DML
THLE
0.00
0.05
0.10
0.15
0.20
3,25
0.30
0.35
0.43
0.0
0.2
0.4
0.0
0.2
0.4
0.5
0.5
1.0
1.2
144
1a) Shift
Shift
Figure 3: Simulated tests for distributional causal effects where the | parameter controls the size of the effect. In plot (a), 3 introduces a shift in the mean only, whereas in (b) and (c) the shift only affects higher moments of the distribution. All simulations with n 2000 samples.
Table 1: Rejection rates the 0.05 level on semi-synthctic datasets.
Dataset
Hypothesis
DATE
DR-DATE
DETT
DR-DETT
DML
Ho
0.02 +0.03
0.05 =0.04
0.01 +0.02
0.03 #0.03
0.04 #0.04
IDHP (n 747)
H1
1.00 +0
1.00 =0
1.00 #0
0.99 #0.03
1.00 #0
Ho
0.02 +0.03
0.03
=0.03
0.00 +0
0.00 +0
0.17 +0.10
LBIDD (n 1000)
H1
0.85 #0.07
0.82 =0.07
0.13 #0.06
0.08 10.06
1.00 #0
Hu
0.03 #0.03
0.03 =0.03
0.00 =0
0.00 =0
0.00 =0
LBIDD (n 2500)
H1
0.82 ++0.8
0.75 0.09
0.19 +0.08
0.13
+0.06
1.00 +0
The results can be found in Table which shows all kernel based test statistics produce valid p-values under the null hepolhesis. Further the DATE based test statistics remonst strong power against the alternative for all three datasets. The DETT test statistic on the other hand only shows good level of power on the IDHP dataset, with more samples required to reject on the LBIDD dataset.
6 Conclusion
We have proposed new doubly robust estimalors for the kernel mean embedding of causal distributions. Our theoretical experimental results show that these converge in a wider array circumstances than the original estimators of these quantitics. Further, we applied these emboddings to the problem of testing for distributional causal effects. We used permutation based approach for testing, and proposed a permutation doubly robust. statistics within permutation testing. We prove the validity of this
that allows us to permutation under exact matching and experimentally validate this approach under inexact matching. The results show that our test statistics are able to pick up causal effects that only manifest as distributional shifts, where traditional mean shift methods fail.
10

---

References
Nachman Aronszajn. Theory of reproducing kernels. Transactions of the American mathematical society, 68 (3):337 404, 1950.
Heejung Bang and James M. Robins. Doubly robust estimation in missing data and causal inference models. Biometrics, 61 (44)99622977, 2005.
Alexis Bellot and Mihaela van der Schaar. kernel two-sample test with selection bias. In Uncertainty in Artificial Intelligence, pp. 214-214. PMLR, 2021.
Thomas Berrett, Yi Wang, Rina Foygel Barber, and Richard Samworth. The conditional permutation test for independence while controlling for confounders. Journal of the Royal Statistical Society Series B: Statistical Methodology, 82(1):175- 197, 2020.
Victor Chernozhukov, Denis Chetverikov, Mert Demirer, Esther Dullo, Christian Hansen, Whitney Newey, and James Robins. Double/debiased debiased machine learning for treatment and structural parameters. The Econometrics Journal, 21(1):C1 C68, 01 2018. ISSN 1368-4221. doi: 10.1111/ectj.1208 URL https: 1111/ect. 12097
Yoichi Chikahara, Makoto Yamada, and Hisashi Kashima. Feature selection for discovering distributional treatment effect modifiers. arXiv preprinl arXiv:2206.000 005166 2022.
Ronald Aylmer Fisher. Design of experiments. British Medical Journal, (3923)):554, 1936.
Arthur Gretton, Olivier Bousquet, Alex Smola, and Bernhard Schölkopf. Mcasuring statistical dependence with hilbert-schmidt norms. In International conference on algorithmic learning theory, pp. 63-77. Springer, 2005.
Arthur Gretlon, Karsten M. Borgwardt, Malte J. Rasch, Bernhard Schölkopf, and Alexander Smola. A kernel two-sample test. The Journal of Machine Learning Research, 13(1):723-777 773, 2012.
Steffen Grünewälder, Guy Lever, Luca Baldassarre, Sam Patterson, Arthur Gretton, and Massimilano Pontil. Conditional mean embeddings regressors. In Proceedings of the 29th International Coference International Conference on Machine Learning, pp. 1803-1810, 2012.
Jennifer L Hill. Bayesian nonparametrie modeling for causal inference. Journal of Computational and Graphical Statistics, 20(1):217-240, 2011.
Edward H Kennedy, Zongming Ma, Matthow D McHugh, and Dylan S Small. Non-parametric methods for doubly robust estimation of continuous treatment effects. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 79(4):1229-122) 2017.
Lihua Lei, Alexander D' D'Amour, Peng Ding, Avi Feller, and Jasjeet Sekhon. Distribution-Iree assessment. of population overlap in observational studies. Technical report, Working paper, Stanford University, 2021.
Diego Tabooda and Fawwar H Kennedy. Counterfactual density estimation using kernel stein discrepancies. arXie preprint arXiv:2309. 16129, 2023.
Diego Martinez-aabodda, Aaditya Ramdas, and Edward H Kennedy. An efficient doubly-robust test for the kernel treatment effect. arXiv preprint NSk::2304.132311 2023.
Afsaneh Mastouri, Yuchen Zhu, Limor Gultchin, Anna Korba, Ricardo Silva, Matt Kusner, Arthur Gretton, and Krikamol Muandet. Proximal causal lcarning with kernels: Two-stage estimation and moment restriction. In International conference on machine learning, pp. 7512-7523. PMLR, 2021.
Jovana Mitrovic, Dino Sejdinovic, and Yee Whye Tch. Causal inference via kernel deviance measures. Advances in neural information processing systems, 31, 2018.
Erica EM Moodie, Olli Saarela, and David A Stephens. A doubly robust weighting estimator of the average treatment effect on the treated. Stol, 7(1):e205, 2018.
11

---

Krikamol Muandet, Kenji Fukumizu, Bharath Sriperumbudur, Bernhard Schölkopf, et al. Kernel mean embedding of distributions: A review and beyond. Foundations and Trendeee in Machine Learning, 10(1-2): 1 1-141, 2017.
Krikamol Muandet, Arash Mehrjou, Si Kai Lee, and Anant Raj. Dual instrumental variable regression. Advances in Newral Information Processing Systems, 15:2710-2721, 2020.
Krikamol Muandet, Motonobu Kanagawa, Sorawit, Saengkyongan and Sanparith Marukatat. Counterfactual mean embeddings. Journal of Machine Learning Research, 22(162):1-71, 2021.
Markus Ojala and Gemma C Garriga. Permutation tests for studying classifier performance. Journal of machine learning research, 11(6), 2010.
Junhyung Park, Uri Shalit, Bernhard Schölkopf, and Krikamol Muandet. Conditional distributional treatment effect with kernel conditional mean embeddings and u-statistic regression. In International Conference on Machine Learning, pp. 8401 8412. PMLR, 2021.
Judea Pearl. Causal inference in statistics: An overview. 2009.
Samuel D Pimentel. Covariate-udaptive randomization inference in matched designs. arXiv preprint arXiv:2207.0500 2022.
Aaditya Ramdas, Rina Foygel Barber, Emmanucl Candès, and Ryan Tibshirani. Permutation tests using arbitrary permulation distributions. Sankhya A, pp. 1 1-22, 2023.
Thomas S. Richardson and James M. Robins. Single world intervention graphs (SWIGs): A unification of the counterfactual and graphical approaches to causality. Center for the Statistics and the Social Sciences, University of Washington Series. Working Paper, 128(30):2013, 2013.
James M Robins and Andrea Rotnitzky. Semiparametric efficiency in multivariate regression models with missing data. Journal of the American Statistical Associalion, 90(429):122-155 129- 1995.
Paul R Rosenbaum. Conditional permutation tests and the propensity score in observational studies. Journal of the American Statistical Association, 79(387):565-5774 1984.
Paul R Rosenbaum. Observational studies. In Observational Studies, pp. 1-17. Springer, 2002.
Donald B Rubin. Estimating causal effects from large data sets using propensity scores. Annals of internal medicine, 127(8 Part (ett__2))7777,, 1997.
Shubhanshu Shekhar, Tmmm Kim, and Aaditya Ramdas. A permutation-fres kernel two-sample test. Advances in Neural Information Processing Systems, 35:18168 18180, 2022.
Y. Shimoni, C. Yanover, E. Karavani, and Y. Goldschmnidt. Benchmarking Framework for Performance- Evaluation of Causal Inference Analysis. ArXiv preprint arXiv: 102. 050465 2018.
Rahul Singh, Maneesh Sahani, and Arthur Gretton. Kernel instrumental variable regression. Adoances in
Neural Information Processing Systems, 32, 2019.
Rahul Singh, Liyuan Xu, and Arthur Gretton. Generalized kernel ridge regression for nonparametric structural functions and semiparametric treatment effects. arXiv e-prints, pp. arXiv -xXiv-2010, 2020.
Le Song, Jonathan Huang, Alex Smola, and Kenji Fukumizu. Hilbert space embeddings conditional distributions with applications to dynamical systems. In Proceedings of the 26th Annual International Conference on Machine Learning, pp. 961 968, 2009.
Elizabeth A Stuart. Matching methods for causal inference: A review and look forward. Statistical science: review journal of the Institute of Mathematical Statistics, 25(1):1, 2010.
Anastasios A. Tsiatis. Seméparometric Theory and Missing Data. Springer, 2006.
Mark J Van Der Laan and Daniel Rubin. Targeted maximum likelihood learning. The international journal of biostalistics, 2(1), 2006.
12

---

A Proofs
For of writing, throughout this appendix we will let:
r(z, 1) =- MY|X=x,T=t (z, = Ay|x=x,T=
(5)
(n)
(6)
A.1 Proof of Theorem [
This proof follows under the following slandard assumptions
The propensity is uniformly bounded away from zero. So there exist an > 0 such that e(x, f) > e for all x the support of the covariates.
We assume that the estimated propensity score is uniformly bounded away from and on support of P(X). So for all in the support of P(X) and for all n have 8 (x t) for some 5 >>> 0.
Now let m be the size of the test we average over. m O(n) where n the sample size.
First note due the consistency property can write k(ys, :)1 (4 rewrite the DR estimator as:
(() (t) t}. This allows
m 1
(1 = 1) - en (2;, 1)) (t), Fm (122 1 ((w.()).))
PY(O)
en (x1,1
Now letting:
m
(1ft,
(x,, (E())(t), (n()..))
DR CY())
8n (Jis:
we can write:
| 11 S pY(0) (f)))))
+
+| Hy
As En (t), is unbiased estimator for (Y()) with parametric convergence rate O(m +)= O(n (n-=))))))))))))))))))))) the convergence rate of our estimator is entirely determined by the convergence rate of (( My consider:
$),
E
(t)
Ry
1ft=tt--nff -
(1 =1 (1) (n f)) (t) fn fnl (xs, 1)e, (zzz. ()
f), vi(t) - pnl 1))
E
2
(1(4 = 1) - en (1 1) en (0, 1)) (1, (,, 1), (c()) F. (x,,
(7)
(n, e) (Ir
(1ft, én (x,, t)) (t) P. (x,, (x,, }
(8)
m
en (x(rit)
Now let (X,T.((H) be random sample. We may write the term 8
(1[T= (x,
Fx, (xXx,))llx,
m
(x,0)
5rTheee the same as for example Kennedy al. 2017
13

---

By applying bounded propensity and bounded kernels we have that this expeclation is bounded, which gives the term is bounded by Ci/n. Now for the first term let (1)) be second independent sample. We may write (7) as:
(1(T= -1 - -1 (x)))()))))))))-0000)00 -F, ex(X, t) (X,t) [1fT 1) - en (11T- - en (X, 1)) (x,0))
771
1
Y71
Ny x,
m
e. (X, (X,,
(ATT cx, 1) (1(T= = - Co (2)
171
E
E
x,
E (xx, (()) - Pn) 24r Hy
x,
m
(X, e) (X)t)
(e(X, t) - (X, (e(X, t) - (X,,t)
Y71
(r(x,)) (act)-Aa(CC -x,)(x))) ,x ()),))))x
772
én (X, 1) (X,t)
(e(x,t) - (X, t)) (e(x, - -Ea(X, - )
E ( - P,Cx,,)- - 1
E
(e, (X,1)
(c(X,))
(x(X) (xxxx)_-((x11)
111
F. [|r(x, t) - (x,
m
én (x, t)
<C2B [le(X,t)
- êx ()) +)) B
[X), [
Therefore we have:
E E||ePts
DR 2 My - C 1C [l)( (x)
ex,
This gives that the rate of convergence is controlled by (e(X,t) ex,XX 1)) and E () - (Xl My By applying Jensen's inequality we have cP0 Il Ry (Jr.aae..) and so by Markov's 's inequality ||<P88 1l Op(nnaTe.)
A.2 Proof of Theorem 2
The proof follows a similar structure to that of Theorem
1 where now have:
1
Σ 1fc { ((ou(t).))
|y - -0 My < pr nv
1=1
My
a Σ fc. (') ((u(().))
DR (O))t
14

---

Now have:
[t, = = (()(((((t)))))
(Y(B))EE
72P'
{afti = t²} (4t)) -((,, + 1ft, = t')r (2), t))
=
= -)
1
- ((
(((())(t)))) - r(a,, 1)) = + - 1
ne
1 t( (1- - 1[4 ) e(xx,,
(e(vs(t).)) ((),)))
1(4 =1]
c(2;,()
e(
1ft, = - 1ft, = t) e(x,,)))) 1(t, = t' e(x,)
(e(w:()))) r((..))) ())
((( r((x,,)) 1ft, e )
c(x2,
Ry
Now as ne / converges to a constant at rate n- 1 the convergence rate just depends on the rate that cyt)) tends zero. Therefore the analysis in the previous proof establishes the rate.
A.3 Proof of Proposition
Ramdas al. (022) demonstrate that can test the null hypothesis
X,, are exchangeable
from a fixed set of permutations S C Sym,, (not necessarily a group) using a statistic T by randomly sampling permutations ou,...,OM from S uniformly and computing:
1 EM 1 T(X (1o0)) 2 T(X)
p
M+1
where X, (Xo(z).....
...,))...... (o())) is the permuted data. Ramdas al. show that p valid p-value in the sense that P(p 2 a) - a under the null hepothesis.
In method the data is placed into bins, denoted by B,, where ranges over the total number of bins k, and under the assumption of exact matching the null hypothesis is:
Ho TB; ...., Tb,, exchangeable for each i.
Further we have split the bins into training Bn = (B....., Bm) and Br = (Bm+1..... Bx}. For B; E Br. we set the possible permutations to be Si i.e. the set permutations on B,. each set B, E BTo we sample olgs.... N++ from Sis,1 and set Sj fa1y..... "......,o++... NN+I.. We then set the total set of permutations S to be:
S,
x
X
SIB
J=m+1
As the samples in each bin are exchangeable, and samples in distinct bins are independent, we have that our data is exchangeable under the permutations S. Therefore can sample from S as in [g to produce valid p-value under Ho.
To go from this to the form in the proof note that sampling the random permutations for S,, choosing a random permulation do from Sj and then laking aı o o is equivalent to sampling N random permutations, oig...., ow and then sampling a random permutation from $ == (Id..1,.-... N.).
15

---

B Derivation of Test Statistics
B.1 Derivation of DMD (Y (1), (0)), Hyl
To derive (MMDDR YY (1), (0), Hy-])² let e(x) c(x, P(T = 1 z). We now write:
DR 1 (0
(f(v),)) -
( (1 (1-t,) (L(v), f(2es, 01) + f 01) }
(Y
A(xx,, 1)
((x;))
- 8(2.)
8, - e (xx.)
(B(y).))
e(x:)(1 - e(xx))
Now have f(z;; k. (z:) Wi4 and F(z;, 0) kg (I:) Wolg Further for ease of writing we let:
kg (ax) = <(x;)kk (ax)),
ki (z,) = (1 e(x;))kk (x:);
a(z;) = (t, e(xx))
(e(=;)(1 c(r:))
This allows us to write the above as
1
a(x.) (e(m.-) + kJ (zo) kl W1t))
and plugging this in obtain
(Y)) Hy)) = |APB) (o
Σ a(x))a(z)) (e(y).) + (xx ((x)) kI,,
Σ a(z))a(;;) (A(1),))) (z.) + kJ (z, - (T.))) it W(((())))
Σ (x,))((x)) 4
W.kk, Wol) Wik, - kJ (z.) Lo. Woku(z,))
Now we introduce some new notation: for a variable we will use subscript to denote the value of T we condition and superscript to denote if the variable is in training test. So T is the test points with T = 0. Further let. K(X,8)= (k(a,,, M the kernel matrix where rows. This allows us to write the test statistic
a 7 ((xxte,yTg , 2ding ( WoL(Y"",
K) TTTTXXXP"""TWLLEEE Wi,((TT., TT))
(ixg(e)K(X))) TT) T Wo Yo WoAKW" xx (dddagg((e)
[))g((-))((AA) xTT WIL(Y?" (X) xx ("T)W, e)
-2dia(e)) " TH WO (X x X"") e) )a.
16

---

B.2 Derivation of MMDDR (Y(1) T 0}, (Y(00) T (9,
We now derive the closed form of the estimator based on the effect of treatment on the treated for the case of a binary trealment. Again begin by deriving simpler form of the difference between embeddinggs:
A((())=0)) - P) = T= R(Y1T=O)
2
(00(,, 1) (E(yes-) P(x,, 1)) (1-1111))) (4) 1(2.11)
na Σ t.) ((v).)
no
(tr00(2,, 1) - (1 t.)) (e(vs,) f(2,, 1))
L - (-=o(x))
(e(y).)) f(z,, 1))
110
c(2)))
Now we have that f(x;,1 1)
- ki (x) Wil and we also let:
B(x;) t e(x,)
c(ax)
Then write the full Lest stat
Arrwirr= oo
Σ ((x,))(a,))
n
1
Σ ((x,)8(z,) (P(v). - (x.) (x.().,.((())
m
(L..) (z,)),
giving the full test statistic as:
87 (I(YTo, yTo) - -KK(7, T W,4(""" TT) K(XP,
TWLL """",,,""""""""""" WWW.OPPY"""KKAO X"' B.
C
Simulation Details
For the simulation in Section
we select:
b 115.0.4,033,02,,001,000 014,0444330222000004300022,,,,,,
0.2
8 3,
and for the second experiment we let:
fo(æ) =
f1(2) = =
a 0.3
σ = 0.2.
C.1 Fit Experiment Plots
Here include the propensity plots for the simulated data in the fit plots:
17

---

Propensity Distribution on Simulated Data
Percent
0.3
0.8
Propensity Distribution (P(T= (X)
Propensity Distribution (P(T = [X))
(a) Both Sided Overlap
(b) Sided Overlap
We can in the second plot with one sided overlap much higher proportion of datapoints concentrated at extreme propensities.
C.2 Simulated Distributional Test Plots
Here we include some plots for the distribution the experiments testing for causal effects on simulated data:
Observed Distribuiion Yby
Y17=1
Y17-0
Y
Y
0.1
0.00
0.00
-2
-2
Comstent Shift
Bernoukk Shift
Unftoom Shift
Causai Distributions on bbyyT
X:.3
X03
Y033
0.00
0.15
0.10
(a)
And also the distribution over propensity scores:
18

---

Propensity Distribution on Simulated Data
Percent
0.0
0.2
0.4
0.6
0.8
1.0
Propensity Distribution (P(T 1 (x))
19


Reviews:
[{"review_id": "rdOH5SKiT2", "reviewer": "Reviewer_hYPV", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\nThe extension of CMEs to double robustness is a natural and valuable contribution.\n\nTheoretical and experimental analysis is relatively solid.\n\n## Weaknesses\n\nThe theoretical analysis seems to be an adaptation of previous works. \n\nThe permutation test does not follow naturally from the kernel statistics but is directly adapted from Ramdas et al. (2023) as a way to reduce the computation. It is more like a practical compromise than a novelty. See Requested Changes for details.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Post rebuttal update: The authors' explanations and/or proposed changes largely addressed my concerns. I changed the answer of \"Claims And Evidence\" to \"Yes\".\n\nThe paper combines Counterfactual Mean Embeddings (CMEs) in machine learning and the classical ideas of doubly robust estimators in causal inference. It uses kernel mean embeddings to build kernelized versions of the classical doubly robust estimators. As to testing the null hypothesis of no treatment effects, the paper uses the recent result of Ramdas et al. (2023) to reduce the computational load.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "## Critical\n\nThe test is of limited novelty and seems to be an afterthought. Despite the statement “this procedure could be applied to efficiently use other trainable test statistics within permutation testing”, my reading (of the main text and A.3) is that, because the proposed statistics are computationally heavy to apply to naive permutation tests, the authors use Ramdas et al. (2023)’s idea which itself can be applied to general test statistics in permutation testing. If this is a misunderstanding, the authors need to discuss the original work and compare the proposed test to it. Otherwise, I request revising the claims such as “This *leads* to *new* permutation-based tests….”, “We propose a new permutation approach…” and “this is the first time this permutation strategy has appeared in the literature” to more humble ones. I also suggest changing the title to “Doubly Robust Kernel Statistics for Distributional Treatment Effects”, without stressing the test.\n\n## Good to have (or some important questions)\n\nThe definition of (one-sided) overlap is a bit deviated from the standard and would confuse inexperienced readers. For overlap, the def contains unspecified $t$, and, if we understand this as “for both $t=0,1$”, then there is no need to require “$<1$”. Also, there are 2*2=4 kinds of “one-sided overlap”, but what is used in the paper are “$e(x,0)>0$” and “$e(x,1)>0$”; I would call them “treatment/control group positivity” respectively (the term “overlap” loses its intuitive meaning if we consider only one of the group).\n\nIn Theorem 1 and 2, why do we need “$\\gamma_{e,n}=o(1)$ and $\\gamma_{r,n}=o(1)$”? Doesn’t this mean “both estimators converge“? The proofs seem to just say “the rate is controlled by the product rate of the two estimators” and we can have the conclusion without saying anything about the rates of the two estimators?\n\nI am not sure why the proposed estimators “have theoretically improved convergence rates when compared to the previous counterfactual mean embedding estimators”? As indicated in the paper, previous CME estimators also converge at the parametric rate, and, according to Theorem 1 and 2, the proposed estimators achieve no better rates than this (although they have double robustness)? \n\nApproximate validity under inexact matching. As far as I understand, in Simulation 3, the matching is exact because both the model and truth are logistic. So I guess the semi-synthetic data do not use the logistic function to generate the treatment and thus the matching is inexact? Even if so, it needs to be explained clearly and the data-generating process should be given in the Appendix.\n\n## Minor \n\nThe symbol $S$ is overloaded for both a permutation group and a statistic. I suggest the symbol $T$ for the latter.\n\nIn the caption of Fig 4, “introduces a shift in the” → “introduces a shift in the distribution”.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ou5ffw8D3M", "reviewer": "Reviewer_8X9u", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "1. In order to estimate the distributional casual effects, the authors introduce a novel estimator to improve the robustness of casual effects estimation. Different to previous works, the estimator is based on semi-parametric statistics.\n2. The theoretical analysis proved the double robustness properties of the proposed framework. Based on that, the proposed estimator can converge to a correct value.\n3. In order to verify the robustness of the proposed estimator, the authors apply it to permutation testing. The results are consistent to the expectation, and agree with the analysis in theoretical section.\n4. they conduct extensive experiments on synthetic, semi-synthetic and real-world datasets. The experimental results indeed prove their idea. The detailed implementation are presented at Github, which is helpful to causal community.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a novel framework, based on Counterfactual Mean Embeddings, for testing distributional causal effects in a diverse array of circumstances, in which a doubly robust estimator is designed from semi-parametric statistics. It is interesting to see that the estimator could represent causal distributions within Reproducing Kernel Hilbert Spaces (RKHS). In theoretical section, they analyze the estimator from kernel space, and prove that the proposed estimator can retain the doubly robust property and have improved convergence rates. The extensive experimental results verify the effectiveness of the proposed framework in estimating distributional casual effects.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. The authors introduce a one-sided overlap assumption, but in theorem 2, they did not amylase the connection with MMD. Is it helpful to obtain convergence rate or lower the convergence rate? . Because, it requires e(x,t) > \\epsilon, it seems like the strict overlap satisfied that?\n2. The authors claim that the proposed estimator is doubly robustness due to the properties of semi-parametric statistics, But it is little difficult to see the connections between them?\n3, The experiments are not solid from my perspective, especially, it lacks robustness certification experiments. \nTypos:\n analyse -> analyze\nKernelized-> kernelized\nParametric-> parametric\nPermuations-> permutations\n…\nPlease proofread the paper again and correct the typos.", "broader_impact_concerns": "nan", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "o3NnZLGXX8", "reviewer": "Reviewer_YytB", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths And Weaknesses Strengths:\n\n1.\tThe paper addresses an interesting and important question.\n\n2.\tThe paper is written in a clear and engaging manner.\n\n3.\tThe proposed method appears to perform well both theoretically and empirically.\nWeaknesses: See next section for more details.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper studies the problem of testing for distributional causal effects. There are two main contributions of the paper:\n\n1.\tThe authors make use of the counterfactual mean embeddings framework and propose an estimator for the distributional embeddings. Unlike previous literature, the proposed estimator is doubly robust. The authors establish theoretical properties for the proposed estimator. This estimator can serve as a test statistic in the testing stage.\n\n2.\tThe authors propose a permutation test that utilizes the above estimator as the test statistic. In particular, the authors employ a smart approach that helps alleviate the computational challenges in permutation tests.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I'm going to list my questions and comments here. I'm generally positive about the paper, and all the comments and questions below are intended to strengthen the work from my perspective.\n\n1.\tMore background in Section 1: It would be helpful if the authors could provide more background before delving into the technical details in the first section. Inform readers who are not familiar with causal inference more about why this problem is significant, and why previous methods may not be adequate.\n2.\tPage 4, the third equation from the bottom, contains a typo: it should be Y(0)|T=1 instead of Y(1)|T=0.\n3.\tAround equation (3) on page 7: I particularly like this idea. It is very nice and effectively addresses the issue. It would be helpful if the authors could provide more intuition about what the procedure is doing. For example, explaining why choosing N = 5 in the experiment does not undermine the power of the test would be beneficial for the readers.\n4.\tThe goal of developing a doubly robust procedure: It seems that the proposed doubly robust estimator would be particularly advantageous when the propensity score is mis-specified. However, in the context of testing, the matching step would also fail, or at least not perform well, if you use propensity score matching; then the test becomes invalid. Conversely, if the propensity score is correctly specified, then the doubly robust estimator would perform similarly to the original estimator. What is the advantage of developing a doubly robust estimator in this scenario? Furthermore, achieving double robustness reduces the effective sample size due to sample splitting. It would be beneficial if the authors could clarify these points and/or provide additional simulation studies to justify their findings.\n5.\tFigure 4 (b) and (c): The size of the test seems to exceed the 0.05 threshold, especially for the red line. Are the authors concerned about the potential for an inflated type-I error rate?\n6.\tExtreme data points: On Page 10, the authors state, \"Therefore, to prevent problems with extreme propensity scores, we remove any data for which e(x, t) < 0.03 for any value of t.\" Is this approach recommended for practitioners in general?\n7.\tImage or graph simulation: In the first paragraph, the authors motivate the problem by stating, \"This can be especially useful when the target variable is high-dimensional or structured (e.g., an image or a graph).\" It is somewhat disappointing that the simulation studies later in the paper are confined to simpler scenarios. It would enhance the paper if the authors could include simulations on image or graph data, or provide stronger motivation for these simpler settings.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: After discussion, this paper is acceptable with minor revisions, outlined below.
1. Sec 2 presumes the reader is already familiar with potential outcomes and causal terminology. Please either add a short explainer with some more background on causal inference for those not familiar with the field, or add an appropriate reference for the reader to get up to speed.
2. Please correct and clarify the following: "This leads us to propose new permutation-based tests" in the Abstract reads still as if the "doubly robust property" and/or "improved convergence rates" lead to the tests, but this is not the case.
3. Please carefully check and fix all typos (all reviewers had minor suggestions in their reviews)

Congratulations on a well-written paper!
Decision: Accept (poster)