Learning Counterfactually Invariant Predictors
Authors: Francesco Quinzan
Cecilia Casolo
Krikamol Muandet
Niki Kilbertus
Yucen Luo
Venue: Submitted to ICLR 2023 (2023)
Keywords: causality
kernel mean embeddings
counterfactual fairness
counterfactual invariance

Abstract: We propose a method to learn predictors that are invariant under counterfactual changes of certain covariates. This method is useful when the prediction target is causally influenced by covariates that should not affect the predictor output. For instance, this could prevent an object recognition model from being influenced by position, orientation, or scale of the object itself. We propose a model-agnostic regularization term based on conditional kernel mean embeddings to enforce counterfactual invariance during training. We prove the soundness of our method, which can handle mixed categorical and continuous multivariate attributes. Empirical results on synthetic and real-world data demonstrate the efficacy of our method in a variety of settings.

LEARNING COUNTERFACTUALLY INVARIANT PREDICTORS
Anonymous authors Paper under double-bling review
ABSTRACT
We propose method to learn predictors that are invariant under counterfactual changes of certain covariates. This method is useful when the prediction tar- get is causally influenced by covariates that should not affect the predictor out- put. For instance, this could prevent an object recognition model from being influenced by position, orientation, or scale of the object itself. We propose a model-agnostic regularization term based on conditional kernel mean embeddings to enforce counterfactual invariance during training. We prove the soundness of our method, which can handle mixed categorical and continuous multivariate at- tributes. Empirical results synthetic and real-world data demonstrate the effi- cacy of our method in a variety of settings.
INTRODUCTION AND RELATED WORK
Invariance, or equivariance to certain transformations data, has proven essential in numerous ap- plications of machine learning (ML), since it can Icad to better generalization capabilities Arjovsky et al. (2019); Chen et al. 20200; Bldem-Redyy & Teh (2020). For instance, in image recognition, predictions ought remain unchanged under scaling, translation,
rotation of the input image. Data augmentation is one of the earliest heuristics developed to promote this kind of invariance, that has become indispensable for training successful models like deep neural networks (DNNs) (Shorten & Khoshgoftaar (2019;; ; Xie et al. (2020). Well-known examples of certain types of "invariance by design" include convolutional neural networks (CNNs) for translation invariance Krizhevsky et al. (2122), group equivariant CNNs for other group transformations Cohen & Welling (2016), recurrent neural networks (RNNs) and transformers for sequential data Vaswani et al. (2017) DeepSetZZaheer et al. (2017) for sets, and graph neural networks (GNNs) for different types of geometric struc- tures Battaglia et al. (2018).
Many real-world applications in modern ML, however, call for arguably stronger notion of in- variance based on causality, called counterfactual invariance. This case has been made for image classification, algorithmic fairness Hardt et al. 2 Mitchell et al. (202I), robustness Bühlmann 2020), and uut-of-distribution generalization Lu et al. (2021). These applications require predic- tors exhibit invariance with respect hypothetical manipulations of the data generating process (DGP) Peters et al. (2016): Heinze-Deml et al.) (2018); RRoja-Carulla et al.) (2018); AArovsky et al. 2019 Bühlmann (2220) In image classification, for instance, we want model that "would have made the same prediction, if the object position had been different with everything else being equal"" Similarly, in algorithmic fairness Kilbertus al. (2017); Kusner al. (2017) introduce notions of interventional and counterfactual fairness, based on certain invariances in the DGP of the causal relationships between observed variables.
Counterfactual invariance has the significant advantage that it incorporates structural knowledge of the DGP. However, enforcing this notion in practice very challenging, since it is untestable in real-world observational settings, unless strong prior knowledge of the DGP is available. In- spired by problems in natural language processing (NLP), Veitch et al. (202I) provide a method to achieve counterfactual invariance based on distribution matching via the maximum mean discrep- ancy (MMD). This method enforces a necessary, but not sufficient condition of counterfactual in- variance during training. Consequently, it is unclear whether this method achieves actual invariance in practice, or just an arguably weaker proxy. Furthermore, the work by Veitch et al. (202T) only considers discrete random variables when enforcing counterfactual invariance, and it only applies to

---

specific, selected causal graphs. To overcome the aforementioned problems, propose general definition of counterfactual invariance and a novel method to enforce it. Our main contributions can be summarized as follows:
Based on a structural causal model (SCM), we provide a new definition of counterfactual invari- ance (cf. Definition 2.2) that is more general than that of Veitch et al. (202I).
We establish a connection between counterfactual invariance and conditional independence that provably sufficient for counterfactual invariance (cf. Theorem 3.2).
We propose new objective function that is composed of the loss function and on the flexible Sulbert-Schmide Conditional Independence Criterion (HSCIC) Park & Muandet (2020), to en- force counterfactual invariance in practice. Our method works well for both categorical and con- tinuous covariates and outcomes, as well as in multivariate settings.
2 PRELIMINARIES AND BACKGROUND
Counterfactual invariance. We introduce structural causal models as in Pearl (2000).
Definition 2.1 (Structural causal model (SCM)), structural causal model is a tuple (U, V, Fu)
such that U is set of background variables that are exogenous the model; V is set of observable (enlogenous) variables; F f= {Jv]ve is set of functions from (the domains of) pa(V) U to (the domain of) V, where Uy C U and pa(V) C V \, (V) such that V (p (V)paaVV, Uv); (iv) Fu is probability distribution over the domain of U. Further, the subsets vV(V) CVV (V) are
chosen such that the graph G over V where the edge V + V is in 9 if and only if V' E (V) is directed acyclic graph (DAG).
We always denote with Y V the outcome (or prediction target), and with Y predictor for that target. The predictor Y not strictly part of the SCM, because we get to tune fy. Since it takes inputs from V, we often treat it as an observed variable in the SCM. As such, it also "derives its randomness from the exogenous variables", i.e., is defined on the same probability space. Each SCM implies unique observational distribution over V (Pearl 2000). but it also entails interventional distributions. Given a variable A V. an intervention A amounts to replacing fA in with the constant function A = a. This yields new SCM, which induces the interventional distribution under intervention +- a. Similarly, we can intervene on multiple variables V 2 A <- a. We then write Y; for the outcome in the intervened SCM, also called potential outcome. Note that the interventional distribution Py;(y) (y) differs in general from the conditional distribution PY (y a).
This could for instance happen due to unobserved confounding effects 1 We can also condition on set of variables W C V in the (observational distribution of the) original SCM before performing an intervention, which we denote by PY, (y w). This is counterfactual distribution: "Given that we have observed W w, what would Y have been had we set A + a, instead of the value had
actually taken?" Note that the sets A and W need not be disjoint. We can now define counterfactual invariance.
Definition 2.2 (Counterfactual invariance). Let A. W be (not necessarily disjoint) sets of nodes in given SCM. A predictor Y counterfactually invariant in A with respect to W, if P Py:w (y w) = (yy w) almost surely, for all a, a' in the domain of A and all w in the domain of W
A counterfactually invariant predictor can be viewed as robust to changes of A. in the sense that the (conditional) nost-interventional distribution of Y does not change for different values of the intervention. Our Definition 2.2 is more general than previously considered notions of counter- factual invariance. For instance, the invariance in Definition 1.1 by Veitch et al. (202I) requires Y; Y almost surely for all a' in the domain of A. First, it does not allow to condition on observed evidence, i.c., it cannot consider "true counterfactuals" and is thus unable to promote- for
'We use PP for distributions is common in the kernel literature Muandet et al. 2021) and the potential outcome notation Y;, instead of Y do(a) for conciseness when mixing conditioning with interventions. "ith a mild abuse of notation, if W = 0 then the requirement of conditional counterfactual invariance becomes F Fy, (y) = Py. (y) almost surely, for all a, a' in the domain of A.
2

---

Second, it may appear stronger in that it asks for equality of random variables instead of equality of distributions. How- ever, (a) contrary to their definition, /eitch et al. (202T) only enforce equality of distributions in practice (via MMD), and (b) since Ya. Y. are (deterministic) functions of the same exogenous (unobserved) random variables, distributional equality is a natural choice for counterfactual invari- ance. We remark that notion of invariance has been studied by Mouli & Ribeiro (2022). In this work, the authors focus on learning classifiers that are counterfactually invariant to distribution shift using asymmetry learning. To the best of our knowledge, however, our work the first attempt to provide a general graphical criterion for invariance, which can be verified from observational data.
example- example--coutterfaceee counterfactual fairness, see Definition 3.6 Kusner al., 2017).
Kernel mean embeddings and conditional measures. Our new objective function heavily relies on kernel mean embeddings (KMEs). We now highlight the main concepts pertaining KMEs and refer the reader to Smola et al. (2007); Schölkopf et al. (2002); [Bellintt & Thomas-Agnan (201 I); Muandet et al. (2017) for more details. Fix a measurable space S with respect to o-aaleera Fey, and consider a probability measure P on the space (3/ For) ). Let H. be a reproducing kernel Hilbert space (RKHS) with bounded kernel ky: 3/ x y - R. i.e., by is such that su k(y, y) < +00. The kernel mean embedding ju of L2 is defined as the expected value of the function k(.,)) with respect to y. i.c. HM == E [k( .y]. The definition of KMEs can be extended to conditional distributions (Fukumizu et al. 2013: Grünewälder et al.) 2012 Song et al.) 2009 2013). Consider two random variables Y, Z. and denote with (x and (2),,F2) ) the respective measurable spaces. These random variables induce probability measure Py,z in the product space Gy Izz. Let Hy be a RKHS with bounded kernel (:())) on Ωr We define the KME of a conditional distribution [- z) via NY Z= = E [ (.) Z z]. Here, the expected value is taken over y. KMEs of conditional measures can be estimated from samples. To illustrate this, consider i.i.d. samples (1,,2),. ..... (yn, 2.). Denote with Ky the kernel matrix with entries [Kyk, = ky yy). Furthermore, let kz be a bounded kernel on Gdz. Then, PYIZ=z can be estimated as
AYIz== Σ th (1) (2) ()) = (Rz - nAT) -1 21),. Azl 2T (1)
where, I is the identity matrix and À is regularization parameter. Here, w x (-), the i-th entry of úx A (-), are the coefficients of kernel ridge regression (Grünewälder al. 2012)
3 COUNTERFACTUALL INVARIANT PREDICTORS
We now establish a simple graphical criterion to express counterfactual invariance as a conditional independence in the observational distribution, rendering it estimable from i.i.d. data, We first repeat the notion of blocked puths (Pearl 2000).
Definition 3.1. Consider a path x of causal graph G. A set of nodes blocks n. if contains a triple of consecutive nodes connected in one of the following ways: N; + Z + N,, N; + Z - Nj, or N; + MM + N5, with N,, & Z, Z E ZZ and neither M nor any descendent of M is in Z.
Theorem 3.2. Les G be causal diagram, and let A, W be two (not necessarily disjoint) sets of nodes in G. Let be a set of nodes that blocks all non-causa paths from A U W to Y. Then, for any SCM compatible with G, any predictor Y that satisfies Y 11 A Z is counterfactually
invariant in A with respect to W.
The proof is deferred to Appendix A Our key observation is that the valid set Z as in Theorem 3.2 acts as d-separator for certain random variables in a graph that allows reasoning about dependen- cies among pre- and post-interventional random variables. This graph simplifies the counterfactual graph by Shhpttser & Pearl (2008), and it generalizes the augmented graph structure described in Theorem by Shhptterr & Pearl (2009). We can then combine the Markov property with covariate adjustment to prove the claim. However, our proof does not rely identification of the counterfac- tual distributions (e.g., by simply applying the do-calculus (Pearl) 20000. Crucially, the existence of measurable set Z as in Theorem 3..2 does not imply the identifiabiliy of counterfactual distribu- tions Py= (y) (see Figure aa) for a counterexample). In particular, the assumptions do not rule out
A non-causal paths from A U W to Y is path connecting A U W and Y in which at least one edge
points against causal ordering. (Shpitser et al. 2012)
3

---

(a)
(b)
(c)
X
X
A
X
Figure 1: (a) An example for Theorem 3.2 where Py; (y) is unidentifiable, due to the confounding
effect denoted by the dashed double arrow. However, by Theorem 3.2 any predictor Y such that Y IL A Z with Z (X) is counterfactually invariant in A. Hence, the criterion in Theorem 3.2 is weaker than identifiability. (b)((c) Causal and anti-causal structure as in Veitch et al. (021)) The variable X is decomposed in three parts. X the part of X that is not causally influenced by A, X the part that does not causally influence Y, and X is the remaining part, that is both influenced by A and that influences Y.
hidden confounding the model. Hence, our method applicable even when (certain parameters of) PY; (y) PY.w(y) (y) cannot be learned from observational data.
Counterfactually invariant predictors. We propose to use the HSCIC(Y, A Z), to promote
counterfactual invariance by encouraging the sufficient conditional independence Y IL A Z. Given loss function L(Y), propose minimize the following loss
LTOTAL (Y) = L(Y) + HSCIC(Ý, Z)
(2)
with a parameter 2 O, regulating the trade-off between accuracy and counterfactual invariance. There is no principled way choose the "right" trade-off parameter. However, in practice we can use a standard approach, which consists of two steps: (i) learn collection of models on the Pareto frontier using different values of 7, (ii) from a collection of models choose the best one following, i.e. MMandet et al. (2021). Next, we develop and justify using the HSCIC, (see Corollary 3.5) for which HSCIC(Y, A Z) = 0 if and only if Y 11 A Z.
Hilbert-Schmidt Conditional Independence Criterion (HSCIC). Consider two random vari- ables Y and A, and denote with Fy) and (AA,FA) the respective measurable spaces. Sup- pose that we are given two RKHSs Hy. HA over the support Y and A respectively. The tensor product space 7y @ HA is defined the space of functions of the form (y)))(y,ar a) f(y)g(a), for all By and g E HA- The tensor product space yields a natural RKHS structure, with kernel & defined by k @ a. Sriperumbudur (2017) for more details on tensor product spaces. With this notation we define:
a') == kyly. K (a, a'). We refer the reader, i.e,, to Szabó &
Definition 3.3 (HSCIC, following Definition 5.3 by Park & Muandet (2020). For (sets of) random variables Y. A, Z the HSCIC between Y and A given is is defined as the real-valued random variable HSCIC(Y, A Z) = HY.AIZ Z. Here, HY,AZ is a veal-valued deterministic funcctioonn defined as Hy.Ajz(Z) = || jYIZ=z @ PA|Zzz with ||-|| the norm induced by the inner product of the tensor product space Hx @ HA.
Our Definition 3.3 differs slightly from Park & Muandet (020)) who define HSCIC using the Bochner conditional expected value. While it is functionally equivalent (with the same implemen- tation, see eq. 3b). it has the benefit of bypassing some technical assumptions required by Paar & Muandet (annndee((000)) (2020). We refer readers to Appendix Cand for comparison with previous approaches. The HSCIC has the following important property.
Theorem 3.4 (following Theorem 5.4 by Park & Muandet (2020)). the the kernel k of Hx 8 HA is characteristid HSCIC(Y, A Z) 0 almost surely if and only Y 11 A
A proof is in Appendix B We remark that "most interesting" kernels such as the Gaussian and Laplacian kernel are characteristic. Furthermore, if kernels are translation-invariin and characteris- tic, then their tensor product is also characteristic kernel (Szabó & Sriperumbudur 2017). Hence, this natural assumption is non-restrietive practice. Combining Theorems 3.2 and we can now use HSCIC to promote counterfactual invariance.
Corollary 3.5. Consider SCM with causal diagram and fix two (not necessarily disjoint) sets of nodes A. W. Let be a set of nodes in G that blocks all non-causal paths from W to
*The tensor product kernel le characteristic if the mapping PY,A + By.a [K(, ,, & a)) injective.
4

---

Y. Then, any predictor Y that satisfies HSCIC(Ý,, A Z) 0 almost surely is counterfactually invariant in A with respect to W.
We do not require A, W. Z or Y be binary categorical, major improvement over existing methods (Chiappa 2019 Xu et al. 2020), which cannot handle continuous multi-variate attributes.
Estimating the HSCIC from samples. Given n samples {(vi, a,, z:))= denote by Ky and KA the corresponding kernel matrices for Y and A (see eq. (T). We then estimate the H
i Y ( = wh Alz() (Ky o KA) U4 (IZ() 2 + (1Z () w (2)(()) (w ((XI(()AAAAZZ())
i 2 ())
(3)
Twx (
where is element-wise multiplication. The vectors zz (-), i (A))()). and Bh A A-)()) are found via kernel ridge regression. Caponnetto & Vito (007)) the convergence (and rates) of Ar ./ZZ Alz() (-)
to H& Y AA (:) under mild conditions. In practice, computing the HSCIC approximation Y AAZZZ may be time-consuming. To speed it up. matrices Kg and KA (Rahimi & Recht 2007 Avron al. 2017 We emphasize that eq. ( allows us to consistently estimate the HSCIC from observational i.idd. samples, without prior knowledge of the counterfactual distributions.
use random Fourier features to approximate the
Mcasuring counterfactual invariance. Besides predictive performance, e.g., mean squared error (MSE) for regression, our key metric of interest is the level of counterfactual invariance achieved by the predictor Y, i.e., measure of how the distribution of Y: changes for different values of a and across all conditioning values w. We quantify the overall counterfactual variance as a single scalar, the Variance of CounterFactuale (VCF):
VCF(Y) = Fw VaraarPAalyy LEy W=w (p) w)]
(4)
That is, we look how the average outcome varies with the interventional value a at conditioning value w and average this variance over w. For deterministic predictors (i.e., point estimators),
which we use in all our experiments, the variance term in eq. 4 zero if and only if counterfactual invariance holds at w for (almost) all a' in the support of PA (i.e., the prediction remains constant). Since the variance is non-negative, VCF (we often drop the argument) is then zero if and only counterfactual invariance holds almost surely. To estimate VCF in practice, we pick d datapoints (w.)kz from the observed data, and for each compute the counterfactual outcomes Y, w, for different values of a'. The inner expectation is simply the predictor output. We use empirical variances with i examples for each of the d chosen datapoints, and the empirical mean of the d variances for the outer expectation. Crucially, VCF requires access to ground-truth counterfactual distributions, which by their very nature are unavailable in practice (neither for training nor at test time). Hence, we can only assess VCF, as a direct measure of counterfactual invariance, in synthetic scenarios. We demonstrate in our experiments, that HSCIC (estimable from the observed data) empirically serves as a proxy for VCF.
Applications of counterfactual invariance. We briefly outline potential applications of counter- factual invariance, which we will subsequently study empirically on real-world data.
Image classification. Counterfactual invariance serves as strong notion of robustness in high- dimensional settings, such as image classification: "Would the truck have been classified correctly had it been winter this exact situation instead of summer?" For concrete demonstration, we will use the dSprites dataset (Matthey et al. (017)), which consists of relatively simple, yet high- dimensional, square black and white images different shapes (squares, ellipses, etc.), sizes, and orientations (rotation) in different xy typooiitinns.
Counterfactual fairness. The popular definition of counterfactual fairness (Kusner et al. 2017) is an incarnation of our counterfactual invariance (see Definition (.2). It is captured informally by the following question after receiving a consequential decision: "Would have gotten the same outcome had I been a different gender, race, age with all else being equal?". Again, we denote the outcome by Y C V and so-called protected attributes by A C V Y such as gender, race, or age protected under anti-discrimination laws (see, e.g., Barocas & Selbst (016)). Collecting all
5

---

(a)
A
A
Figure 2: (a) Assumed causal structure for the synthetic experiments (see Section 4.1 The pre- cise corresponding generative random variables are described in Appendix E (b) Assumed causal structure for the Adult dataset, where A consists of the protected attributes gender and (g).(c) Causal structure for the constructed dSprites ground truth, where A = {Pos.X), U = {Seale),
C = (Shape, Pos.Y): X {Color, Orientation), and Y {Outcome}. is unobserved.
remaining observed covariates into W becomes special case of our counterfactual invariance.
Y. the definition of counterfactual fairness clearly
Definition 3.6 (Counterfactual Fairness, Definition 5 by Kusner et al. (2017)). A predictor Y is counterfactually fair if under any context W w and A a, holds PY./W,AY W, w.
Py FY,/WAA( (y w, a), for all y and for any value a' attainable by A.
Text classification. Veitch et al. 22021) motivate the importance of counterfactual invariance in text classification tasks. We now provide a detailed comparison with our method on the (limited) causal models studied by Veitch much broader class of causal models. Specifically, we consider the causal and anti-causal structures (Figure 1 in Veitch et al. (202T). Both diagrams consist of a protected attribute Z, an observed covariate X, and the outcome Y. Veitch al. (222)) provide necessary conditions for counterfactual invariance. They prove that if Y Y almost surely, then assuming their proposed causal and anti-causal structures (see Figure 1 b-c)) it holds Y II. Z and Ý I1 Z Y, respectively. That is, in practice they enforce a consequence of the desired criterion instead of a prerequisite. Our work complements the results by Veitch et al. (2021). as shown in the following corollary, which is
al. (2021), emphasizing that our definition and method applies
direct consequence of Theorem 3.2
Corollary 3.7. Under the causal and anti-causal graph, suppose that Z and Y are not confounded. Y IL Z. it holds v.; (y) = -P (y) almost surely, for all z, z' in the domain '. Z.
We remark that the notion invariance studied by Veitch et al. 2021 is enforced on the true counterfactuals. Our definition, and Corollary 3.7 the other hand, only requires invariance of the resulting distribution, which is a weaker requirement than Veitch et al. (202T). However, we provide
experiments in Appendix F.6 show that method, based Corollary 3.7 and the method by
Veitch et al. 2021) have similar effect in practice.
EXPERIMENTS
4.1 SYNTHETIC EXPERIMENTS
We begin our empirical assessment of HSCIC synthetic datasets, where ground truth is known and systematically study how performance influenced by the dimensionality of the variables. We simulate different datasets following to the causal graphical structure in Figure aa) The datasets are composed of four sets of observed continuous variables: (i) the prediction target Y, (ii) the variable(s) want to be counterfactually invariant in A. (iii) observed covariates that mediate effects from on Y, and (iv) observed confounding variables Z. The goal is to learn a predictor Y that is counterfactually invariant in A with respect to W = = U U Z. We consider various artificially generated datasets for this example, which mainly differ in the dimension of the observed variables and their correlations. A detailed description of each dataset is deferred to Appendix E
Model choices and parameters. For all synthetic experiments, we train the model using fully connected networks (MLPs). We use the MSE loss Luse (Y) as the predictive loss L in eq. 2 for continuous outcomes Y. We generate 4k samples from the observational distribution in each setting and use an 80 to 20 train to test split. All metrics reported are on the test set. We perform hyper- parameter tuning for MLP hyperparameters based on random strategy (see Appendix for details). The HSCIC(Ý, A Z) term is computed as in eq. using Gaussian kernel with amplitude 1.0 and length scale 0.1. The regularization parameter À for the ridge regression coefficients in eq. (T) set to À= = 0.01. We set d = 1000 and k 500 in the estimation of VCF.

---

3,00 XI1"
X10
210
X10
VCF distribaution
5.25
3.00
HSCIC
140
-13
MSB
HSCIC
1,33
1400
1.30
VCF distribuution
273
1.b
2.50
1.30
00
1.35
0.0
02
04
65 055
1.0
02 04
06
055 1.
regulerization parameter
regularization parameter V
Figure 3: (Left) variance of the counterfactual distributions for 100 random datapoints with lines representing -rd-order polynomial regression and shaded areas being 95% confidence intervals. (Center) trade-off between the accuracy and counterfactual invariance. We observe that the HSCIC decreases, as the MSE increases. Vertical bars denote standard errors over 15 different random seeds. (Right) Correspondence between the HSCIC and the VCF, for increasing y Again, vertical bars denote standard errors over 15 different random seeds.
Model performance. We first perform set of experiments to study the effect of the HSCIC, and to highlight the trade-off between accuracy and counterfactual invariance. For this set of experiments. generate dataset as described in Appendix E. Figure B (left) shows the variance term of VCF for different regularization parameters "/ as a function of the values of Z (i.c., before taking the outer expectation). We observe that increasing values of y lead to consistent decrease of the variances w.r.t. the interventional value a. for increasing 7, demonstrating the expected trade-off in raw predictive performance and enforcing counterfactual invariance. Finally, Figure (right) highlights the usefulness of HSCIC as a measure of counterfactual invariance, being in strong agreement with VCF (see discussion after eq.
shows the values attained by the HSCIC and MSE
(right)
Comparison with baselines. We now compare our method against baselines in two settings, which we refer as Scenario and Scenario 2. These two settings differ in how the conditioning set Z affects the outcome Y. with Scenario 1 exhibiting higher correlation of Z with both the mediator X and the outcome (see Appendix E.2 for details). Since counterfactually invariant training has not received much attention yet, our our choice of baselines for experimental comparison is highly limited. tootthhherr with the fact Veitch et al. (20T)) in practice only enforce distribu- tional equality implies that our method subsumes theirs in the causal setting they have proposed. We benchmarked our method against Veitch et al. (202T) in the limited causal and anti-causal settings of Figure bbc- in Appendix E.6 showing that our method performs on par or better than theirs. Since counterfactual fairness is a special case of counterfactual invariance, we also compare against two methods proposed by Kusner et al. (0017) (in applicable settings). We compare to the Level (only use non-descendants of A as inputs Y) and the Level 3 (assume an additive noise model and in addition to non-descendants, only use the residuals of descendants of A after regression on A as inputs to Y) approaches of Kusner et al. (0117). We refer to these two baselines as CF1 and CF2 respectively. We summarize the results on the two scenarios Table I For suitable choice of y, our method outperforms the baseline CF2 in both MSE and VCF simultaneously. While CF1 satisfies counterfactual invariance perfectly by construction (VCF- = 0), its MSE is substantially higher than for all other methods. Our method provides to flexibly trade predictive performance for counterfactual invariance via a single tuning knob and pareto-dominates existing methods
Multi-dimension variables. We perform a third set of experiments to assess HSCIC's s perfor- mance in higher dimensions. We consider simulated datasets (described in Appendix E.3 where independently increase the dimension of Z and A in two different simulations, leaving the rest of the variables unchanged. The results in Tables for different regularization coefficients and different dimensions of A and Z demonstrate that HSCIC can handle multi-dimennional variables while maintaining performance, as counterfactual invariance is approached when y increases. We provide results where both A and Z are multivariate in Appendix E.3
4.2 HIGH-DDMENSIONAL IMAGE EXPERIMENTS
We consider the image classification task on the dSprites dataset (Matthey et al.) 2017). Since this dataset fully synthetic and labelled (we know all factors for each image), we consider a causal model as depicted in Figure c).. The full structural equations are provided in Appendix E.4 where we assume a causal graph over the determining factors of the image, and essentially look up the corresponding image in the simulated dataset. This experiment is particularly challenging due to the mixed categorical and continuous variables in C (shape, y-pos) and X (color, ori sentation), continuous A (x-pos). All variables except for scale are assumed be ob-
7

---

Table 1: Performance of the HSCIC against baselines CF1 and CF2 on two synthetic datasets
(see Appendix E.2 Notably, for y 5 in Scenario I and y = 13 in Scenario 2 we outperform CF2
in MSE and VCF simultaneously.
Scenario
Scenario 2
MSE x10
HSCIC x10
VCFx10
MSEx x 100
HSCIC
VCF
x10
Y 0
0.36 + 0.50
1600 +
20.33 + 1.00
2.00 0.10
1700 = 20
239.0 0.4
Y 5
17.00 + 0.03
877 3
6.84 + 1.00
43.00
1410 80
230.0 = 6.0
y = 10
19.80 2.00
8
5.89 0.60
50.00 =
1210 = 50
190.0 = 3.0
= 13
22.00 + 1.00
790
5.78 0.50
133.00 1 6.00
990 - 30
157.0 = 7.0
CFI
24.44 3.00
0.00
218.00 = 10.00
80 00
0.0
CF2
19.50 + 2.00
1400 + 10
7.50 + 1.00
137.00 + 4.00
1770 = 30
167.0 = 2.0
Table 2: Results of the MSE (x 10 for readability), HSCIC, VCF for increasing dimension of A (top) and Z (bottom), on synthetic datasets as Appendix E.3 dimensional in both cases.
All other variables are one-
dimA=2
dimA=5
dimA=10
dimA=20
MSE
HSCIC
VCF
MSF
HSCIC
VCF
MSE
HSCIC
VCF
MSE
HSCIC
VCF
35
0.0230
0.025
1.53
0.0177
0.013
10
0.0150
0.004
1.45
0.0126
0.0010
0.0214
0.022
1580
0.0157
0.010
0.0135
0.002
33
0.0115
0.0004
210
0.0200
0.022
1450
0.0144
0.008
100
0.0127
0.001
60
0.0111
0.0003
dimZ=2
dimZ=3
dimZ=5
MSE
HSCIC
VCB
MSE
HSCIC
VCF
MSE
HSCIC
VCF
0.002024
11
0.03407
53.9
0.033180
0.0062
0.002023
1.29
10-5
49
0.03406
0.016
21.9
0.033178
0.0061
20
0.002019
4.40 10
57
0.03403
0.009
18
0.033 176
0.0039
served, and all variables jointly with the actual high-dimensional image determine the outcome Y. Our goal is to learn a predictor Y that counterfactually invariant in the x-position with respect to all other observed variables. In the chosen causal structure, ushape,y-pos, E C block all non- causal paths from A U C U U to Y. Hence, we seek to achieve Y Il x-pos (shape, y-pos via the HSCIC operator. To accommodate the mixed input types, Y puts an MLP on top of fea- tures extracted from the images via convolutional layers concatenated with features extracted from the remaining inputs via an MI.. Figure 4 demonstrates that HSCIC achieves improved VCF ? increases up to a certain point while affecting MSE, an inevitable trade-off.
4.3 FAIRNESS WITH CONTINUOUS PROTECTED ATTRIBUTES
We apply our method to the popular UCI Adult dataset (ohavi & Becker 1996). Our goal is to predict whether individual's 's income is above certain threshold Y E {0, based on collection of (demographic) information including protected attributes such as gender and age. We follow follow[Nabi & Shpitser| (2018); Chiappa (201)), where a subset of variables are selected from the dataset and a causal structure is assumed as in Figure ((bb (see Appendix E.5 and Figure 7 for details). We choose gender (considered binary in this dataset) and age (considered continuous) as the protected attributes A. We denote the marital status, level of education, occupation, working hours per week, and work class jointly by X and combine the remaining observed attributes in C. Our goal is to learn predictor Y that is counterfactually invariant A with respect to W C X. We remark that achieving fairness for continuous or even mixed categorical and continuous protected attributes is an ongoing line of research (cven for non-causal fairness notions) (Mary et al. 2019: Chiappa & Pacchiano, 202I). but directly supported by HSCIC.
We MLP with binary cross-entropy loss for Y. Since this experiment is based on real data, the true counterfactual distribution cannot be known. Hence, we follow Chiappa & Pacchiano (202T) and estimate possible true SCM by inferring the posterior distribution over the unobserved vari-
8

---

0.040
A1900
VCF distrbbtttooo
0.008
MSD
HSCTC
0.0
0.006
01485
0.55
01800
0.5
Bi
2.11
U5 1.5
regularization parameter Y
regularization parameter
regulatization parameter
Figure 4: Results MSE, HSCIC operator and VCF for the dSprites image dataset exper- iment. The HSCIC operator decreases steadily with higher values of y. Similarly, a necessary increase MSE can be observed. For both y = 0.5 and = an overall decrease of VCF is observed compared the not-regularized setting.
12 1 C
denatty
Accuracy
HSCIC
VCF
(81.725 + 0. 0.0481)8
0.01067 + 3.4 x 10-
0.06685 = 0.006
(81.347 0.0715)%
0.00739 + 10-
0.05156 0.003
EX
VCF distribution
Figure 5: (Left) Distribution of VCF values (unnormalized) for different choices of 7/ We observe less variance and mass zero for regularization parameter 7 (Right) Results accuracy, HSCIC and VCF, showing a considerable improvement of HSCIC and VCF for = 1.
ables using variational autoencoders (Kingma & Welling 2014). Even though this only provides approximate VCF, Figure (left) shows that HSCIC achieves more counterfactually fair outcome distributions (more mass near zero) than an unconstrained classifier (7 = 0). F (right) high- lights once more that the HSCIC operator is in agreement with the VCF, again trading off accuracy.
DISCUSSION AND FUTURE WORK
We studied the problem of learning predictors Y that are counterfactually invariant in changes of certain covariates. We put forth a formal definition of counterfactual invariance and described how generalizes existing notions. Next, we provided a novel sufficient graphical criterion to charac- terize counterfactual invariance and reduce it to conditional independence statement in the obser- vational distribution. Our method does not require identifiability of the counterfactual distribution exclude the possibility of unobserved confounders. Finally, propose an efficiently estimable, model-agnostic regularization term to enforce this conditional independence (and thus counterfac- tual invariance) based on kernel mean embeddings of conditional measures, which works for mixed continuous/categorical multi-dimensional variables. We demonstrate the efficacy of our method in regression and classification tasks involving controlled detailed simulation studies, high-dimensional images, and in fairness application, where it outperforms existing basclines.
The main limitation of our work, shared by all studies in this domain, the assumption that the causal graph is known. Another limitation that our methodology is applicable only when our graphical criterion is satisfied, requiring certain confounders are not generally excluded). From an ethics perspective, the increased robustness of counterfactually invariant (or societal benefits of counterfactually fair) predictors are certainly desir- able. However, this presupposes that all the often untestable assumptions are valid. Overall, causal methodology should not be applied lightly, especially in high-stakes and consequential decisions. A critical analysis of the broader context systemic factors may hold more promise for societal benefits, than a well-crafted algorithmic predictor.
variables to be observed (albeit unobserved
In light of these limitations, interesting and important direction for future work is to assess the sensitivity of our method to misspecifications of the causal graph or insufficient knowledge of the required blocking set. Finally, we believe graphical criterion and KME-based regularization can also be useful for causal representation learning, where one aims to isolate causally relevant, autonomous factors underlying the data generating process of high-dimensional data.

---

REFERENCES
Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprint arXiv:1907.022. 02893, 2019.
Haim Avron, Michael Kapralov, Cameron Musco, Christopher Musco, Ameya Velingker, and Amir Zandich. Random fourier features for kemel ridge regression: Approximation bounds and sta- tistical guarantees. In International conference on machine learning, volume 70, pp. 253-262, 2017.
Solon Barocas and Andrew D. Selbst. Big data's disparate impact. California Law Review, 104, 2016.
Peter Battaglia, Jessica B Hamrick, Victor Bapst, Alvaro Sanchez-Gonzalez, Vinicius Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, et al. Relational inductive biases, deep learning, and graph networks. arXiv preprint arXiv:1806.01200 2018.
Alain Berlinet and Christine Thomas-Agnan Reproducing kernel Hilbert spaces in probability and statistics. Springer Science & Business Media, 2011.
Benjamin Bloem-Reddy and Yee Whye Teh. Probabilistic symmetries and invariant neural networks. The Journal of Machine Learning Research, 21: 21:90-1, 2020.
Peter Bühlmann. Invariance, causality and robustness. Statistical Science, 35(3):404-422 2020.
Andrea Caponnetto and Ernesto De Vito. Optimal rates for the regularized least-squares algorithm. Foundations of Computational Mathematics, 7(3):331-368, 2007.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. International conference on machine learning, pp. 1597-1607. PMLR, 2020.
Silvia Chiappa. Path-specific counterfactual fairness. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 7801-7808, 2019.
Silvia Chiappa and Aldo Pacchiano. Fairness with continuous optimal transport. arXiv preprint
arXiv:2101.02084 2021.
Erhan Çinlar and E. 8Cmlar. Probability and stochastics, volume 261. Springer, 2011.
Taco Cohen and Max Welling. Group equivariant convolutional networks. In International confer- ence on machine learning, pp. 2990-2999. PMLR, 2016.
Nicolae Dinculeanu. Vector integration and stochastic integration in Banach spaces, volume 48. John Wiley & Sons, 2000.
Kenji Fukumizu, Arthur Gretton, Xiaohai Sun, and Bernhard Schölkopf. Kernel measures of condi- tional dependence. In Advances in neural information processing systems, volume 20, 2007.
Kenji Fukumizu, Le Song, and Arthur Gretton. Kernel bayes' rule: Bayesian inference with positive definite kernels. Journal of Machine Learning Research, 14(1):375333733, 2013.
Arthur Gretton, Olivier Bousquet, Alexander Smola, and Bernhard Schölkopf. Measuring statis- tical dependence with hilhert-schmidt norms. In Algorithmic Learning Theory, volume 3734, pp- 63-77, 2005.
Steffen Grüneeääler, Guy Lever, Arthur Gretton, Luca Baldassarre, Sam Patterson, and Massimil- iano Pontil. Conditional mean embeddings as regressors. In International conference machine learning. 2012.
Moritz Hardt, Eric Price, Eric Price. and Nati Srebro. Equality of opportunity in supervised learning. In Advances in Neural Information Processing Systems, vol- ume 29, 2016. URL http://cweedaaaaooooooooorrrrrrr pttpp//pppppe//////ee/////
10

---

Christina Heinze-Demi Jonas Peters, and Nicolai Meinshausen. Invariant causal prediction for nonlinear models. Journal of Causal Inference, 6(2), 2018.
Niki Kilbertus, Mateo Rojas Carulla, Giambattista Parascandolo, Moritz Hardt, Dominik Janzing, and Bernhard Schölkopf. Avoiding discrimination through causal reasoning. Advances in neural information processing systems, volume 30, 2017.
Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. In 2nd International Conference on Learning Representations, ICLR 2014, 2014. 1312. 6114
orq/ abs/
Ronny Kohavi and Barry Becker. Uci adult data set. UCI Meachine Learning Repository, 1996.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Advances in Neural Information Processing Systems, volume 25, 2012. URL https::/pproceeddngg neuriss.erppp..... cctpppappee//21////1eeel/ c399862d3b9ddee
Paper. pdf
Matt J. Kusner, Joshua R. Loftus, Chris Russell, and Ricardo Silva. Counterfactual faimesss. In Advances in Neural Information Processing Systems, pp. 4066 607664076, 2017.
Chaochao Lu, Yuhuai Wu, José Miguel Hemúández-Lobato, and Bernhard Schölkopf. Invariant causal representation learning for out-of-distribution generalization. International Conference on Learning Representations, 2021.
Jeremie Mary, Clément Calauzènes, and Noureddine El Karoui. Fairness-awand learning for con- tinuous attributes and treatments. In International Conference on Machine Learning, volume 97, pp.
PMLR, 2019.
Loic Matthey, Irina Higgins, Demis Hassabis, and Alexander Lerchner. dsprites: Disentanglement testing sprites dataset. http:///tttb..commmmeppp/////ptt
2017.
Shira Mitchell, Eric Potash, Solon Barocas, Alexander D' A'mour, and Kristian Lum. Algorithmic fairness: Choices, assumptions, and definitions. Annual Review of Statistics and Its Application, 8:141-163, 2021.
S. Chandra Mouli and Bruno Ribeiro. Asymmetry learning for coumterfactually--nvrrinn classifica- tion in OOD tasks. In Proc. of ICLR, 2022.
Krikamol Muandet, Kenji Fukumizu, Bharath Sriperumbudur, and Bernhard Schölkopf. Kernel mean embedding distributions: review and beyond. Foundations and Trends in Machine Learning, 10(1-2):1-141, 2017.
Krikamol Muandet, Motonobu Kanagawa, Sorawit Saengkyongam, and Sanparith Marukatat. Coun- terfactual mean embeddings. Journal of Machine Learning Research, 22(162);1-71, 2021.
Razich Nabi and Ilya Shpitser. Fair inference on outcomes. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
Junhyung Park and Krikamol Muandet. A measure-theoretic approach to kernel conditional mean embeddings. In Advances in Neural Information Processing Systems, pp. 21247-21259, 2020.
Judea Pearl. Causality: Models, Reasoning and Inference. Cambridge University Press, 2000.
Jonas Peters, Peter Bühlmann, and Nicolai Meinshausen. Causal inference by using invariant pre- diction: identification and confidence intervals. Journal of the Royal Statistical Society: Series B (Statistical Methodology). 78(5):947-1012, 2016.
Ali Rahimi and Benjamin Recht. Random features for large-scale kernel machines. In Advances in Neural Information Processing Systems, pp. 1177-1184, 2007.
Mateo Rojas-Carulle Carulla, Bernhard Schölkopf. Richard Turner, and Jonas Peters. Invariant models for causal transfer learning. The Journal of Machine Learning Research, 19(1):130--134) 2018.

---

Bernhard Schölkopf, Alexander J Smola, Francis Bach, al. Learning with kernels: support vector machines, regularization, optimization, and beyond. MIT press, 2002.
Connor Shorten and Taghi M Khoshgoftaar. A survey on image data augmentation for deep learning.
Journal of big data, 6(1):1-48, 2019.
Ilya Shpitser and Judea Pearl. Identification of joint interventional distributions in recursive semi- markovian causal models. In Proceedings of the AAAI Conference on Artificial Intelligence, pp- 1219-1226, 2006.
Ilya Shpitser and Judea Pearl. Complete identification methods for the causal hierarchy. Journal of Machine Learning Research, 9:1941-1979, 2008.
Iya Shpitser and Judea Pearl. Effects of treatment on the treated: Identification and generalization. In Proceedings of the Twenty-Fifih Conference on Uncertainty in Artificiia Intelligence, pp. 514- 521. 2009.
Ilya Shpitser, Tyler VanderWeele, and James M Robins. the validity of covariate adjustment for estimating causal effects. arXiv preprint arXiv: 2203.3515, 2012.
Alex Smola, Arthur Gretton, Le Song, and Bernhard Schölkopf. A hilbert space embedding for distributions. In International Conference on Algorithmic Learning Theory, pp. 13-31. Springer, 2007.
Le Song, Jonathan Huang, Alexander J. Smola, and Kenji Fukumizu. Hilbert space embeddings of conditional distributions with applications to dynamical systems. In International Conference on Machine Learning. volume 382, pp. 961-968, 2009.
Le Song, Kenji Fukumizu, and Arthur Gretton. Kernel embeddings of conditional distributions: A unified kernel framework for nonparametric inference in graphical models. IEEE Signal Process- ing Magazine, 30(4):98-111, 2013.
Zoltán Szabó and Bharath K. Sriperumbudur. Characteristic and universal tensor product kernels. Journal of Machine Learning Research, 1-233:11-23:299 2017.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Infor- mation Processing Systems, volume 30, 2017.
Victor Veitch, Alexander D' Amour, Steve Yadlowsky, and Jacob Eisenstein. Counterfactual invari- ance spurious correlations in text classification. In Advances in Neural Information Processing Systems, pp. 16196-16208, 2021.
Qizhe Xie, Zihang Dai, Eduard Hovy, Thang Luong, and Quoc Le. Unsupervised data augmentation for consistency training. In Advances in Neural Information Processing Systems, volume 33, pp. 6256-6268, 2020. URL httpps:/pprcceedggsss. neurips. cc/pappee//22///i1eeee
pdf
Renzhe Xu, Peng Cui, Kun Kuang, Bo Li, Linjun Zhou, Zheyan Shen, and Wei Cui. Algorith- mic decision making with conditional fairness. In Proceedings of the Conference Knowledge Discovery and Data Mining, pp. 2125-2135, 2020.
Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnbbas Poczos, Russ R Salakhutdinov, and Alexander J Smola. Deep sets. In Advances in Neural Information Processing Systems, volume 30, 2017. URL Dtt nee innuripss. 222 httpg//...eerpppppppppppeee/e//a pttpc//pppp//221//1111.
12

---

A
U
Y
(a)
(b)
(c)
Figure 6: (a) A causal graph 9, which embeds information for the random variables of the model
in the pre-interventional world. (b) The corresponding graph G for the set W = { X). The variables A and X are copies of A and X respectively. construction, any intervention of the form A + does not affect the group W = =
(c) The post-interventional graph g, By
A MISSING PROOF OF THEOREM 3.2
In this section, we give proof of Theorem 3.2 which restate for completeness.
A.1 OVERVIEW OF THE PROOF TECHNIQUES
Theorem 3.2. Les be causal diagram, and let A, W be two (not necessarily disjoint) of nodes in G. Let be set of nodes that blocks all non-causa paths from A U W to Y. Then, for any SCM compatible with G, any predictor Y that satisfies Y 1L A invariant in A with respect W.
Z is counterfactually
Our proof technique generalizes the work of SShitser & Pearl (200)) To understand the proof technique, note that conditional counterfactual distributions of the form Py; w) involve quantities from Iwo different worlds. The variables W belong the pre-interventional world, and the interventional variable Y belongs to the world after performing the intervention intervention A a. Hence, we study the identification conditional counterfactual distributions using diagram that embeds the causal relationships between the pre- and the post-interventional world. After defining this diagram, we prove that some conditional measures in this new model provide an estimate for Py: w). We then combine this result with the properties of Z to prove the desired result.
A.2 A GRAPHICAL CRITERION FOR CONDITIONAL INDEPENDENCE
In this section, we discuss well-known criterion for conditional independence, which will then use to prove Theorem 3.2 To this end, we use the notion of blocked path, which we restate for clarity.
Definition 3.1. Consider a path x of causal graph A set of nodes blocks n. if contains a triple of consecutive nodes connected in one of the following ways: N; Z N₁, N; < Z - Nj, or N; W M + N3, with N,, N₂ & Z, E Z, and neither M nor any descendent of M is in
Using Definition A.I we define the concept of d-separation follows.
Definition A.1 (d-Separation). Consider a causal graph G. Two sets of nodes X and Y of G said to be d-separated by third set if every from any node of X to any node of is blocked by Z.
Definition A.1
graphical criterion for conditional independence. In fact, the following well- known results holds (Pearl 2000).
A non-causal paths from A U W to Y is path connecting A U W and Y in which at least one edge points against causal ordering. Shpitser al. 2012
13

---

Lemma A.2 (d- de-arrrrtionn Criterion). Consider a causal graph G. and suppose that two of nodes X and Y of 9 are d-separated by Z. Then, X independent of given Z in any model induced by the graph G.
We use the notation X g Y Z to indicate that X and are d-separated by U in G.
A.3 IDENTIFIABILITY OF CONDITIONAL COUNTERFACTURE DISTRIBUTIONS
natural way study the relationships between the pre- and the post-interventional world is to use the counterfactual graph (Shpitser & Pearl 2008). However, the construction of the counterfactual graph rather intricate. For our purposes it sufficient to consider simpler construction. Consider an SGM with causal graph G, and fix set of observable random variables W. We define the corresponding graph G' in the following three steps:
1. define G' to be the same graph as 9;
2. add a new node W to G', for each node W of the set W;
3. for each node W of the set W and for each parent W of W, if e W then add an edge
W + W' to G": if W & W add an edge W + W go
An illustration of this graph presented in Figure 6 Note that any node W defined as above does not have any descendants in g'. In the reminder of this section, denote with W the set of all nodes W in g' defined as above. We remark that this construction generalizes the work by Shpitser & Pearl (2099).
Fix a second set A of nodes of G, and consider interventions of the form A <- a, as in the statement of Theorem 3.2 We prove that any conditional counterfactual distribution of the form PY; w (y
w) are identihable in G if and only if the corresponding probability distributions PY w (y w) identifiable in 9aa The following theorem generalizes Theorem by Shpitser Pearl (2099)
Theorem A.3. Let G be causal diagram, and consider sets of nodes A. W (not necessarily disjoint). Consider the corresponding graph G' as defined above. If the distribution Pyzww( w) identifiable in G, then the distribution IP w/W (y w) identifiabee in Gy for any model induced by G. Furthermore, the estimand PY P/|W( (y w) in 9' is correct for Fy:iw(y w).
To prove Theorem A.3 we introduce additional concepts. We first introduce the notion of a C'-orest and the notion of an edge (Shpitser Pearl 2006).
Definition A.4 (Cresrtst). Let G be causal diagram, and consider complete sub- sub-graph H. of G. Denote with R. the maximal root set of H. We say that H is a R--ooted C-forest if a subset of its bi-directed ares forms spanning tree all vertices in H. and all the observable nodes of H have at most one child.
In our analysis we also use the following definition.
Definition A.5 (Edge). Let G be causal diagram, and fix set of nodes A. Consider two R-rooted C-forests H. K of such that (i) H is sub-graph of K;; (ii) H and do not contain any variable in A; (ii) the nodes of R. are ancestors of Y in the graph Ga. Then, we say that H and K. form an edge for Px:(y) in G.
There is connection between these concepts and the identifiability of counterfactual distributions, as shown in the following thcorem.
Theorem A.6 (Theorem 4 by (Shpitser& & Pearl 20061). Let be causal diagram, and fix set of nodes A. Suppose that there exist two sub-graphs of 9 that form an edge for (y: Then, Pv; (y) identifiable in G.
Using these concepts, we can prove Theorem A.3
Proof of Theurem A.3 For simplicity, denote with P the induced measure on G/- We first prove that PY::W(( w) is identifiable in if and only if Jp+ ( (y w) is identifiable in G We
distinguish two cases, based whether W is d-separated from Y in G' not.
14

---

Case 1: W is d-separated from Y in l.. By Lemma A.2 Hence, F r/w (y w) is identifiable and only if
we have that Y is independent of W.
P Y (y = Py = Py; (y)
(5)
identifiable in Co. Suppose that Py (y:w((y w) is not identifiable. Then, (y) is not identifi- able, and P Y W (y w) is also not identifiable by eq. (5).
Case 2: of A U W is not descendent of Y in G (otherwise it has no effect on Y). Under this assumption, there exists set of random variables U C W such that there exists an edge for P ((YuT); (y, m) = F+ (,, u) in G' It follows that Y (y, u) is not identifiable. Since U C w, follows that the distribution IP+ Y, v.wlo (y, w) is also not identifiable.
not d-separated from in Saa Assume without loss of generality that any variable
We conclude by showing that the estimand expression is correct. To this end, note that since G' is just causal diagram, the estimand of the post-interventional distribution P r/W (y w) correct
for PVaxIA,,XX aa, x). Since the set W only contains variable copies of X, as claimed. C
A.4 VALID ADJUSTMENTS FOR CONDITIONAL INTERVENTIONAL DISTRIBUTIONS
Here we discuss a criterion for the identification of conditional distributions, which we will then to prove Theorem 3.2 We follow Pearl (2000) to this end, and use the d-separation criterion to define valid adjustment sets for conditional counterfactual distributions.
Consider model with causal graph G, and fix of observed variables A. We define auxiliary graph G1. by adding to G an additional node This node is a parent the nodes in the set A, and has no other neighbour. We modify the structural assignments of the nodes A, so that for I 0 the values of A are determined as in G. whereas for I = 1 the values of A are set to A a. The node corresponds to a Bernoulli distribution, with I indicating that the intervention took place. This construction is important, because the following lemma holds.
Lemma A.7. Consider an SGM and fix two disjoints groups of observed variables X and Y. Denote with G1 the auxiliary graph as defined above, with respect to an intervention A + a. Let Z a set of nodes of G such that Y m.g, I| A, Z. Then, it holds
Py 2) (Yj a, 2).
Here, P the post- inssevettevvtn distribution, after assigning A a.
Proof. It holds
PY Zy z) = =PYJA 2,
(by definition of P*)
=PYJLA,,ZY i= a,
(by the definition of I)
= =PYILA,Z(Y i=0,a,
(by Lemma A.2
= =FYIAZ(Y a,z),
(by the definition of I)
as claimed.
The following lemma characterizes all sets that fulfills the condition as in Lemma A.7
Lemma A.8. Consider an SGM G, and denote with G1 the auxiliary graph as defined above, with respect to intervention A &- Let Z set of nodes that blocks all non-causal paths from A to Y. Then, it holds Y 1Lg I A, Z. In particular, it holds py Pxzz(X (y = PYIA,Z(y a, z).
Proof. Denote with any path from I to Y in G1. Since I is a parent of every node in A, and since I has no parents, then any path π can decomposed into two paths मe. T2. where m is a single edge from to A, and 22 is path from A to Y. If is a causal path, then A acts separator for this path. If и is not a causal path from to Y, then it must be undirected (since has parents). It follows that Ty is undirected path from A to Y, which is d-separated by Z. Hence, π is deseaartee.. We conclude that Y 1L. A. Z. The second part of the claim follows by Lemma A.7
15

---

A.5 PROOF OF THEOREM 3.2
Theorem A.3 tells us that we can identify conditional counterfactual distributions in G. by identifying distributions on G'. We combine this observation with the notion of a valid adjustment set to derive a closed formula for the identification of the distributions of interest.
Proof Theorem
3.2
Let g' be the augmented graph obtained by adding nodes W to G, as de- A.3 Denote with F the induced measure on ga Suppose that it holds
scribed in Section
P (y: (
(w)
P a, z)dP W Z w)
(6)
for any intervention A + and for any possible value attained by W. Then the claim follows. In fact, assuming that eq. (6) holds, we have that
Py w) PYJA.Z(Y a, z)dP 2 2/ w)
(assuming eq. (6)
= f PYJA.((y a', 2)dd 2 w)
(since Y T. A Zi
-Y;;, w).
(assuming eq. (6)
Hence, the proof of Theorem 3.2 boils down proving eq. 6 To this end, since breaks all non-causal paths from W Y. then by construction Z is a d-separator between W and Y in the post-interventional graph G/a Hence, it holds
Px, W
I Pr w, z)d 2/ W
(by conditioning)
/ Pyiz(y z)dP w.z( (w,z)
(since Y 1o W (Z)
f PYJA,W.(Y a, w, w,2)) 2)
z/w w).
(by Lemma A.7
The claim follows by applying Theorem A.3 the equation above, since it holds T Fy:ww(y /w).
Y/W w)
B MISSING PROOF OF THEOREM 3.4
We prove that the HSCIC can be used to promote conditional independence, using similar tech- nique as Park & Muander (2020). The following theorem holds.
Theorem 3.4 (following Theorem 5.4 by Park Muandet (2020)) the kernel ls of Hx Ha characteristid 5 HSCIC(Y, A Z) = 0 almost surely if and only i 11 A Z.
Proof. By definition, we can write HSCIC(Y, A Z) HY,AI Z, where HY,AIZ is a real- valued deterministic function. Hence, the HSCIC is real-valued random variable, defined over the same domain U2 of the random variable X.
We first prove that if HSCIC(Y, A Z) 0 almost surely, then it holds Y TT A Z. To this end, consider an event ! C 2x that occurs almost surely, and such that it holds (Hy.A o X)) for all w E Ω². Fix a sample w e St' and consider the corresponding value z Z(w), in the support of Z. It holds
/ k(y @ a. )dTY.AZZZZ NY.AIZ=Z
(by definition)
MYYZER & PLAZZE
(since S2)
= -fbr(s, )XPP 2 /KAlA, kA(a -dP AIZ=Z
(by definition
= -f
k -)
) JdPPIzZ PAIZER:
(by Fubini's Theorem)
'The tensor product kernel l: characteristic if the mapping PY,A + By.a [k[ ,,y y
a)) injective.
16

---

with by and KA the kernels of Hy and HA respectively. Since the kernel of the tensor product space Hy @ HA characteristic, then the kernels ky and kA are also characteristic. Hence, holds PY,A/Z=R, = = PPIiz=,,PAiz-- Paiz=a, for all w E Ω.. Since the event 52' occurs almost surely, then F PrIZ=z P almost surely, that is Y TT A Z.
Assume now that Y TT A Z. By definition there exists an event 52" S Zzz such that PY,A PYIZ=z PAIZ= for all samples w E QQ", with Zn (ww). It holds
PY,A/ / k(y @ a, .d
(by definition)
k- a, -IPZZ PAIZ=E
(since (')
/ ky )AA YUPYIZE PAIZ=a
(by definition of k)
J Ky(y, JIPY & Jala AIZARZ
(by Fubini's Theorem)
PYIZ=zu @ PAIZ=E-
(by definition)
The claim follows.
C CONDITIONAL KERNEL MEAN EMBEDDINGS AND THE HSCIC
The notion of conditional kernel mean embeddings has already been studied in the literature. We show that, under stronger assumptions, our definition is equivalent to the definition by Park & Muan- det 2020).
C.I CONDITIONAL KERNEL MEAN EMBEDDINGS AND CONDITIONAL INDEPENDENCE
We show that, under stronger assumptions, the HSCIC can be defined using the Bochner conditional expected value. The Bochner conditional expected value is defined as follows.
Definition C.1. Fix two random variables Y, Z taking value in Banach space H. and denote with (S2 F, P) their joint probability space. Then, the Bochner conditional expectation of Y given Z is any H-valued random variable X such that
[ YdP YdP= XdIP
for all E oZZ) F, Bochner expected value. Any random variable X as above is version of [Y] Z].
F. with (Z) the o-algebra generated by Z. We denote with E [Y
Z the
The existence and almost sure uniqueness of the conditional expectation shown in Dinculeanu (200). Given RKHS H with kernel & over the support of Y. Para & Muandet (2020) define the corresponding conditional kernel mean embedding as
PYz E [k(:,)) Z
Note that, according to this definition, WY|z is H-valued random variable, not single point of H. Park & Muandet 2020) use this notion to define the HSCIC as follows.
Definition C.2 (The HSCIC according Park & Muandet (2200). Consider (sets of) random variables Y, A, Z, and consider two RKHS Hy, HA over the support of Y and A respectively. The HSCIC between Y and A given defined as the real-valued random variable
(| HYIz(w) [A][(W)]|
for all samples w in the domain Giz of Here, ||-I| the metric induced by the inner product of the tensor product space Hy & Hz.
We show that, under more restrictive assumptions, Definition C.2 be used to promote conditional independence. To this end, the notion of regular version.
17

---

Definition C.3 (Regular Version, following Definition 2.4 by Cinlar & òCinlar (011h). Consider two random variables Y, Z. and consider the induced measurable spaces (St FyY and (Ωzz Fz).
A regular version Q2 for Y is mapping Q: Oz x Fx + [0, +ool: (w, y) H Qwly) such that:
(i) the map + (x) is FA FA-measurable for all yi (ü) the map y Qy(y) is measure (fty Fx) for all w; (iii) the function Qy(y) is version for E [r Z
The following theorem shows that the random variable in Definition conditional independence.
C.2 can be used promote
Theorem C.4 (Theorem 5.4 by Park Muandet (2020) With the notation introduced above, sup- pose that the kernel k of the tensor product space Hx X H. is characteristic. Furthermore, sup- pose that Py Y A admits a regular version. Then, (YYA)IZ(W) PY|z (w) & Paz(w)] almosst surely if and only if 11 A Z.
Note that the assumption of the existence of a regular version is essential in Theorem C.4 In this work, HSCIC not used for conditional independence testing but as conditional independence measure.
C.2 EQUIVALENCE WITH OUR APPROACH
The following theorem, shows that under the existence a regular version, conditional kernel mean embeddings can be defined using the Bochner conditional expected value. To this end, use the following theorem.
Theorem C.5 (Following Proposition 2.5 by Cinlar & òCinlar (211 Following the notation in- troduced in Desinition
C.3 suppose that Priz Z) admits regular version Qu(y). Consider kernel k over the support of Y. Then, the mapping
f k(.r))Qu(y))
version of E (k(())) Z].
As a consequence of Theorem we prove the following result.
Lemma C.6. Fix two random variables Y. Z. Suppose that Pviz admits regular version. Denote with Oz the domain of Z. Then, there exists subset C Oz that occurs almost surely, such that NY z (w) (Z for all E Ω. Here, (x(Z=Z(()) the embedding of conditional measures as in Section 2
Proof. Let (y) be a regular version of Pyiz. Without loss of generality we may assume that holds (y {Z- = Z(w))) Qw By Theorem C.5 there exists an event Ω C Uz that occurs almost surely such that
MY mYz(w) (w) Elk(y, )) (W) k(y, ddQu(y)
for all w E Q2 Then, for w E 92 it holds
/ k(x, )dQ
PY (w)
(it follows from eq. 7
= -ʃ [JK( k(x, JdPxIA A(w)))
(v) = Priz(y [Z = Z(w))))
-MX(A=A(01+
(by definition in Section
as claimed.
As consequence of Lemma C.6 we can prove that the definition of the HSCIC by Park & Muandet 2020 is equivalent to ours. The following corollary holds.
Corollary C.7. Consider (sets of) random variables Y, A, Z, and consider two RKHS Hy, Ha over the support of Y and A respectively. Suppose that Py, Ayy. Z) admits a regular version. Then, there exists set Q C GA that occurs almost surely, such that
|X (x) (xiz(24) (w) [A]Z(W)]|
A a (w)
18

---

Here, Hy,A is real-valued deterministic futermiinisiicffactiion defined as
Hy,Aiz(z) = 1 -YZ==2 PA|Z=2||
and ||-||| is the metric induced by the inner product of the tensor product space Hx & HA.
We remark that the assumption of the existence regular version is essential in Corollary C.7
D CONDITIONAL INDEPENDENCE AND THE CROSS-COVARIANCE OPERATOR
In this section, we show that under additional assumptions, our definition of conditional KMEs is equivalent to the definition based on the cross-covariance operator, under more restrictive assump- tions.
The definition of KMEs based on the cross-covariance operator requires the use of the following well-known result.
Lemma D.I. Fix two RKHS Hx and Hz, and let {pp)) and { be orthonormal bases of Hx and Hz respectively. Denote with (Sx Hz) the set of Hilbert-Schmiae operators between
Hx and Hz. There is isometric isomorphism between the tensor product space Hx @ Hz and HS(H Hz), given by the map
T: =1
au9i
cisl 9i)
For a proof of this result see i.e.. Park Muandel 2002 This lemma allows us to define the cross-covariance operator between two random variables, using the operator T'
Definition D.2 (Cross-Covaiie Covariance Oprator). Consider
random variables X, Z. Consider responding mean embeddings μx.Z. Px and pz. as defined in Section 3 The cross-covariance
operator is defined Σx.2 Z T (px,7 μxx & pz). Here, T is the isometric isomorphism in Lemma D.1
well-known that the cross-covariance operator can be decomposed into the covariance of the marginals and the correlation. That is, there exists unique bounded operator Ay z such that
Σx = 11/2 YY Av,z o Ez
Using this notation, we define the normalized conditional cross-covariance operator. Given three random variables Y, A, and corresponding kernel mean embeddings, this operator is defined as
AY.AIZ AY. Av z AZ,A-
This operator was introduce by Fukumizu (007). The normalized conditional cross-covariance can be used to promote statistical independence, as shown in the following theorem.
Theorem D.3 (Theorem 3 by Fukumizu et al. (2007). Following the notation introduced above, define the random variable A = (A, Z). Let Pz be the distribution of the random variable Z, and denote with L (P())) the space of the square integrable functions with probability Pz. Suppose that the tensor product kernel ky 6 ka 60 kz is characteristic. Furthermore, suppose that Hz. + R is dense in T.2 (.P). Then, holds
AYĀz =
if and only if Y A
Here, Ay Alz is an operator defined as in eq.
By Theorem D.3 the operator Ay Alz also be used to promote conditional independence. How- method is more straightforward since requires less assumptions. In fact, Theorem D.3 requires to embed the variable Z in a RKHS. In contrast, our method only requires the embedding on the variables Y and A.
19

---

E EXPERIMENT SETTINGS
Additional information about the experiments is now provided. The interested reader may refer to the source code provided in the supplementary material. In all cases, the experiments were per- formed on an Apple MI Pro. No external GPU sources were used for the experimental setup.
E. DATASETS FOR MODEL PERFORMANCE WITH THE USE OF THE HSCIC
The first set of synthetic experiments involves three different dataset simulations. The data- generating mechanism corresponding to the results in Figure is the following:
~~ (0,1)
A = zz +EA
-- - A + sin (2Z) + 1
=xp
X = exp
"5xx
1
Y
exp { XZ) sin (2XZ) 5A + J
2
f.i.d. where EA ~ N (0, 1) and Ex, Ex ~ N (0,
In the first experiment, Figure 3 shows the results of feed-forward neural networks consisting of hidden layers with 20 nodes each, connected with a rectified linear activation function (ReLU) and linear final layer. Mini-batch size of 256 and the Adam optimizer with a learning rate of 10-3 for 100 epochs were used. We set the range of trade-off parameter E [0, 1] for all the experiments except the comparison against baselines CFI and CF2.
E.2 DATASETS FOR COMPARISON WITH BASELINES
The simulation procedure for the Scenario and Scenario 2 in Table respectively are the following. Scenario 1:
Z ~ (0,1)
A = z +
1
X A 6x +2Z
2
Y exp (XXZ sin (2XZ) + 5A +
5 -EX,
2
i.dd. N (0, 1) and sy itd N (0,0.1). Scenario 2:
where Ax
N~ (0,1)
A Z EA
1 1 1 A ex +exxp X
5
2
Y = exp (-2²) + AX + ger.
where CA, EX f.ddd. N (0, 1) and EY ey
lid. N (0,00.).
Analysing the results in Table [ the same hyperparameters as in the previous setting. This also holds for the baseline methods CFI and CF2. In this table, both Scenario and Scenario 2 were considered. The results shown Figure 3 and Table | are the average and standard deviation resulting from respectively 10 and 4 random seeds runs.
20

---

E.3 DATASETS FOR BULTI-DIMENSIONS VARIABLES EXPERIMENTS
The data-generating mechanisms for the multi-dimensional settings of Tables now shown. Analysing the results Table 2 (top), given dimA = Di 2 2, the datasets were generated from:
~~ ~~
A; Z + S for E (1,
D2 A +
X exp
=xxp
A sin + 0x
D1 A
Y Exp
-2
XZ + ayy
where Ex,5Y fid N (0, 0.1) and & A chosen 64 and the same hyperparameters are used as in the previous setting. The neural network architecture is trained for 70 epochs.
Dj
~ i.d N (0, 1). In this setting, the mini-batch size
For the results in Table Sablee (bottom) the following data-generating process used:
D2 A
Z₁ Z2
LDz
iid N (0,1)
Z + EA
i=1
Dr
X = exp
=expp
-
+ sin
Σ Z
A 0.1.
2
D₂
Y = exp
-
A
A
+
Z, + A + +.1
1=1
with dimZ D2 2 and EA V (), Ex, 3.dd N (0, 0.1). Here, we used mini-batci size of 32, learning rate of 10 and a number of epochs of 500.
are The results Tables are the average obtained from three random seeds runs on the same data-split.
We tested the method on a further setting, consisting of bi-dimensional Z and A (dimA = 2, dimZ = 2). Specifically, we have Z {Z,, Z2) and A {A1, A2). The data- getet-eeneraiinggg mechanism the following:
Ze (3.000) A = 2 +EA A2 = f-p (Z) + Z2)} + +EAz Z, ~ (0,
X exp
A sin (2- A,) (Z + Zz) (A + A2) 0.1
{
1 -
Y =xp
exp
sin (2 -2)) + (Z) Z2) + Ai A.2
where EY, EX, Lid 0.11. In Table the trade-off between accuracy and counter- factually invariant predictions is once again shown, implying that the proposed method can also be applicd in settings where both Z and are not unidimensional. In Table average and standard deviation of the results from four runs with random seed and re-sampled data presented.
21

---

Table 3: Results of MSE, HSCIC, VCF (all times 10 for readability) on synthetic data with
bi-dimensional and Here dimZ dimA 2, dimX d,mi =.
MSE
HSCIC
VCF
0.315 + 0.055
13854.1 + 8.47
24.6 = 1.17
0.336 = 0.0525
13854.0 + 8.46
22.9 + 0.69
0.393 + 0.162
13853.9 8.46
20.4 2.02
E.4 HIGII- DIMIENIIMNSION IMAGE DATASET
The simulation procedure for the results shown in Section 4.2 is the following.
shape ~ P(shape)
y-pos P(y-pos)
color ~ P(color)
orientation P(orientation)
x-pos = (x), where x ~ (s(hape + y-pos, 1)
scale = round (x-pos
y-pos shape + (s)
24
24
Y = shape
x-pos + scale² sin(y-pos) + cy:
where (O, 1) and ey ~ (0, 0.01). The data has been generated via a matching procedure on the original dSprites dataset.
In Table 4 the hyperparameters of the layers of the convolutional neural network are presented. Each of the convolutional groups also has ReLU activation function and a dropout layer. Two MLP architectures have been used. The former takes as input the observed tabular features. It is composed by two hidden layers of 16 and 8 nodes respectively, connected with ReLU activation functions and dropout layers. The latter takes as input the concatenated outcomes of the CNN and the other MLP. It consists of three hidden layers of 8, and 16 nodes, respectively. Eegureepreeentt the averaged results of four random seeds runs with new sampled data.
E.5 FAIRNESS WITH CONTINUOUS PROTECTED ATTRIBUTES
The pre-processing of the UCI Adult dataset was based upon the work of (Chiappa & Pacchiano
2021. Referring the causal graph in Figure 7 variational autoencoder (Kingma & Welling
2014 was trained for each of the unobserved vanables Hm H₃ and H,. The prior distribution of these latent variables is assumed
be standard Gaussian. The posterior distributions P(Hm) V)) (V), (V) are modelled as 10-dimensional Gaussian distributions, whose means and vari- ances are the outputs of the encoder.
M
R
Y
Figure 7: Assumed causal graph for the Adult dataset, in Chiappa Pacchiano (021). The
variables Hm. Hi, H, are unobserved, and jointly trained with the predictor
22

---

Table Architecture of the convolutional neural network used for the image dataset, as described in Appendix E.4
layer
filters
kernel size
stride size
padding size
convolution max pooling
16
2
1
0
convolution max pooling
64
2
1
1
2
0
convolution
64
2
max pooling
1
2
0
convolution
16
5
3
max pooling
1
0
The encoder architecture consists of a hidden layer of 20 hidden nodes with hyperbolic tangent ac- tivation functions, followed by a linear layer. The decoders have two linear layers with hyperbolic tangent activation function. The training loss of the variational autoencoder consists of recon- struction term
Error for continuous variables and Cross-Entrop ones) and the Kullback-Leiibe divergence between the posterior and the prior distribution of the latent variables. For training, we used the Adam optimizer with learning rate of 10 10-2, 30 epochs, mini-batch size 128.
Loss for binary
The predictor Ý is the output of a feed-forward neural network consisting hidden layer with hyperbolic tangent activation function and linear final layer. In the training used the Adam optimizer with learning rate 10 10-3 3 mini- batch size 128, and trained for 100 epochs. The choice of the network architecture is based on the work of (Chiappa & Pacchiano 2021b.
The estimation of counterfactual outcomes is based on a Monte-Carle approach. Given a data point, 500 values of the unobserved variables are sampled from the estimated posterior distribution. Given an interventional value for A, a counterfactual outcome is estimated for each of the sampled unob- served values. The final counterfactual outcome estimated as the average of these counterfactual predictions. In this experiment setting, we have k = 100 and d = 1000.
In the causal graph presented in Figure 7 A includes the variables age and gender, C includes nationality and race, M marital status, L level of education, R the set of working class, occupation, and hours per week and Y the income class. Compared to (Chiappa & Pacchiano 2021). we include the race variable in the dataset part of the baseline features C. The loss function is the same as Equation but Binary Cross-Entropy loss is used instead of Mean-Squared Error loss:
CTOTAL (Y) -BCC Y) 7 HSCIC (ir, (Age, Gender) (z)
where the set blocks all the non-causal paths from W A. In this example we have W {C U M U L.R.. The results in Figure 5 (center, right) refer one with conditioning set = (Race, Nationality). The results in Table (right) are the average and standard deviation of four random seeds.
E.6 BASELINE EXPERIMENTS
We provide an experimental comparison against the method by Veitch et al. (222)) To this end, we consider the following artificial causal structure (see Figure [b)):
Z ~ N 10
A = Z + EA
1
X exp - sin (A) + sx
10"
1
Y exp (-XZ] sin (2XZ) + 5A + SY.
2
10
23

---

Table 5: Results of the MSE. HSCIC, VCF of our method and the baseline Veitch al. (2021) applied to the causal and anti-causal structure in Figure bb-ce. Although the graphical assumptions are not satisfied, our method shows an overall decrease of HSCIC, VCF in both of the graphical structures, outperforming Veitch et al. (2021) in terms of accuracy and counterfactual invariance,
x10
HSCIC x10
VCFx10"
1.20 0.03
22.88 1 1.42
89 1: 0.50
0.17
21.14
1.50
4: 0.30
10.45 1 0.35
18.62 1 1.43
87 1: 0.93
Our Method
Voitch al. 2021
MSR x 10
HSCIC x
VCF 10
MSEx 10
HSCIC x10
10
09.78 + 1.10
20.20 + 0.53
50.63 0.25
70.23 + 1.14
22.16 + 0.57 19.59
50.69 + 0.26
69.99
1: 0.53
50.78
70.60 1.24
50.85 1: 0.28
where EA Lid N (0. 1) and EY ~ N (0, 0.1). The data-generating mechanism of the anti- causal structure is the following (see Figure Tc)):
Z N (0.1)
(
1
3
3 10 EA
Y = exp
(5
eY
A Z
10
X exp
A- A2
+ Y
10 EX
2
where EY Lid N (0,0.1) and Ex 1,dd NV (0, 1). We compare our method with different choices for the trade-off parameter 7, against the method by Veitch et al. (202T). In the causal settings pre- sented Figure [egure[[bb) [b-c), an unobserved confounder Z between and Y is included. the graphical structure Figure (b. our method presents as regularization term in the model training HSIC(Y, A), as the independence Ý ILL A is enforced. Here, HSIC is the Hilbert-Schmidt Independence Crite- rion, which is commonly used to promote independence (see, i.e., Gretton et al. (2005); Fukumizu et al. (2007). In the selected graphical sstructure, this is the same independence criterion enforced by Veitch et al. (202T), leading the two methods converge. In the anti-causal graphical setting presented in F proposed by Veitch et al. (2021), the regularization term used in our method is sill HSIC(Ý, A), while in the method of Veitch et al. 2022) is HSCIC(Ý,, A Y). In Table 5 the results of accuracy, HSCIC(Ý, A Z) and VCF are presented.
In the experiments, the predictor Ý is feed-forward neural network consisting of hidden layers with 20 nodes each, connected with a rectified linear activation function (ReLU) and linear final layer. Mini-batch size of 256 and the Adam optimizer with a learning rate of 10 for 500 epochs were used.
E.7 COMPARISON HEURISTIC METHODS EXPERIMENTS
We provide an experimental comparison of the proposed method with some heuristic methods, specifically data-augmentation based methods. We consider the same data-generating procedure and causal structure as presented in E The heuristic methods considered are data augmentation and causal-hased data augmentation. In the former, data augmentation is performed by generat-
5.i.d ing N = 50 samples for every data-point by sampling new values of A 01, aN PA and leaving Z, X, Y unchanged. Differently, the latter causal-based data augmentation method, we also take into account the causal structure given by the known DAG. Indeed, when manipulating the variable A, its descendants (in this example X) will also change. In this experiment, a predictor for
X as X fo (A, Z) is trained on 80% of the original dataset. In the data augmentation mechanism, for every data-point {a, e, 2, y). N 50 samples are generated by sampling new values of A as
24

---

Table 6: Results of MSE and VCF (all times 102 for readability) on synthetic data of our method with trade-off parameters 7 1 and 7 with the heuristic methods data augmentation and causal-based data augmentation.
VCF
MSE
data augmentation
3.12 + 0.16
0.003 + 0.001
causal-based data augmentation
3.04
+ 0.16
2.80 + 0.13
0.044 0.022
2.34 0.19
0.19 = 0.072
a1 ON 1.iid PA. estimating the values of X as ø1 = fe((1, "N, = fa z), while leaving the values of and Y unchanged. Heuristic methods such as data-augmentation methods do not theoretically guarantee to provide counterfactually invariant predictors. The results of an empirical comparison are shown in Table 6 It can be shown that these theoretical insights are supported by experimental results, as the VCF metric measure counterfactual invariance is relevantly lower in both the two settings of the proposed methods (? = and 1).
In these experiments, dataset of architecture used for predicting and Y are feed-forward neural networks consisting of 8 hidden layers with 20 nodes each, connected with rectified linear activation function (ReLU) and linear final layer. Mini-batch size of 256 and the Adam optimizer with a learning rate 10 for 100 epochs were used.
1000 is used, along with 500 and d 500. The
25


Reviews:
[{"review_id": "qUNmPiV23eN", "reviewer": "Reviewer_Udyy", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a method to obtain predictors that are counterfactually-invariant to certain observed variables. Authors first show that counterfactual invariance of a predictor with respect to the variables A can be ensured given a set of variables Z that block all paths from A to Y. That is, a counterfactual invariant predictor satisfies the conditional independence $\\hat{Y} \\perp A | Z$. Then, the authors propose a conditional independence criterion and show that the proposed approach can learn counterfactual invariance in different tasks. ", "summary_of_the_review": "Comparison with existing notions of counterfactual invariance is incorrect and the claims of generality with respect to these existing notions is not well-supported. Problems in empirical evaluation: experiments do not seem to follow the assumptions in the main theorem, missing relevant baselines and missing procedure to choose $\\gamma$, an important regularization hyperparameter. \n", "strength_and_weaknesses": "Paper is well written and the proposed method is a sound way of obtaining counterfactual-invariant predictors given a set of observed adjustment variables under a known causal graph. It is also interesting that one does not need identifiability of the exact counterfactual distributions from observational data in order to learn general counterfactual-invariant predictors (as in Theorem 3.2). \n\nI have the following major concerns:\n\n1. The comparison with existing notions of counterfactual invariance is incorrect, possibly due to the use of different notations. In [1], Veitch et al. define Y(a) using a counterfactual change A=a “leaving all else fixed” including the exogenous variables, thus giving the true counterfactual for that observation. The current paper claims that the counterfactual-invariance definition in [1] only implies invariance to interventions Y|do(a) , which is incorrect. While the practical method proposed in [1] cannot differentiate between interventions and counterfactuals (as discussed in Section 3 of [1]), the definition itself is general. \n\n   Further, a definition of counterfactual-invariance with a clearer notation is provided in [2] that explicitly shows conditioning of the observed variable. This definition is more general than the one  proposed in the current paper as it requires almost-sure equality of representations (and not just in-distribution equality). \n\n2. Experimental evaluation: \n\n   1. All the experiments consider the case when A (protected attribute) directly causes the target Y. This seems to be in contradiction with Theorem 3.2 which requires that a set of nodes Z blocks all paths from A to Y. In experiments, the original counterfactual fairness [3] setting should also be considered where A does not directly cause Y. \n   2. Additional baselines: Since one of the contributions of the paper is to use HSCIC for conditional independence testing, other more classical approaches (e.g., [4,5]) for the same should be compare against. Further, since the current experiments with A directly causing Y are more suited to path-specific counterfactual fairness [6], this should be added as baseline whenever possible. \n   3. Authors should provide a method/heuristic to select the regularization parameter $\\gamma$ using only the training/validation data. \n\n3. I am not able to appropriately judge the novelty of the proposed method. \n\n   1. Authors should discuss more in detail how Theorem 3.2 and its proof generalizes the work in [7,8]. For example, the construction of counterfactual graph in the proof of Theorem 3.2 is claimed to be more general than the one in [8], but not elaborated further. \n   2. A different definition of HSCIC (functionally equivalent to [9] with same implementation) is introduced with one fewer restriction but it is unclear from the text how the reduced assumption impacts the current task. For instance, do the current tasks exhibit violations to the original stronger assumption in [9]? \n\n\nOther questions/comments:\n\n1. Please discuss/refer Figure 1(b-c) in the main text. \n2. Please use the same precision for numeric values in Table 1. Also, indicate what value of hyperparameter $\\gamma$ is chosen based just on the training data (without looking at test performance) for a fair comparison with the baselines. \n3. In Table 2, with dimA=2 and $\\gamma=1$, VCF strangely increases but HSCIC decreases. Is there a reason for this anomalous behavior? \n\n\n**References**\n\n[1] Victor Veitch, Alexander D’Amour, Steve Yadlowsky, and Jacob Eisenstein. Counterfactual invariance to spurious correlations in text classification. In Advances in Neural Information Processing Systems, 2021\n\n[2] Mouli, S. Chandra, and Bruno Ribeiro. \"Asymmetry learning for counterfactually-invariant classification in ood tasks.\" *International Conference on Learning Representations*. 2022.\n\n[3] Kusner, Matt J., et al. \"Counterfactual fairness.\" *Advances in neural information processing systems* 30 (2017).\n\n[4] Zhang, Kun, et al. \"Kernel-based conditional independence test and application in causal discovery.\" *arXiv preprint arXiv:1202.3775* (2012).\n\n[5] Doran, Gary, et al. \"A Permutation-Based Kernel Conditional Independence Test.\" *UAI*. 2014.\n\n[6] Chiappa, Silvia. \"Path-specific counterfactual fairness.\" *Proceedings of the AAAI Conference on Artificial Intelligence*. Vol. 33. No. 01. 2019.\n\n[7] Ilya Shpitser and Judea Pearl. Complete identification methods for the causal hierarchy. Journal of Machine Learning Research, 9:1941–1979, 2008.\n\n[8] Ilya Shpitser and Judea Pearl. Effects of treatment on the treated: Identification and generalization. In Proceedings of the Twenty-Fifth Conference on Uncertainty in Artificial Intelligence,, pp. 514–521, 2009.\n\n[9] Junhyung Park and Krikamol Muandet. A measure-theoretic approach to kernel conditional mean embeddings. In Advances in Neural Information Processing Systems, pp. 21247–21259, 2020.\n\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "ex2eJyH5cg9", "reviewer": "Reviewer_RdaH", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper aims to learn predictors that are invariant under counterfactual changes of certain covariates. To achieve this goal, the authors design a counterfactual invariant predictor. The implementation is highly based on the Kernel mean embeddings and conditional measures. By deriving the sufficient condition of the independence between Y and A, the authors propose an optimizable target to learn the independence. In the experiments, many experiments are conducted to verify the model effectiveness.\n", "summary_of_the_review": "See the above comments.", "strength_and_weaknesses": "In general, the paper is well written. The studied problem is interesting, which can be used in a lot of scenarios, for example fairness and so on. The proposed model is interesting, and the theory seems to be solid. My concerns come from two point: the first one is that, why we must use Kernel mean embeddings as the tool to test independence. Whether can we use other techniques? If there are advantages, please detail them to make this paper model compact. The second concern is about the experiment, fairness should be a very common direction. There should be more baselines, without enough baselines, it is hard to say the proposed model is effective.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "ZQszyeqIwh", "reviewer": "Reviewer_yLyi", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors propose counterfactual invariance, which is more general than some previous invariance objects. They show that the counterfactual invariance is equivalent to a conditional independence relation. Hence, by restricting conditional independence, the predicted target is counterfactual invariance. In the implementation, they add the restriction on conditional independence into the loss function, which can be achieved by HSICC. ", "summary_of_the_review": "#-----------------After rebuttal----------------------#\n\nI want to thank the authors for their rebuttal. For me, I think the proposed method is novel, and the paper indeed provides a good perspective for \"invariant predictors\". After reading other reviewers' comments, I realize that there are some parts unclear or unsuitable right now. More importantly, I think the anonymous comments make sense. Although the authors give responses, it is not very clear to me. I think this part is quite important and needs some more discussion, which possibly needs another round of review if there are no new discussions from the authors or other reviewers. Hence I adjust my score right now.\n\n\n#--------------------------#\n\n\nOverall, the paper is of high quality. The contribution is novel and the writing is clear. I like this paper and I am happy to see it accepted. ", "strength_and_weaknesses": "Strength:\n1. It is a very impressive result that even if the counterfactual is not identifiable, the counterfactual invariance can still be achieved.\n2. The paper is clearly written.\n3. The proposed counterfactual invariance is very novel\n\nWeaknesses:\nCurrently I could not give any specific weaknesses of this paper, but only some questions. See the questions in the next part.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "XU4KqWy4eJ", "reviewer": "Reviewer_zfVq", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a method to learn predictors that are invariant under counterfactual changes of certain covariates. The proposed method adds a model-agnostic regularization term based on the notion of conditional independence and kernel mean embeddings, to enforce the so called \"counterfactual invariance\" during training phase. Theoretical soundness of the proposed method is proved. Empirical experiments on both synthetic and real-word data sets are presented as well.", "summary_of_the_review": "The proposed method and corresponding theory of this work is a nice and significant improvement over the work of [Veich et al. (2021)]. However, it misses some key experiments which greatly affects the practicality of the proposed method. ", "strength_and_weaknesses": "Strength:\n1. The notion of counterfactual invariance and the corresponding theory of the proposed method is a nice improvement over that of [Veitch et al. (2021)].\n2. The proposed method is very intuitive.\n3. The presentation of this paper is clear, organized, and easy to follow.\n4. The literature review is thorough, with connections and improvements clearly stated.\n\nWeakness:\n1. I wonder how practical it is in reality to use the proposed method. For example, the estimation of HSCIC seems computationally expensive, what would be the extra computational cost for the added the regularization term?\n2. I understand that the proposed method has theory guarantees for counterfactual invariance which many heuristic method does not possess. However, in practice, does it really outperform those heuristic methods? I personally think this is very important and should be clearly answered in the paper (the answer does not necessarily have to be a \"yes\", but a clear comparison should at least be presented). For example, in the experiments, I would expect to see some comparison with those well-known methods like data augmentation. However, the current experiment section seems limited, and does not contain any such information. I would encourage authors to add more informative experiments to convince readers like myself that we should actually use the proposed method instead of the other famous heuristics we've been used to. \n3. Also, as mentioned by the authors themselves, the assumption of a known causal graph seems very strong, and could be another potential reason to prevent people from actually using the proposed method.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject