Learning Counterfactually Invariant Predictors
Authors: Francesco Quinzan
Cecilia Casolo
Krikamol Muandet
Yucen Luo
Niki Kilbertus
Venue: Accepted by TMLR ()

Abstract: Notions of counterfactual invariance (CI) have proven essential for predictors that are fair, robust, and generalizable in the real world. We propose graphical criteria that yield a sufficient condition for a predictor to be counterfactually invariant in terms of a conditional independence in the observational distribution. In order to learn such predictors, we propose a model-agnostic framework, called Counterfactually Invariant Prediction (CIP), building on the Hilbert-Schmidt Conditional Independence Criterion (HSCIC), a kernel-based conditional dependence measure. Our experimental results demonstrate the effectiveness of CIP in enforcing counterfactual invariance across various simulated and real-world datasets including scalar and multi-variate settings.

Published in Transactions on Machine Learning Research (07/2024)
Learning Counterfactually Invariant Predictors
Francesco Quinzan"
Department of Computer Science, The University of Oxford
Cecilia Casolo
Techincal University Munich
Helmholtz Munich
Munich Center for Machine Learning (MCML)
Krikamol Muandet
CISPA Helmholtz Center for Information Security
Yucen Luo
Citadel Securities
Niki Kilbertus
Techincal University of Munich
Helmholtz Munich
Munich Center for Machine Learning (MCML)
equal contribution
Reviewed on OpenReview: https: // openreviee. net/ forum? id=pRt VwIDPs
Abstract
Notions of counterfactual invariance (CI) have proven essential for predictors that are fair, robust, and generalizable in the real world. We propose graphical criteria that yield a sufficient condition for predictor be counterfactually invariant terms of a conditional independence the observational distribution. In order to learn such predictors, we propose model-agnostic framework, called Counterfactually Invariant Prediction (CIP), building on the Hilbert-Schmidt Conditional Independence Criterion (HSCIC), kernel- based conditional dependence mcasure. Our experimental results demonstrate the effectiveness of CIP in enforcing counterfactual invariance across various simulated and real-world datasets including scalar and multi-variate settings.
1 Introduction
Invariance, equivariance certain data transformations, has proven essential in numerous applications of machine learning (ML), since it can lead to better generalization capabilities (Arjovsky et al., 2019; Chen et al., 2020; Bloem-Reddy & Teh, 2020). For instance, in image recognition, predictions ought to remain unchanged under scaling, translation, or rotation of the input image. Data augmentation, an carly hcuristic to promote such invariances, has become indispensable for successfully training deep neural networks (DNNs) (Shorten & Khoshgoftaar, 2019; Xie et al., 2020). Well-known examples of "invariance by design" include convolutional neural networks (CNNs) for translation invariance (Krizhevsky et al., 2012), group oquivariant NNs for general group transformations (Cohen & Welling, 2016), recurrent neural networks (RNNs) and transformers for sequential data. (Vaswani et. al., 2017), DeepSet (Zaheer et al., 2017) for sets, and graph neural networks (GNNs) for different types of geometric structures (Battaglia et al, 2018).
Part of this work done while Francesco Quinzan visited the Max Planck Institute Intelligent, Systems, Tüibinge, Germany.

---

Published in Transactions on Machine Learning Research (07/2024)
Many applications in modern ML, however, call for arguably stronger notions of invariance based causality. This case has been made for image classification, algorithmic fairness (Hardt et. al., 2016; Mitchell et al., 2021), robustness (Bühlmann, 2020), and out-of-distribution generalization (Lu et al., 2021). The goal is invariance with respect to hypothetical manipulations of the data generating process (DGP). Various works develop methods that assume vonaal distributions (across environments or between training and test) to be governed by shared causal mechanisms, but differ due to various types of distribution shifts encoded by the causal model (Peters et al., 2016; Heinze-Deml 2019; Btihlmann, 2020; Subbaswamy et al., 2022; Yi et al., 2022; Makar et al., 2022). Typical goals include to train predictors invariant to such shifts, to learn about causal mechanisms and to improve robustness against spurious correlations or out of distribution generalization. The term "counterfactual invariance" has also been used in other out of distribution learning contexts unrelated to our lask, e.g., to denote invariance to certain symmetry transformations (Mouli & Ribeiro, 2022).
al., 2018; Rojas- Carulla et al., 2018; Arjovsky et al.,
While we share the broader motivation, these works are orthogonal to ours, because even though counterfactual distributions are also generated from the same causal model, they are fundamentally different from such shifts (Peters et al., 2017). Intuitively, counterfactuals are about events that did not, but could have happened had circumstances been different in a controlled way. A formal discussion of what mean by counterfactuals is required to properly position our work in the existing literature and describe our contributions. The main contribution of this work is combining a solid theoretical foundation provided in Theorem 3.2 and the flexible, easy-to-use, practical learning framework. Specifically, our work provides a graphical criterion for counterfactual causal inference under specific injectivity conditions, alongside a new model-agnostic learning framework named Counterfactually Invariant Prediction (CIP), which employs a kernel-based conditional dependence measure effective across mixed variable types. We empirically assess CIP in (semi-)synnthtia settings and show that it effectively enforces counterfactual invariance.
2 Problem setting and related work
2.1 Preliminaries and terminology
Definition 2.1 (Structural causal model (SCM)). A structural causal model is a (U, such that is a set of background variables that are ezogenous to the model; V is a set of observable (endogenous) variables; F = (fvlvev is a set of functions from (the domains of) pa(V)UU Uv to (the domain of) V, where Uv C U and pa(V) C VV(V) such that V = (v((aVV),, Uv); ; Pu is a probability distribution over the domain of U. Further, the subsets pa(V) CV (V} are chosen such that the graph & over V where the edge V + V is in G if and only if V' E pa(V) is a directed acyclic graph (DAG).
Observational distribution. An SCM implics unique observational distribution over V, which can be thought of as being generated by transforming the distribution over IUu via the deterministic functions in F iteratively to obtain a distribution over V.
Interventions. Given variable A E V, an intervention A + amounts to replacing fa in F with the constant function setting A to a. This yields a new SCM, which induces the interventional distribution under intervention A + a.² Similarly, we can intervene on multiple variables V 2 A + a. For an outcome (or prediction target) variable Y V, we then write Ya for the outcome in the intervened SCM, also called potential outcome. Note that the interventional distribution PY, (y) differs in general from the conditional distribution (y) a).3 This is typically the case when Y and A have shared ancestor, i.c., they are confounded. In interventional distributions, potential outcomes are random variables via the exogenous variables u, ie., Y. (u) where u Pu. Hence, interventions capture "population level" properties, i.e., the action is performed for all units u.
Counterfactuals. Counterfactuals capture what happens under interventions for a "subset" of possible units u that are compatible with observations W = w for a subset of obscrved variables W V. This can be described in a three step procedure. (i) Abduction: We restrict our atlention to units compatible with the
Note that all randomuess stems from Pu. The observational distribution is well-defined and unique, essentially bocause every DAG allows for a topological order. 2The observational distribution in intervened SCM is called interventional distribution of the base SCM.
We use P for distributions (common in the kernel literature) and Y. instead of the do notation.
2

---

Published in Transactions on Machine Learning Research (07/2024)
observations, i..,, consider the new SCM SW -U, (U,V,F,PPy Fu//w (ii) Intervention: Within S", perform an intervention A f- a on some variables A (which need not be disjoint from W). (iii) Prediction: Finally, we are typically interested in the outcome Y in an interventional distribution of Sw, which we denote by Py:) and call a counterfactual distribution: "Given that we have observed W = =ww, what would Y have been had we set A < a, instead of the value A has actually taken?". In this notation the conditioning set. W conditioned on the pre-interventional world. Counterfactuals capture properties of a "subpopulation" u ~ Pu|w=w compatible with the observations." Even for fine-grained W, there may be multiple units u in the support of this distribution. In contrast, "unit level counterfactuals" often considered in philosophy contrast Y (u) with Y for single unit u. Such unit level counterfactuals are too fine-grained in our setting. Hence, our used definition of counterfactual invariance is:
Definition 2.2 (Counterfact ual invariance). Let A, W be (nol necessarily disjoint) sets of nodes in a given SCM. Then, Y is counterfactually invariant in A w...t. W if FY:::== = (y-,) almost surely, for all a. a' in the domain of A and all w in the domain of W.5
Predictors in SCMs. Ultimately, we aim at learning a predictor Y for the outcome Y. Originally, the predictor Y is not part of the DGP, because we get to learn fy from data. Using supervised learning, the predictor fv depends both the chosen inputs X c V well as the target Y. However, once fr is fixed, it is a deterministic function with arguments X C so (U, (Ý), [fy], Pu) is a valid SCM and we can consider Y an observed variable with incoming arrows from only X. Hence, the definition of counterfactual invariance can be applied to the predictor Y.
Kernel mean embeddings (KME). Our method relies on kernel mean embeddings (KMEs). We describe the main concepts pertaining KMEs and refer the reader to Smola el al. (2007); Schölkopf et al. (2002); Berlinet & Thomas-Agnan (2011); Muandet et al. (2017) for details. Fix a measurable space D with respect to a o-algebra Far, and consider a probability mcasure P on the space (3/ F.r). Lct H be a reproducing kernel Hilbert space (RKHS) with a bounded kernel ky: 3/ x V + R, i.e., by is such that supyea k(y,y) < oc. The kernel mean embedding Me of is defined as the expected value of the function ky. with respect to y, i.e., up == IE [A( y)] The definition of KMEs can be extended to conditional distributions (Fukumizu et al,, 2013; Grünewälder et al., 2012; Song et al., 2009; 2013). Consider two random variables Y, S, and denote with Fx) and FF) the respective measurable spaces. These random variables induce a probability measure SYS in the product space fy x Ωg. Let Hy be a RKHS with a bounded kernel ky(-,-) on Ωy We define the KME of a conditional distribution P Pvis(- s) via Jyss=s == E [Ay( .y) S = s]. Here, the expected value is taken over y. KMEs of conditional measures can be estimated from samples (Grünewälder et al., Pogodin et al. (2022) recently proposed an efficient kernel-basse regularizer for learning features of input data that allow for estimating target while being conditionally independent of a distractor given the target. Since CIP ultimately enforces conditional independence (see Theorem 3.2), we believe it could further benefit from leveraging the efficiency and convergence properties of their technique, which we leave for future work.
2.2 Related work and contributions
While we locus on counterfactuals in the SCM framework (Pearl, 2000; Peters et al., 2016), there are different incompatible frameworks to describe counterfactuals (von Kügelgenn et al., 2022; Dorr, 2016; Woodward, 2021), which may give rise to orthogonal notions of counterfactual invariance.
Research on algorithmic fairness has explored a plethora of causal "invariance" notions with the goal of achieving fair predictors (Loftus et al., 2018; Carey & Wu, 2022; Plecko & Bareinboim, 2022). Kilbertus et al. (2017) conceptually introduce notion based on group-level interventions, which has been refined to take into account more fine- fineggrained context by Salimi et al. (2019); Galhotra et al. (2022), who then obtain fair predictors by viewing it as database repair problem or a causal feature selection problem, respectively. A counterfactull-level definition was proposed by Kusner et al. (2017) and followed up by path-spccific
*Note that conditioning in an Interventional distribution different from a counterfactual and our notation is quite subtle here Py: (y] J W = w) * WWith an abuse of notation, if I = 0 then the requirement of conditional comterfactual invariance becomes Py, (y) = Py_ (y) almost surely, for all a, a' in the domain of A. The "almost surely" part in our definition merely refers to the type of equality of distributions and is not related to the aalmost sure" in a.s.-CT defined by Fawkes & Evans (2023).
3

---

Published in Transactions on Machine Learning Research (07/2024)
counterfactual notions (Nabi & Shpitser, 2018; Chiappa, 2019), where the protected attribute may take different values along different paths to the outcome. Recently, Dutta et al. (2021) developed an information theoretic framework to decompose the overall causal influence allowing for exempted variables and properly dealing with synergies across different paths.
Our focus is on counterfactuals because they are fundamentally more expressive than mere interventions (Pearl, 2000; Bareinboim et al., 2022), but do not require fine-grained path- or variable-level judgment of "allowed" and "disallowed" paths or variables, which may be challenging to devise in practice. Since CI already requires strong assumptions, we leave path-specific counterfactuals-ove terms of ideetifiability (Avin et al., 2005)- for future work. While our Definition 2.2 requires equality in distribution, Veitch et al. (2021) suggest a definition of a counterfactually invariant predictor fy which requires almost sure equality of Y; and Y:: where we view Y an observed variable in the SCM as described above. Fawkes & Evans (2023) recently carefully formulated varions precise technical definitions of what counterfactual invariance may mean in different contexts, such as "almost sure CT", "distributional CT", and "CI of predictors". They provide various connections between them such as the fact that fy being F-CI is equivalent to Y being D-CI conditioned on X, yielding equivalence to the definition of counterfactual fairness (Kusner et al., 2017). The notion of counterfactual invariance in Definition 2.2 is most closely related to D-CI from Fawkes & Evans (2023), but we do not enforce conditioning on the intervening variable.
more challenging in
Inspired by problems in natural language processing (NLP), Veitch el al. (2021) aim al "stress-testing" models for spurious correlations. It differs from our work in that they (i) focus only two specific graphs, and (ii) provide a necessary but not sufficient criterion for CI in terms of a conditional independence. Their method enforces the conditional independence via maximum mean discrepancy (MMD) (in discrele sellings only)- However, enforcing a consequence of CT, does not necessarily improve CT. Indeed, Fawkes & Evans (2023, Prop. 4.4) show that while a.s.-CI implies certain conditional independencies, no set of conditional independencies implics any bounds on the difference in counterfactuals. On the contrary, distributional notions of CI such as D-CT our Definition 2.2 are weaker than a.s.-CI (see (Fawkes & Evans, 2023, Lem. 2.4). Therefore, these weaker notions indeed be written oquivalently as conditional independencies in the observational distribution under additional assumptions about the data generating mechanism (Fawkes & Evans, 2023, Lem. A.3). For example, Fawkes & Evans (2023, Lem. A.3) shows that D-CI can be implied by conditional independence in special settings where the counterfactual distribution is identified from the observational one. Similarly, our reduction of CI (as in Definition 2.2) ) to conditional independence in Theorem 3.2 requires a strong injectivity assumption that essentially amounts to being able to remove exogenous uncertainty.
Contributions. We provide such a sufficient graphical criterion for D-C1 under an injectivity condition of structural equation. Depending on the assumed causal graph, this can also come at the cost of requiring certain variables to be observed. As our main contribution, we propose model-agnostic learning framework, called Counterfactually Invariant Prediction (CIP), using a kernel-based conditional dependence measure that also works for mixed categorical and continuous, multivariate variables. We evaluate CIP extensively in settings and demonstrate its efficacy enforcing counterfactual invariance even when the strict assumptions may be violated.
3
Counterfactually invariant prediction (CIP)
3.1 Sufficient criterion for counterfactual invariance
We will now establish sufficient graphical criterion to express CT conditional independence in the observational distribution, rendering it estimable from data. First, need some terminology.
Graph terminology. Consider a path ı (a sequence of distinct adjacent nodes) in DAG g. A set of nodes S is said to block n, if ग contains a triple of consecutive nodes A, B, C such that one of the following hold: (i) A + B- + or A B < C' or A + B + C and B E S; (ii) A + B +- C and neither B nor any descendent of is S. Further, we call π causal path between sets of nodes A, B, when is a directed path from a node in A to a node in B. A causal path æ a proper causal path if it only intersects A at the first. node in æ. Finally, we denote with GA the graph obtained by removing from all incoming arrows
4

---

Published in Transactions on Machine Learning Research (07 (07/2022 2024)
(a)
(b)
A
L
A
X
X
X
(d)
S
c
(f)
A
A
A
Figure 1: (a) Exemplary graph in which a predictor Y with Ý i CI in A w.r.t. {L,S}. (b)-(c) Causal and anti-causal structure from Veitch al. (2021) where X not causally influenced by A, X does not causally influence Y, and X is both influenced by A and influences Y. (d) Assumed causal structure for the synthetic experiments, see Section 4.1 for details. (e) Assumed causal graph for the UCI Adult dataset (Section 4.3), where A [Gender, Age]. (f) Causal structure for our semi-synthetic image experiments (Section 4.2), where A = [Pos.X}, U = (Scale), C {Shape, Pos. Pos.Y), L {Color, Orientation), and Y = [Outcome}.
into nodes in A. We now define the notion of valid adjustment sets (Shpitser et al., 2010, Def. 5), which our graphical criterion for CI relies on. Definition 3.1 (valid adjustment set). Let G be a cousal graph and let X, Y be disjoint (sets of) nodes in G. A sel of nodes S is a valid adjustment set for (X, Y), if (i) no element in S is descendant in Gx of any node W X which lies on proper causal path from X to Y, and (ti) S blocks all non non-causal paths from X to Y in g.
We now state the sufficient graphical criterion that renders CI equivalent to a conditional independence. The proof builds on known results from the literature such as the full characterization of valid adjustment sets fully observed structural causal models and the backdoor criteria, but the final statement requires a substantial original theoretical contribution.
Theorem 3.2. Let G be causal graph, A, W be two (not necessarily disjoint) sels of nodes in 9, such 0, let S be valid adjustment set for (A LAUW) U,W, Y). Further, for any random variable X E W. \ A denote with gx ba(XX,Ux] its structural equation, and suppose that pa(X) S CAU U.W. Suppose that gx is injective in the variable u." Then, in all SCMs compatiibl with G, if a predictor Y satisfies Y AUW S, then Y is counterfactually invariant in A with respect to W.
that (A U UW)nY b =
Assumptions. First, we do not assume the SCM (or any structural equations) to be known. We do assume the causal graph to be known. This a standard assumption widely made in the causality literature, even though it is a strong one (Cartwright, 2007). The additional assumption of Theorem 3.2, namely injectivity of gx is satisfied but more general than wide variety of commonly considered models in causality such the widely used Additive Noise Models (Peters et al., 2017). More broadly, Fawkes & Evans (2023, Lem. A.3) show that to achieve CI from the observational distribution and the causal graph, additional assumptions are always required.
3.2 Example use-cases of counterfactually invariant prediction
Fig. (a) shows an exemplary graph in which the outcome Y is affected by (disjoint) sets A (in which we want to be CI), L and S (inputs to fz). We consider W LUAUS. Here we aim to achieve Y AUL s to obtain CI A w.r.t. W. In our synthetic experiments, we also allow S to affect A, see Fig. (d). Let us further illustrate concrete potential applications of CI, which we later also study in our experiments.
TThe injectivity of gx defined follows. Consider two pairs {p, u) and {p, u') with p the support of pa(X) and a. in the support of Ux. Suppose that U (p. u) # and U (p. u() Then, holds g(p. = g(p, ) and only if n'.
5

---

Published in Transactions on Machine Learning Research (07/2024)
Counterfactual fairness. Counterfactual fairness (Kusner et al., 2017) informally challenges a consequential decision: "Would I have gollen the same oulcome had my gender, race, or age been different with all else being equal?". Here Y C V denotes the outcome and A C V \ Y the protected aftributes such as gender, race, or age pro-pproected under anti-discrimination laws (Barocas & Selbst, 2016)-by A CVV VYY. Y. Collecting all remaining observed covariates into W == V Y counterfactual fairness reduces to counterfact.aa invariance. In experiments, we build a semi-synthetic DGP assuming the graph in Fig. (ce) for the UCI adult datasct. (Kohavi & Becker, 1996).
Robustness. CI serves as a strong notion of robustness in settings such as image classification: " "Would the bruck have been classified correclly had il been winter in this silualion inslead of summer?" For concrete demonstration, we use the dSprites dataset (Matthey et al., 2017) consisting of simple black and white images of different shapes (squarcs, cllipsos, ...), sizes, orientations, and locations. We devise a DGP for this datasct. with the graph depicted in Fig. 1(().
Text classification. Veitch et al. (2021) motivate the importance of counterfactual invariance in text classification tasks. Specifically, they consider the causal and anti-causal structures depicted in Veitch et al. (2021, Fig. 1), which we replicate in Fig. ((b.c). Both diagrams consist of protected attributes A, observed covariates X, and outcomes Y. To apply our sufficient criterion to their settings, we must assume that A and Y are unconfounde.. We show that CIP still performs on par with Veitch et al. (2021) even when this assumption is violated. Theorem 3.2 provides a sufficient condition for CI (Definition 2.2) in terms of the conditional independence Y IL A U W S. We next develop operator HSCIC(Y, A U W ()) that is (a) efficiently estimable from data, (b) differentiable, (c) a monotonic measure of conditional dependence, and (d) is zero if and only if Y IL A A W | S. Hence, it is a practical objective to enforce CL.
3.3 HSCIC for conditional independence
Consider two sets of random variables Y and A U W, and denote with Fv) and FALw) the respective measurable spaces. Suppose that we are given two RKHSs Hy, HAUw over the support of Y and AU W respectively. The tensor product space Hy & HAuw is defined as the space of functions of the form (f @ gy, [a, w)) = f(y)a((a, w)), for all f E Hy and g E HAuw. The tensor product space yields a natural RKHS structure, with kernel k defined by k(y @ [a, w), @ [a', w')) ==
w). [a', w'].. We refer the reader to Szabó & Sriperumbudur (2017) for more details on tensor product spaces.
Definition 3.3 (HSCIC). For (sets of) random variables Y, AU W. S, the HSCIC between Y and AUW given S is defined as the veal-valued random variable HSCIC(Y, AUW S) = HY, o where HY,A.W]S is a real-valued deterministic function, defined as HY (uW = [|lY, AUW/Ss, HY|S= & PALwiS=s! ll with II-I1 the norm induced by the inner product of the tensor product space Hx @ HAUW.
Our Definition 3.3 is motivated by, but differs slightly from Park & Muandet (2020, Def. 5.3), which relies on the Bochner conditional expected value. While it is functionally equivalent (with the same implementation, see Eq. (2)), ours has the benefit of bypassing some technical assumptions required by Park & Muandet (2020). The HSCIC has the following important property.
Theorem 3.4 (Theorem 5.4 by Park & Muandet (2020)). If the kernel k of Hx @ HAuw is characteristic", HSCIC(Y, A U W S) = abmost surely if and only if Y LLAWWWS
Because "most interesting" kernels such as the Gaussian and Laplacian kernels are charactcristic, and the tensor product of translation-in inn-iivaiia characteristic kernels is characteristic again (Szabó & Sriperumbudur, 2017), this natural assumption is non-restrictive in practice. Combining Theorems 3.2 and 3.4, we can now use HSCIC to reliably achieve counterfactual invariance.
Corollary 3.5. Under the assumptions of Theorem 3.2, if HSSICCY, AUW S) 0 almost surely, then Y is counterfactually invariant. in A with respect to W.
In addition, since HSCIC is defined in terms of the MMD (Definition 3.3 and Park & Muandet (2020, Def. 5.3)): it inherits the weak convergence property, i.c., if HSCIC(Ý, AUW S) converges to zero, then the counterfactual distributions (for different intervention values a) weakly converge to the same distribution. We refer to Simon- Gabriel & Schölkopf (2018); Simon-Gabriel et al. (2020) for precise characterization. Hence,
The tensor product kernel is characteristic if PY,AUW + Fy.laww) [A[ .y @ [a, w]]] is injective.
6

---

Published in Transactions on Machine Learning Research (07/2024)
HSCIC decreases, the predictor approaches counterfactual invariance and we need not drive HSCIC all the way to zero to obtain meaningful results.
3.4 Learning counterfactually invariant predictors (CIP) Corollary 3.5 justifies our proposed objective, namely to minimize the following loss
CCTP Y = ((Y)+))))))))))) + HSCIC(Y, S),
[CIP loss]
(1)
where L(Y) is a task-dependent loss function (c.g., cross-entropy for classification, mean squared error for regression) and 7 2 0 is a parameter that regulates the trade-off between predictive performance and counterfactual invariance.
The meaning of 7 and how to choose it. The second term in Eq. (1) amounts to the additional objective of CI, which is typically at odds with predictive performance within the observational distribution L. In practice, driving HSCIC to zero, i.e, viewing our task as a constrained optimization problem, typically deteriorates predictive performance too much to be useful for pesiiction--spcciallly in small data settings." 8 As the choice of y amounts to choosing an operating point between predictive performance and CT, camnot: be selected in a data-driven fashion. As different settings call for different tradeoffs, we advocate for employing (i) Train unconstrained predictor for base predictive performance (e.g.. 92%
the following procedure:
accuracy or 0.21 MSE). (ii) Fix tolerance level a, indicating the maximally tolerable loss in predictive performance (e.g., at most 5% drop in accuracy or at most 10% increase in MSE). (iii) Perform a log-spacod binary search on 7 (e.g., on [10-1, 104)) to find the largest 7 such that the predictive performance of the resulting predictor achieves predictive performance within the tolerance a. A similar search for the optimal value of y can conducted, when there a fixed requirement for a maximum tolerance counterfactual invariance as measured by HSCIC.
Estimating the HSCIC from samples. The key benefit of HSCIC as a conditional independence measure is that it does not require parametric assumptions on the underlying probability distributions, and it is applicable for any mixed, multi-dimensional data modalities, long we define positive definite kernels them. Given n samples { a, Wa, Wi), denote with Kg the kernel matrix with entries (Ry == (S,,,, and let R RA.w be the kernel matrix for A U W. We estimate the Hy,AUWIS = Hy, (t,Auw))() as
A Y.WWIS = o Y.UWIS (R& FEyORAm RAUW) ve AUWIS
(2)
- W (Y ( (UTUWS 1 Y K Kx (GSUWIRRAWWAAUUS)
where ② is element-wise multiplication. The functions S ( AuWiS @AUWIs()), and ALWAS = (uWI(s))(()) are found via kernel ridge regression. Caponnetio & Vito (2007) provide the convergence rates of the estimand Ar YAUiI under mild conditions. In practice, computing the HSCIC approximation by the formula in Eq. (2) can be computationally expensive. To speed it up, random Fourier features to approximate Kg and RaLw (Rahimi & Recht, 2007; Avron et. al., 2017). We emphasize that Eq. (2) allows us to consistently estimate the HSCIC from observational i.i.d. samples, without prior knowledge of the counterfactual distributions.
use
3.5 Measuring counterfactual invariance.
Besides predictive performance, e.g., mean squared error (MSE) for regression or accuracy for classification, our key metric of interest. is the level of counterfactual invariance achieved by the predictor Y. First, emphasize again that counterfactual distributions are generally not identified from the observational distribution (i.c., from available data) mcaning that CI is generally untestable in practice from observational
iIn particular, HSCIC does regularize posed prohlem, f.e., does merely break ties between predictors with equal ((Y). Hence also need not decay to zero as the sample size increases.
7

---

Published in Transactions on Machine Learning Research (07 2224)
x10-3
x10 3
x10-2
x x10-3
x10-2
100
7.5
method CF Level Level2
trade parameter
5.0
5.0
5.0
MSE
3.5
VCF
VCF
5.0
naive bandine
500
2.5
2.5
2.5
3.0
230
0.0
0.0
0.0
0.0
10
10
10
10
10-
10
10
10
10
10a
0.0
trade-off parameter Y
trade-off parameter y
MSE
Figure 2: Results on synthetic data. Left: tradc-off between MSE and counterfactual invariance (VCF). Middle: strong correspondence between HSCIC and VCF. Right: performance of CIP against baselines CF1 and CF2 and the naive baseline. As increases, CIP traces out a frontier characterizing the trade-off between MSE and CL. CF2 and the naive baseline are Pareto-dominated by CIP, i.e., we can pick y to outperform CF2 in both MSE and VCF simultaneously. CF1 has zero VCF by design, but worse predictive performance than CIP at near zero VCF. Error bars are standard errors over 10 seeds.
data. We can thus only evaluate CI in (semi-)synthettic settings where have access to the full SCM and thus all counterfactual distributions.
A mcasure for CI must capture how the distribution of Y changes for different values of a' across all couditioning values w (which may include an observed value = =a) We propose the Variance of CounterFactuan (VCF) as metric of CI
VCF(Y) == VCF(Y)==Ew--))
Ew (E
(3)
That is, we quantify how the average outcome varies with the interventional value a' at conditioning value w and average this variance over w. For deterministic predictors (point estimators). which we use in all our experiments, the prediction is a fixed value for each input (w./ [8] y) and we drop the inner expectation of Eq. (3). In this case, the variance term in Eq. (3) is zero if and only if P Pv:/w- w (y) = (y) almost surcly. Since the variance is non-negative, the outer expectation is zero if and only if the variance term is zero almost surely, yielding the following result,
Corollary 3.6. For point-estimators, Y is counterfactually invariant in A w.r.t. W if and only if VCF(Y) almost surely.
Estimating VCF in practice requires access to the DGP to generate counterfactuals. Given d i.idd examples (wi) from fixed observational da dataset we sample & intervention values from the marginal PA and compute corresponding predictions. The inner expectation is simply the deterministic predictor output, and we compute the empirical expectation over the observed w values and empirical variances over the k sampled interventional values (for each w). Since the required counterfactuals are by their very nature unavailable in practice, our analysis of VCF limited to (semi-)synthetic settings. Notably, the proposed procedure for choosing y does not require VCF. Our experiments corroborate the weak convergence property of HSCIC- small HSCIC implics small VCF. Hence, HSCIC may serve strong proxy for VCF and thus CI in practice.
In practical terms, HSCIC is considered the relevant metric, which is also estimable from data. Theoretically. however, counterfactual invariance is only implicd by an exact HSCIC value of zero (Corollary 3.5). Even if HSCIC exhibits coutinuity, meaning it approaches zero as distributions converge weakly, VCF may still be preferable for directly assessing counterfactual invariance because it provides a more interpretable metric. Our experimental investigations into VCF aim to establish the practical utility of HSCIC measure of counterfactual invariance.
4 Experiments
Baselines. As many existing methods focus on cruder purely observational or interventional invariances (see Section 2.2), our choice of baselines for true counterfactual invariance is highly limited. First, we compare
8

---

Published in Transactions on Machine Learning Research (07 2224)
10-dimensional A
50-dimensional A
x10-
x10
x10-
X10
1.0
0.8
0.8
VCF
MSB
0.6
VCF
MSE
0.4
0.4
0.2
0.0
10 100 500 trade-off parameter y
100 500 trade-off parameter
5
100 500
5 50 100 so0 trade-off parameter
trade-off parameter
Figure 3: MSE and VCF for synthetic data with 10- and 50-dimensional A for different y and 15 random seeds per box. CIP reliably achieves CI as y increases.
CIP to Veitch et al. (2021) in their two limited settings, showing that method performs par with theirs. Next, compare to two methods proposed by Kusner et al. (2017) in settings where they apply. CF1 (their 'Level I') consists of only using non -ondeescess of A as inputs to fy. CF2 (their 'Level 2') assumes an additive noise model and uses the residuals of descendants of A after regression on A together with non-descendants of A as imputs to fg. We refer to these two baselines as CF1 and CF2 respectively. We also compare CIP to the 'naive baseline' which consists training a predictor ignoring A. In settings where A is binary, we also compare Chiappa (2019), devised for path-wise counterfactual fairness. Finally, develop heuristics based on data augmentation as further possible basclines.
4.1 Synthetic experiments
First, we generate various synthetic datasots following the causal graph in Fig. 1dd) They contain (i) the prediction targets Y, (ii) variable(s) we want to be CI in A, (iii) covariates L mediating effects from A on Y, and (iv) confounding variables S. The goal to learn a predictor Y that is CT in A w.r.t. W = AULU. The datasets cover different dimensions for the observed variables and their correlations.
Model choices and parameters. For all synthetic experiments, we train fully connected neural networks (MLPs) with MSE loss LuSE (Y) as the predictive loss C in Eq. (1) for continnous outcomes Y. We gencrate 10k samples from the observational distribution in each setting and use an to 20 train-lest split. All metrics reported are on the test set. We perform hyper-parameter tuning for MLP hyperparameters based on random strategy. The HSCIC(Ý, U WW S) term is computed as in Eq. (2) using a Gaussian kernel with amplitude 1.0 and length scale 0.1. The regularization parameter À for the ridge regression coefficients is set to x = 0.01. We set = 1000 and k 500 in the estimation of VCF.
Model performance. We first study the ef- fect of the HSCIC on accuracy and counterfac- tual invariance on the simulated dataset. Fig. 2 (left.) depicts the expected trade-off between MSE and VCF for varying 7/ whereas Fig. 2 (middle) highlights that HSCIC (estimable from observa- tional data) is a strong proxy of counterfactual invariance measured by VCF (see discussion after Eq. (3)). Figure 2 (right) compares CIP to baselines for simulated non-additive noise model. For suit- able choice of 7, CIP outperforms the bascline CF2 and the naive baseline in both MSE and VCF simultaneously. While CF1 achieves perfect CI by construction (VCF 0), its MSE higher than CIP at almost perfect CI (VCF near zero). To conclude, our method flexibly and reliably trades predictive performance for counterfactual invariance via a single parameter 7 and Pareto-dominates existing methods.
1.2
1.0-
MSE
VCF
0.1
0.6
0.0
0
10
100
500
10
100
500
trade-off parameter
trade-off parameter Y
Figure 4: On the dSprites image dataset, CTP trades off MSE for VCF and achieves almost full CI as increases. Boxes are for 8 random seeds.

---

Published in Transactions on Machine Learning Research (07/2024)
Effect of dimensionality of A. A key advantage of CIP is that it can deal with multi-dimensiona A. We consider simulated datasets, where we gradually increase the dimension of A. The results in Fig. 3 for different trade-off parameters y and different dimensions of A demonstrate that CIP effectively enforces CI also for multi-dimensional A.
4.2 Image experiments
We consider an image classification task on the dSprites dataset (Matthey al., 2017), with a causal model depicted in Fig. 1(f). This experiment is particularly challenging due to the mixed categorical and continuous variables in C (shape, y-pos) and L (color, orientation), with continuous A (x-pos). We seek predictor Y that is CI in the x-position w.r.t. all other observed variables. Following Theorem 3.2, we achieve Y. LL {x-pos, scale, color, orientation) (shape, y-pos via the HSCIC operalor. To accommodate the mixed imput types, we first extract features from the images via a CNN and from other inputs via an MLP. We then use an MLP on all concatenated features for Y. Fig. 4 shows that CIP gradually enforces CI as y increases and illustrates the inevitable increase of MSE.
4.3 Fairness with continuous protected attributes
Finally, we apply CIP to the widely-used UCI Adult dataset (Kohavi & Becker, 1996) and com- pare it. against a 'naive baseline' which simply ignores A, CF1, CF2, and path-specifie counter- factual fairness (PSCF) (Chiappa, 2019). Like CF1, PSCF always achieves VCF = 0 by design.
We explicitly acknowledge the shortcomings of the UCI Adult dataset to reason about social jus- tice (Ding et al., 2021). Instead, we chose it due to previous investigations into plausible causal structures based on domain knowledge (Zhang et al., 2017). The task is to predict whether an individual's 's income is above a threshold based on demographic information, including protected attributes. We follow Nabi & Shpitser (2018);
accuracy (%)
2
VCF
10
74
10-1
10-4
Chiappa (2019), where a causal structure is sumed for subset of the variables as in Fig. ((e).
0 10 100 1000 dH CRAPSEF NB trade-off parameter y
0 10 100 1000 CF2 NB
trade-off parameter
We choose gender and age as the protected at- tributes A, collect marital status, level of educa- tion, occupation, working hours per week, and work class into L, and combine the remaining observed attributes in C. Our aim is learn predictor Y that is CI in A w.r.t. W = CCUL. Achieving (causal) fairness for (mixed categorical and) continuous protected attributes is under active investigation (Mary et al., 2019; Chiappa & Pacchiano, 2021), but directly supported by CIP.
Figure 5: Accuracy and VCF on the Adult dataset. CIP achieves better VCF than CF2 and the naive baseline (NB), improved in accuracy compared to PSCF and is on par with CF1 in accuracy at VCF x 0.
We use an MLP with binary cross-entropy loss for Y. Since this experiment is based on real data, the true counterfactual distribution cannot be known. Following Chiappa & Pacchiano (2021) we estimate a possible SCM by inferring the posterior distríbution over the unobserved variables using variational autocncoders (Kingma& & Welling, 2014). Even though CF1, PSCF always achieves VCF by design, Figure shows that CIP gradually achieves CI and even manages to keep a constant accuracy after an initial drop. It outperforms PSCF in accuracy and reaches comparable accuracy to CF1 when reaching VCF 0. It Pareto-dominates CF2 and can achieve better VCF than the naive baseline, even though NB is more accurate for 2 5 than CTP.
5 Discussion and future work
We developed CTP, a method to learn counterfactually invariant predictors Y. First, we presented a sufficient. graphical criterion characterize counterfactual invariance and reduced it to conditional independence in the observational distribution under an injectivity assumption of causal mechanism. We then built on
IIn all hoxplots, boxes represent the interquartile range, the horizontal line the median, and whiskers show minimum and maximum values, excluding outliers (dots).
10

---

Published in Transactions on Machine Learning Research (07/2024)
kernel mean embeddings and the Hilbert-Schmidt Conditional Independence Criterion to devise efficiently estimable, differentiable, model-agnostic objective to train CT predictors for mixed wawiiuuuuustttttiiiiiioooo multi-dimennional variables. We demonstrated the efficacy of CIP in extensive empirical evaluations on various regression and classification tasks.
A limitation of this work is the computational cost of estimating HSCIC, which may limit the scalability of CIP to very higl-dimensional settings even when using efficient random Fourier features. While the increased robustness of counterfactually invariant predictors are certainly desirable in many contexts, this presupposes the validity of our assumptions. Thus, an important direction for future work is assess the sensitivity of CIP to misspecifications of the causal graph insufficient knowledge of the roquired adjustment set. Lastly, envision our graphical criterion and KME-based objective to be useful also for causal representation learning to isolate causally relevant, autonomous factors underlying the data.
Another key limitation of our work, shared by all studies in this domain, is the assumption that the causal graph is known. Assessing the validity of a causal DAG in real-world settings is challenging, since one can often conjecture plausible confounding mechanisms for any missing edge. For this reason, in this work we found it challenging provide convincing examples that corroborate the "potential validity" of our assumptions. However, our empirical results demonstrate that CIP is effective even when these assumptions are violated. Similarly, our real-world experiments show comparable invariance and accuracy trade-off trend as the simulated experiments. We believe this to be better indicator of validity and robustness than isolated toy examples where assumptions may be satisfied. Practitioners using our proposed framework are recommended to take these considerations into account before they apply the method to their own data, and combine causal discovery methods with domain knowledge to assess the validity of the underlying DGP.
Acknowledgements
Francesco Quinzan acknowledges funding from ELSA: European Lighthouse on Secure and Safe Al project (grant agreement No. 101070617 under UK guarantee). Cecilia Casolo is supported by the DAAD programme Konrad Zuse Schools of Excellence in Artificial Intelligence, sponsored by the Federal Ministry of Education and Research.
References
Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz, Invariant risk minimization. arXiv preprinl arXio:1907.0282 2019.
Chen Avin, Llya Shpitser, and Judea Pearl. Identifiability of path-specific effects. 2005.
Haim Avron, Michael Kapralov, Cameron Musco, Christopher Musco, Ameya Velingker, and Amir Zandieh. Random fourier features for kernel ridge regression: Approximation bounds and statistical guarantees. In International conference on machine learning, volume 70, pp.
2017.
Elias Bareinboim, Juan Correa, Duligur Tbeling, and Thomas Ieard. On pearl's hierarchy and the foundations of causal inference. Probabilistic and causal inference: the works of judea pearl, pp. 507-556. 2022.
Solon Barocas and Andrew D. Selbst. Big data's disparate impact. California Law Review, 104, 2016.
Peter W Battaglia, Jessica B Hamrick, Victor Bapst, Alvaro Sanchez-Gonzaler, Vinicius Zambaldi, Matcusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, et al. Relational inductive biases, deep learning, and graph networks. arXiv preprint orXiv:1806.010 01261, 2018.
Alain Berlinet and Christine Thomas-Agnan. Reproducing kernel Hilbert spaces in probability and statistics.
Springer Science & Business Media, 2011.
Benjamin Bloem-Reddy and Yee Whye Teh. Probabilistic symmetries and invariant neural networks. The Journal of Machine Learning Research, 21:90-1, 2020.
Peter Bühhlaan. Invariance, causality and robustness. Statislical Science, 35(3):404 -26, 2020.
11

---

Published in Transactions on Machine Learning Research (07/2024)
Andrea Caponnetto and Ernesto De Vito. Optimal rates for the regularized least-squares algorithm. Founda- lions of Compulational Muthematics, 7(3):331-300 368, 2007.
Alycia N Carey and Xintao Wu. The causal fairness field guide: Perspectives from social and formal sciences. Frontiers in Big Data, 5:892837, 2022.
Nancy Cartwright. Hunting couses and using them: Approaches in philosophy and economics. Cambridge University Press, 2007.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp- 1597-1607. PMLR, 2020.
Silvia Chiappa. Path-specific counterfactual fairness. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 7801-7808, 2019.
Silvia Chiappa and Aldo Pacchiano. Fairness with continuous optimal transport. arXiv preprint arXiw:2101.02054 2021.
Taco Cohen and Max Welling, Group equivariant convolutional networks. In International conference on machine learning, pp. 2990 2999. PMLR, 2016.
Frances Ding, Moritz Hardt, John Miller, and Ludwig Schmidt. Retiring adult: New datasets for fair machine learning. In Advances in Neural Information Processing Systems, volume 34, 2021.
Cian Dorr. Against counterfactual miracles. The Philosophical Review, (25 2016.
Sanghamitra Dutta, Praveen Venkatesh, Piotr Mardziel, Amupam Datta, and Pulkit Grover. Fairness under feature exemptions: Counterfactual and observational measures. IEEE Transactions on Information Theory, 67(10):6675-671) 6710, 2021.
Jake Fawkes and Robin J Evans. Results on counterfactual invariance. arXiv preprint arXin:2307.08519, 2023.
Kenji Fukumizu, Le Song, and Arthur Gretton. Kernel bayes' rule: Bayesian inference with positive definite kernels. Journal of Machine Learning Research, (113)335337788 2013.
Sainyam Galhotra, Karthikeyan Shanmugam, Prasanna Sattigeri, and Kush R Varshney. Causal feature selection for higc fairness. In Proceedings of the 2022 International Conference Management of Dala, pp. 276 285, 2022.
Steffen Grünewälder, Guy Lever, Arthur Grelton, Luca Baldassarre, Sam Patterson, and Massimiliano Pontil. Conditional mean embeddings as regressors. In International conference on machine learning, 2012.
Moritz Hardt, Eric Price, Eric Price, and Nati Srebro. Equality of opportunity in supervised learning.
In Advances in Newral Information Processing Systems, volume 29, 2016. URL https: https://proceedinggs Paper .pf.er.dff....
Christina Heinze-Deml. Jonas Peters, and Nicolai Meinshausen. Invariant causal prediction for nonlinear models. Journal of Causal Inference, 6(2), 2018.
Niki Kilbertus, Mateo Rojas Carulla, Giambattista Parascandolo, Moritz Hardt, Dominik Janzing, and Bernhard Schölkopf. Avoiding discrimination through causal reasoning. In Advonces in neural information processing systems, volume 30, 2017.
Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. In 2nd International Conference on Learning Representations, ICLR 2014, 2014. URL http:://axiiv httt:://arssi..r//bbb/////ss wwtpoo////////.oeeeeoooo
Ronny Kohavi and Barry Becker. Uci adult data set. UCT Meachine Learning Repository, 1996.
12

---

Published in Transactions on Machine Learning Research (07/2024)
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional
neural networks. In Advonces in Neural Information Processing Systems, volume 25, 2012. URL https: pdf.
//proceedings.neut neurips. httpc//////.//.cc/////////6//6/0
Matt J. Kusner, Joshua R. Loftus, Chris Russell, and Ricardo Silva. Counterfactual fairness. In Advances in
Neural Information Processing Systems, pp. 4066-4076, 2017.
Joshua R Loftus, Chris Russell, Matt Kusner, and Ricardo Silva. Causal reasoning for algorithmic fairness. arXiv preprinl Xrx: 05859, 2018.
Chaochao Lu, Yuhuai Wu, José Miguel Hernández-Lobato, and Bernhard Schölkopf. Invariant causal representation learning for out-of-distribution generalization. In International Conference on Learning Representations, 2021.
Maggie Makar, Ben Packer, Dan Moldovan, Davis Blalock, Yoni Halpern, and Alexander D D'Amour. Causally motivated shortcut removal using auxiliary labels. In International Conference Artificial Intelligence and Statistics, pp. 739 766. PMLR, 2022.
Jeremie Mary, Clément Calauzènes, and Noureddine Karoui. Fairness-aware learning for continuous attributes and treatments. In International Conference on Machine Learning, volume 97, pp. 4382 4391. PMLR, 2019.
Loic Matthey, Irina Higgins, Demis Hassabis, and Alexander Lerchner. dsprites: Disentanglement testing sprites datasct.
2017.
Shira Mitchell, Eric Potash, Solon Barocas, Alexander D' and Kristian Lum. Algorithmic fairness: Choices, assumptions, and definitions. Arawal Review of Statistics and Its Application, 8:141 163, 2021.
S. Chandra Mouli and Bruno Ribeiro. Asymmetry learning for counteractuuall--nnvriiaa classification in OOD tasks. In Proc. of ICLR, 2022.
Krikamol Muandet, Kenji Fukumizu, Bharath Sriperunbudur, and Bernhard Schölkopl. Kernel mean embedding of distributions: A review and beyond. Foundations and Trends in Machine Learning, 10(1-2): 1-141, 2017.
Razich Nabi and Πlya Shpitser. Fair inference outcomes. In Proceedings of the AAAI Conference Artificial Intelligence, volume 32, 2018.
Junhyung Park and Krikamol Muandet, A measure-theoretic approach to kernel conditional mean embeddings. In Advances in Neural Information Processing Systems, pp. 21247-21259, 2020.
Judea Pearl. Causality; Models, Reasoning and Inference. Cambridge University Press, 2000.
Jonas Peters, Peter Bühlmann, and Nicolai Meinshausen. Causal inference by using invariant prediction: identification and confidence intervals. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 7854):947-1012, 2016.
Jonas Peters, Dominik Janzing, and Bernhard Schölkopf. Flements of causal inference: foundations and learning algorithms. The MIT Press, 2017.
Drago Plecko and Elias Bareinboim. Causal fairness analysis. arXiv preprint arXtc:2207.11385 2022.
Roman Pogodin, Namrata Deka, Yazhe Li, Danica J Sutherland, Victor Veitch, and Arthur Gretton. Efficient conditionally invariant representation lcarning. arXiv preprint arXiv:2212. 086455, 2022.
Ali Rahimi and Benjamin Recht. Random features for large-scale kernel machines. In Advances in Newral Information Processing Systems, pp. 1177-1184, 2007.
Mateo Rojas-Carulla, Bernhard Schölkopf, Richard Turner, and Jonas Peters. Invariant models for causal transfer learning. The Journal of Machine Learning Research, 19(1):1309-133 -342,, 2018.
13

---

Published in Transactions on Machine Learning Research (07/2024)
Babak Salimi, Luke Rodriguez, Bill Howe, and Dan Suciu. Capuchin: Causal database repair for algorithmic fairness. arXiv preprint Xr::11902 08283, 2019.
Bernhard Schölkopf, Alexander J Smola, Francis Bach, et al. Learning with kernels: support vector machines, regularization, optimization, and beyond. MIT press, 2002.
Connor Shorten and Taghi M Khoshgoftaar. A survey on image data augmentation for deep learning. Journal of big data, 6(1):1-48, 2019.
Nya Shpitser, Tyler J. VanderWeele and James M. Robins. On the validity of covariate adjustment for estimating causal effects. In Proceedings of the "wenty-Sirth Conference on Uncertainty in Artificial Intelligence, pp. 527-536, 2010.
Carl-Johann SimrieGGabiel and Bernhard Schölkopf. Kernel distribution embeddings: Universal kernels, characteristic kernels and kernel metrics on distributions. Journal of Machine Learning Research, 19(44): 1-29, 2018.
Carl-Johann Simon-Gabriel, Alessandro Barp, Bernhard Schölkopf, and Lester Mackey. Metrizing weak convergence with maximum mean discrepancies. under review, 2020.
Alex Smola, Arthur Gretton, Le Song, and Bernhard Schölkopf. A hilbert space embedding for distributions. In International Conference on Algorithmic Learning Theory, pp. 13-31. Springer, 2007.
Le Song, Jonathan Huang, Alexander J. Smola, and Kenji Fukumizu. Hilbert space embeddings of conditional distributions with applications to dynamical systems. In International Conference on Machine Learning, volume 382, pp. 961 968, 2009.
Le Song, Kenji Fukumizu, and Arthur Gretton. Kernel embeddings of conditional distributions: A unified kernel framework for nonparametric inference in graphical models. IEFE Signal Processing Magazine, 30 (4):98 111, 2013.
Adarsh Subbaswamy, Bryant Chen, and Suchi Saria. A unifying causal framework for analyzing dataset shift-stable learning algorithms. Journal of Causal Inference, 10(1):64-880 89, 2022.
Zoltán Szabó and Bharath K. Sriperumbudur. Characteristic and universal tensor product kernels. Journal of Machine Learning Research, 18:233:1 -23:22, 2017.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30, 2017.
Victor Veitch, Alexander D' D'Amorr, Steve Yadlowsky, and Jacob Fisenstein. Counterfactual invariance to spurious correlations in text classification. In Advances in Neural Information Processing Systems, pp. 16196-16208, 2021.
Julius von Kügelgen, Abdirisak Mohamed, and Sander Beckers. Backtracking counterfactuals. arXiv preprinl arXiv:2211.00/78 2022.
James Woodward. Causation with human face: Normative theory and descriptive psychology- Oxford University Press, 2021.
Qizhe Xie, Zihang Dai, Eduard Hovy, Thang Luong, and Quoe Le. Unsupervised data aug- mentation for consistency training. In Advances in Neural Information Processing Systems, volume 33, pp. 6256-6268, 2020. URL https://proceeddngs neurips cttccc//pppcac.ccccccccccccp/t cc/paper/2000//11e/ M4fa0061a822119957788331111 Pappeeiddd-Pap..p....... pdf..
Mingyang Yi, Ruoyu Wang, Jiacheng Sun, Zhenguo Li, and Zhi-Ming Ma. Breaking correlation shift via conditional invariant. regularizer. In International Conference Learning Representations, 2022.
14

---

Published in Transactions on Machine Learning Research (07/2024)
Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Russ R Salakhutdinov, and Alexander J Smola. Deep sets. In Advonces in Neural Information Processing Systems, volume 30, 2017. URL https: .df.
//proceedings neurips. http:h///////....cec..aaa.a//aaa
Lu Zhang, Yongkai Wu, and Xintao Wu. Achieving non-diserimination in data release. In International Conference Knowledge Discovery and Data Mining, pp. 1335-1344, 2017.
15


Reviews:
[{"review_id": "NEYZnpt6Hh", "reviewer": "Reviewer_sMhK", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n\nCounterfactual invariance is an important but less studied problem in the machine learning community.\n\nThe development of the approach is solid and the theoretical analysis is rigorous.\n\n### Weaknesses\n\nBoth the main Theorem 3.2 and the learning approach eq1 are quite straightforward. See Requests for details.\n\nConsideration of real-world applicability is limited.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper reduces counterfactual invariance to conditional independence under unconfoundedness (conditioning on adjustment variables) and, importantly, a set of conditional variables W whose exogenous information is determined, given themselves and the treatment variable; the latter is achieved by assuming that each of W has a structural function that is injective in its exogenous noise. Based on the previous result, the paper proposes a learning approach (called CIP), adding a regularisation term that trades accuracy for the aforementioned conditional independence. Experiments on (semi)-synthetic datasets show the effectiveness of the approach.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "### Critical\n\nThe relationships of the current work to (Fawkes & Evans, 2023) should be clearer. \n\n1. Besides D-CI, what are a.s.-CI and F-CI (Fawkes & Evans, 2023) is unclear. Your def of D-CI also includes an “a.s.” which is confusing, and I assume it is not the same as a.s.-CI. I also assume F-CI is not your D-CI applied to $\\hat{Y}$ (if so, the eq of F-CI and D-CI is trivial and needs not to be proved).\n2. The statement “On the contrary, the weaker notion of D-CI in Definition 2.2…” is confusing and incorrect. First, I believe you mean D-CI is weaker than a.s.-CI, but the “the weaker notion of D-CI in Definition 2.2” at first reads to me “our def 2.2 of D-CI is weaker than the D-CI defined in  (Fawkes & Evans, 2023)”. Second, whatever a.s.-CI means, I guess it shouldn’t be stronger than D-CI? Finally, and most importantly, I believe you can get Th 3.2 not because D-CI is stronger or weaker than a.s.-CI, but because you introduce the injectivity.\n3. I think it is unfair to say (Fawkes & Evans, 2023) “*only* [prove] for special cases where the counterfactual distribution is identifiable”. Although I am not sure what is exactly assumed in (Fawkes & Evans, 2023), in which you could add an explanation, your injectivity assumption in essence says exogenous uncertainty, which is the core challenge of counterfactual inference, can be removed! This is in the similar spirit of assuming counterfactual identifiability and is not weaker.\n\n### Good to have\n\nThe def of SCM allows a set of structural functions rather than a specific one; this deviates from the standard and is worth a caveat for inexperienced readers.\n\nDef 3.1 and Th A.2 together are just the well-known backdoor criteria and this should be mentioned.\n\nI am not sure why we need VCF in addition to the conditional indep itself to measure CI? In the Middle of Fig 2, you show they are strongly correlated, which is as expected.\n\nIn Fig 5, why the VCF of PSCF is not presented?\n\nThe proof of Th 3.2 is quite pretentious. Lemma A.5-7 are just different ways of saying “exogenous information of W is determined, given W themselves the treatment variable”. I guess we can find a short proof for Lemma A.7 directly.\n\nI am not sure why you seem to stress the CIP eq1, by saying it is “our main contribution”, more than Th 3.2. For me, getting to eq1 from Th 3.2 is trivial, and eq1 is a trade-off (if we can ensure CI and at the same time prediction is great, I’d say this is more interesting than Th 3.2 alone, but surely this should be future work).\n\nOf course, discussions of a couple of real-world examples demonstrating how the assumptions could be satisfied would be very interesting, but I guess this would be challenging.\n\n### Minor\n\nIn “Even for granular W, there may be multiple units…”, I think “granular” should rather be “fine-grained”?\n\nLemma A.6, (y) → (t) in the last eq.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "gy53hXpRvE", "reviewer": "Reviewer_iEPE", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nThe paper is well written and easy to follow. The methodology is motivated by important machine learning applications such as algorithmic fairness and robustness in image and text classifications.\n\nWeaknesses:\n1. The motivation for using kernel mean embeddings is not well discussed. What are the major advantages of the method over other nonparametric conditional independence measures? Also, how sensitive is the performance of the proposed method to the choice of kernel?\n2. The optimization problem (1) does not seem very easy to solve. In particular, when an MLP is used as the learner, can the problem be solved efficiently? How was the optimization carried out in the experiments?\n3. Instead of the penalized form of (1), one can consider a constrained form of the problem: minimize $\\mathcal{L}(\\hat{\\mathbf{Y}})$ subject to $\\mathrm{HSCIC}(\\hat{\\mathbf{Y}},\\mathbf{A}\\cup\\mathbf{W}\\mid\\mathbf{S})\\le\\gamma$. The constrained form could be more appropriate for certain applications. For instance, in the study of algorithmic fairness, it would be desirable to treat counterfactual invariance as a requirement subject to a tolerance level rather than a property to trade-off.\n4. Can the optimality of the predictor be studied in some way? Since prediction and counterfactual invariance are two fundamentally different goals to trade-off, it seems that the usual notion of optimality in terms of MSE or causality no longer applies. Is it possible to develop a concept of optimal predictors, say, optimal prediction among the class of nearly counterfactually invariant predictors?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper formalizes a notion of counterfactual invariance and establishes a sufficient condition for counterfactually invariant predictors via conditional independence. The paper then proposes a nonparametric framework to learn such predictors by using a kernel-based conditional independence measure as a regularization term, thereby striking a trade-off between predictive performance and counterfactual invariance.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The paper can be strengthened by adding more discussions on the form of the learning framework, numerical optimization procedures, and optimality issues.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "YpTr2BYNcT", "reviewer": "Reviewer_8wSh", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The paper is clearly written.\n- The authors provide a detailed introduction to preliminaries and related works.\n- The proposed method of counterfactually invariant prediction is sensible.\n- The result appears to be sound, although I am not familiar with the literature to carefully go through the proof.\n\nWeaknesses:\n- The use of HSIC may limit the efficiency and scalability of the method.\n\nI am not sufficiently familiar with the literature and existing works to list other major weaknesses, and am open to hearing thoughts from other reviewers.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper develops a graphical criteria such that the predictor is counterfactually invariant in terms of conditional independence in observational distribution. The authors then propose an estimation method based on kernel-based conditional dependence measure, and provide experiment results on simulated and real-world data.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- I would suggest moving the contributions part or at least provide a summary in the introduction. Currently, it may be unclear what the specific goal is after reading the introduction.\n- What is the computational complexity and run time of the method?\n- When going through the proofs, most of the theoretical results are based on existing results. Can the paper clarify which part of the theoretical results are new?", "broader_impact_concerns": "I am not aware of any broader impact concern.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: I build my judgement on what I read and discussed with reviewers. In particular I agree that the manuscripts described a useful contribution  about the research subject of counterfactual invariance. The model agnostic which is developed and described together with a 
computationally efficient learning framework suggest the paper to be of interest to the research community. I also agree that the application to fairness and robustness in image and text classifications give an additional motivation for acceptance. Furthermore, what described is sound. The works is theoretically solid even if maybe of a highly specialized are.
Decision: Accept (poster)