Natural Counterfactuals With Necessary Backtracking
Authors: Guang-Yuan Hao
Jiji Zhang
Biwei Huang
Hao Wang
Kun Zhang
Venue: NeurIPS 2024 poster (2024)
Keywords: causal model
counterfactual reasoning
counterfactual generation
normalizing flows
Primary area: causal_inference
TL;DR: We propose a framework of natural counterfactuals and a method for generating counterfactuals that are more feasible with respect to the actual data distribution.

Abstract: Counterfactual reasoning is pivotal in human cognition and especially important for providing explanations and making decisions. While Judea Pearl's influential approach is theoretically elegant, its generation of a counterfactual scenario often requires too much deviation from the observed scenarios to be feasible, as we show using simple examples. To mitigate this difficulty, we propose a framework of natural counterfactuals and a method for generating counterfactuals that are more feasible with respect to the actual data distribution. Our methodology incorporates a certain amount of backtracking when needed, allowing changes in causally preceding variables to minimize deviations from realistic scenarios. Specifically, we introduce a novel optimization framework that permits but also controls the extent of backtracking with a "naturalness'' criterion. Empirical experiments demonstrate the effectiveness of our method. The code is available at https://github.com/GuangyuanHao/natural_counterfactuals.

Natural Counterfactuals With Necessary Backtracking
Guang- Guang-Yuan Hao Hao Jiji Zhang',, Biwei Huang", Hao Wang', Kun Zhang
Chinese University of Hong Kong, PUniversity of California San Diego. RRtteers University CCanegie Mellon University. 'Mohamed bin Zayed University of Artificial Intelligence guangyuanhaoBottlokk com, jijjihhan@cchk, edu hk bih007@ucsd. edu, hw4880cs ww4880cs.rutgers rutgers. edu, kunz1 10mu0mmm.... edu
Abstract
Counterfactual reasoning is pivotal in human cogmition and especially important for providing explanations and making decisions. While Judea Pearl's 's influential approach is theoretically elegant, its generation of a counterfactual scenario often requires too much deviation from the observed scenarios be feasible, as we show using simple examples. To mitigate this difficulty, we propose a framework of natural counterfactuals and a method for generating counterfactuals that are more feasible with respect to the actual data distribution. Our methodology incorporates a certain amount of backtracking when needed, allowing changes causally preceding variables to minimize deviations from realistic scenarios. Specifically, we introduce a novel optimization framework that permits but also controls the extent of backtracking with "naturalness" criterion. Empirical experiments demonstrate the effectiveness of our method. The code is available at https: //github. com/ Cuungyaaaiialuuuueeoooorrrrruuu
Introduction
Counterfactual reasoning, which aims to answer what feature of the world would have been some other features had been different, is often used in human cognition, to perform self-reflection, provide explanations, and inform decisions [29, 6]. For Al systems to achieve human-like abilities of reflection and decision making, incorporating counterfactual reasoning is crucial. Judea Pearl's s structural approach to counterfactual modeling and reasoning [26] has been especially influential in recent decades. Within this framework, counterfactuals are conceptualized as being generated by surgical interventions on the variables to changed, while leaving its causally upstream variables intact and all downstream causal mechanisms invariant. These counterfactuals are thoroughly non- backtracking in that the desired change is supposed to happen without tracing back to changes in causally preceding variables. Reasoning about these counterfactuals can yield valuable insights into the consequences of hypothetical actions. Consider scenario: sudden brake high-speed bus caused Tom to fall and injure Jerry. Non- non-backtracking counterfactuals would tell that if Tom had stood still (despite the sudden braking), then Jerry would not have been injured. Pearl's approach supplies principled machinery to reason about conditionals of this sort, which usually useful for explanation, planning, and responsibility allocation.
However, such surgical interventions are sometimes so removed from what is can be observed that is difficult or even impossible to learn from data the consequences of such interventions. In the previous example, preventing Tom's fall in a sudden braking scenario requires defying mechanisms that are difficult or even physically impossible to disrupt. As a result, there are likely to be no data points in the reservoir of observed scenarios that are consistent with person standing still during
"The first two authors contributed equally to this work.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

sudden braking. If so, it can be very challenging to learn to generate such a counterfactual from the available data, as we will demonstrate in our experiments.
In this paper, we introduce a notion of "natural counterfactuals" to address the above issue with nonbbacktracking counterfactuals. Our notion will allow certain amount of backtracking, to keep the counterfactual scenario "natural" with respect to the available observations. For example, rather than the unrealistic scenario where Tom does not fall at a sudden bus stop, a more natural counterfactual scenario to realize the change to not-falling would involve changing at the same time some causally preceding events, such as changing the sudden braking to gradually slowing down. On the other hand, our notion also constrains the extent of backtracking; in addition to naturalness criterion, we formulate an optimization scheme to encourage minimizing backtracking while meeting the naturalness criterion.
As we will show empirically, this new notion of counterfactual is especially useful from a machine learning perspective. When interventions lead to unrealistic scenarios relative to the training data, predicting counterfactual outcomes in such scenarios can be highly uncertain and inaccurate [13]. This issue becomes particularly pronounced when non-parametric models are employed, as they often struggle to generalize to unseen, out-of-distribution data [31]. The risk of relying on such coun- terfactuals is thus substantial, especially in high-stake applications like healthcare and autonomous driving. In contrast, our approach amounts searching for feasible changes that keep generated counterfactuals within their original distribution, employing backtracking when needed. This strategy effectively reduces the risk of inaccurate predictions and ensures more reliable results.
In short, our approach aims to achieve the goal of ensuring that counterfactual scenarios remain sufficiently realistic with respect the actual data distribution by permitting minimal yet necessary backtracking, It is designed with two major elements. First, we need criteria to determine the feasibil- ity of interventions, ensuring they are realistic with respect the actual data distribution. Second, we appeal to backtracking when and only when it is necessary to avoid infeasible interventions, and need to develop an optimization framework realize this strategy. To be clear, our aim is not to propose uniquely correct semantics for counterfactuals or to show that our notion of counterfactuals superior to others for all purposes. Rather, our goal is to develop notion of counterfactual that is theoretically well-motine well-motivated and practically useful in some contexts and for certain purposes. There may in the end be a general semantics to unify fruitfully our notion and other notions of counterfactual in the literature, but we will not attempt that in this paper. Our working assumption is that there can be different but equally coherent notions of counterfactuals, and we aim to show that the notion developed in this paper is particularly useful for generating counterfactual instances that are feasible with respect to data.
The key contributions of this paper include:
Developing a more flexible and realistic notion of natural counterfactuals, addressing the limitations of non-backtracking reasoning while keeping its merits as far possible.
Introducing an innovative and feasible optimization framework to generate natural counter- factuals.
Detailing a machine learning approach to produce counterfactuals within this framework, with empirical results from simulated and real data showcasing the superiority of our method compared to non- bonbbacktracking counterfactuals.
2 Related Work
Non-backtracking Counterfactual Generation. As will become clear, our theory is presented in the form of counterfactual sampling or generation. [28, 18, 8, 30] use the deep generative models to learn a causal model from data given a causal graph, and then use the model to generate non-bac non-backtracking counterfactuals. Our experiments will examine some of these models and demonstrate their difficulties in dealing with interventions that unrealistic relative to training data, due to the fact that the causal model learned from data is not reliable in handling inputs that are -itto--distribttion.
Moreover, one may argue that the envisaged counterfactual may be unnatural be relevant for practical purposes. For example, from a legal perspective. Tom's causing Jerry's 's injury could be given a "necessity defense," acknowledging that the sudden braking left him with no alternatives [7]. Hence, for the purpose of allocating responsibility,, reasoning about the counterfactual situation of Tom standing still despite the sudden braking is perhaps irrelevant even misleading.
2

---

Backtracking Counterfactuals. Backtracking in counterfactual reasoning has drawn plenty of attention in philosophy [14], psychology [9], and cognitive science [12]. [14] proposes theory that is in a spirit similar to ours, in which backtracking is allowed but limited by some requirement of matching as much causal upstream as possible. [12] shows that people use both backtracking and non-backtracking counterfactuals in practice and tend to use backtracking counterfactuals when explicitly required to explain causes for the supposed change in a counterfactual. [35] is the first work, as far as we know, to formally introduce backtracking counterfactuals in the framework of structural causal models. The main differences between this work and ours are that [35] requires "full" backtracking given causal model, all the way back to exogenous noise terms, and measures closeness in terms of these noise terms, whereas we limit backtracking to what is needed to render the counterfactual "natural", and the measure of closeness in our framework can be defined directly on endogenous, observed variables, which is arguably desirable because changes to the unobserved variables are by definition outside of our control and not actionable. More detailed comparisons can be found in Sec. F of the Appendix.
Algorithmic Recourse and Counterfactual Explanations. Algorithmic recourse (AR) [16] and counterfactual explanations (CE) [36, 10, 23, 3, 24, 34, 32, 37] represent two leading strategies within explainable AI that heavily exploit counterfactual reasoning or reasoning about intervention effects. Various studies [16, 1, 27, 15, 37] explore concepts of feasibility akin to that of "naturalness" in this work, though apply them in quite different contexts. The objectives of CE and AR are pinpoint minimal alterations to an input (in CE) or minimal interventions (in AR) that would either induce a desirable output from predictive model (in CE) or lead to desirable outcome (in AR). In contrast, our work here focuses primarily on generating counterfactuals. While this paper does not directly address CE or AR, our notion of natural counterfactuals is likely to be very relevant to these tasks.
3 Preliminaries
In this section, we begin by outlining various basic concepts in causal inference, followed by an introduction to non-backtracking counterfactuals.
Structural Causal Models (SCM). We assume there is underlying recursive SCM [26] of the following sort to represent the data generating process. An SCM M == (.,.))(UU) consists of exogenous (noise) variables U {UI, ..., Un}. endogenous (observed) variables V = (Vi...., ) functions f = {fi,... .... fxx ), and a joint distribution p(U) of noise variables, which are assumed be jointly independent. Each function, Sa E f, specifies how endogenous variable V, is determined by its parents PA, C V:
V := (PA., U.), i i= N
Such an SCM entails a (causal) Bayesian network over the observed variables. consisting of the directed acyclic graph over V in which there is an arrow from each member of PA; to V,, and the joint distribution of V induced by f and P(U). In our setting, we assume this causal graph is known and samples from this joint distribution are available, but f and P(U) are not given, though some assumptions on f and p(U) will be needed for identifiability.
Local Mechanisms. Functions in f are usually regarded as representing local mechanisms. In this paper, however, will use the term "local mechanism" to refer to the conditional distribution of an endogenous variable given its parent variables, i.c., p(V, (PA.) for i = I...,N, which can be estimated from the available information. Note that a local mechanism in this sense implicitly encodes the properties of the corresponding noise variable; given a fixed value of PA, the noise U determines the probability distribution of V, [26]. Hence, the term "local mechanism" will also be used sometimes to refer to the distribution of the noise variable p(U,).
set of endogenous variables A C V is represented by replacing the functions for members of A with constant functions X = x², where X E A and x. is the target value of X, and leaving the functions for other variables intact [26].²
Intervention. Given an SCM, intervention
Non- Non-Backtracking Counterfactuals. Let A, B, and E be sets of endogenous variables. A general counterfactual question takes the following form: given evidence E = e, what would the value of B have been if A had taken the value setting at? In this paper, we focus on a special case of this
3 Following standard notation, use a superscript to signal counterfactual values.
3

---

question in which E = V. i.., the evidence is a complete data point covering all observed variables. That is, given an actual data point, we consider what the data point would have been if variable had taken a different value than its actual value. The Pearlian, non-backtracking reading of this question takes the counterfactual supposition of A a to be realized by an intervention on A [26]. This means that in the envisaged counterfactual scenario, all variables in the causal upstream of A keep their actual values while A takes a different value. As mentioned. a potential problem is that such a scenario is outside of the support of available data, and so it is often unreliable to make inferences about the downstream variables in the scenario based on the available data.
4 A Framework for Natural Counterfactuals
Do(.) and ('aange() Operators. Using Pearl's influential do-operator, the non- -ancttttracking mode appeals to do A a²), intervention set the value A a", to generate counterfactual instance. However, in our framework, the counterfactual supposition is not necessarily realized by an intervention on A, while keeping all its causal upstream intact. Instead, when do a²) results in a counterfactual setting that violates a naturalness criterion, some backtracking will be invoked. To differentiate from the intervention do(A a²) we will often use change(-) and write change (A a') ) to denote a desired modification in A. Different semantics for counterfactuals correspond to different interpretations of the change operator. In this paper, we explore an interpretation that connects the change operator to the do operator in relatively straightforward manner.
The basic idea is that change [ = a²) will correspond to do(C -c*) for some set C that includes A and possibly some of A's causal ancestors. When C = {A}, this is equivalent to a non-backtracking interpretation. In general, however, some variables in A's 's causal upstream need to change together with in order to keep the counterfactual scenario within the relevant support. central component of our approach is design way determine C and c", given a request of change A = )²). We will call the resulting do( C = c²) the cast-backtracking feasible (LBF) intervention for change(A A a"). Once the LBF intervention is determined, inferences can be made in the same fashion as in Pearl's approach [26].
To determine the LBF intervention for change (A = a²), we formulate it as an optimization problem to search for a minimal change of A's causal ancestors that, together with changing A to a", satisfy a "naturalness" criterion. Let AN(A) (A) denote the set of A ancestors in the given causal graph together with A itself. We define the optimization framework, Feasible Intervention Optimization (FIO), as follows:
minimize an(A)*
D(an( ()) an))n(A))
s.t.
==a² 9n (a((n()))) > e.
where an(A) and an(A)* represent the actual value setting and the counterfactual value setting of (N(A) respectively (note that A C AN((A). 9-)(-) measures the naturalness of the counterfactual value setting of AN (A) and e is a small constant. So the optimization has a naturalness criterion as constraint. On the other hand, D(-) is a distance metric designed to encourage the counterfactual value setting to invoke the least amount of backtracking. Below we develop these two components in Sec. 4.1 and Sec. 4.2, respectively. Once we obtain the counterfactual value an (n)) by FIO, the variable set for the LBF intervention includes A and other variables corresponding to the difference between an and an(A A), i.e., A is always included in C even when an ()) an(
4.1 Naturalness Constraints
As indicated previously, the intended purpose of the naturalness constraint is to confine the counter- factual instance sufficiently within the support of the data distribution. Roughly and intuitively, the more frequently a value occurs, the more it is considered to be "natural". Moreover, we would like to use a measure of naturalness that takes into account local mechanisms according to the given causal graph rather than just the joint distribution. Therefore, we propose to assess naturalness by examining the distribution characteristics, such as density, of each variable's 's value V, v given the variable's
local mechanism p p(V, ).az). where V, (A) and paž denotes its parental value setting.* 4
"The notion of "naturalness"" can have various interpretations. In our context, it is defined by the observed data distribution, as we assume we can only access observed data and intend the counterfactuals be empirically

---

4.1.1 Local Naturalness Criteria
We start by proposing some measures of the naturalness of one variable's value, vz. within the counterfactual data point (n) in this section, followed by defining measure of the overall naturalness of an A)" in the next.
Informally, a value satisfies the criterion of local e-natural generation if it is a natural outcome of its local mechanism. The proposed measures naturalness will depend on the specific value V, vj. alongside its parent value PAj pa noise value Uj = =uj and the corresponding local mechanism, expressed by p(V, PPA, pa.) p() The cumulative distribution function (CDF) for noise variable Uj at U uj is F(u,) _ pUU and for the conditional distribution PA, = paf) at V vj is F(V, JIPo3) - p(V, = vilpaz)dV
We propose the following potential criteria based normapy-n density, CDF of exogenous variables, and CDF of conditional distributions, respectively. The entropy- normallzedd naturalness measure evaluates the naturalness of V in relation to its local mechanism p(V IPA, CDseb measures, namely the latter two criteria, consider data points in the tails to be less natural. Each of these criteria has its intuitive appeal, and their relative merits will be discussed subsequently. Below, we use gi (Vj) to represent a (local) naturalness measure of vj. We consider the following three possible measures:
)aa). The
(1) Entropy-Normalizee Measure:g((y}) = [ex]lpa3)=#(:slpp) where H(Vslpa7) =
E log PVV, |pa))];
(2) Exogenous CDF Measure: gi(v)) = min(F((u)), F(ū)))
(3) Conditional CDF Measure: 9v(33) min(F(v3lpa)). F(vilpa5));
where the function min(-) returns the minimum of the given values. When gi (v)) > c, we say the
Vÿ given its causal parents' values satisfies the criterion of local e-natural generation.
Some comments on these choices are order:
Choice (1): Entropy-Normalized Measure. Specifically, Choice (1), p(vj (Vslme)) can be rewritten as plog pa])+B/ log p(V,) p(V,lpaa)) /Paa)), where log p(v Ipaz) can be seen as the measure of surprise of vÿ given paj and E[- log (V, [lpa))] can considered as the expectation of surprise of the local mechanism p(V, Ipa;)) [2]. Hence, the measure quantifies the relative naturalness (i.e., negative surprise) of V; j. Implementing this measure is usually straightforward when employing parametric SCM where the conditional distributions can be explicitly represented.
Choice (2): Exogenous CDF Measure. If using a parametric SCM, we might directly measure differences on exogenous variables. However, in a non- non-parametric SCM, exogenous variables are in general not identifiable, and different noise variables may have different distributions. Still, we may consider using the CDF of exogenous variables align the naturalness of different distributions, based on a common assumption for non-parametric SCMs in the machine learning system. The assumption is that the support of the local mechanism p( (V, ,PA, pa;) does not contain disjoint sets, the function fj in the SCM is monotonically increasing with respect to the noise variable Uj. which is assumed to follow a standard Gaussian distribution [20]. Data points from the tails of standard Gaussian can be thought of as improbable events. Hence, V, vī satisfies local e-natural generation when its exogenous CDF F(u;) falls within the range (e, 1 e). In practice, for single variable, U, is a oneddimensional variable, and it is casier to enforce the measure than Choice (I), which involves conditional distributions.
Choice (3): Conditional CDF Measure. The measure treats particular value in the tails of local mechanism p( V5lpo)) as unnatural. Hence, V, Vī meets local n-natural generation when F(V, = (aa) falls within the range (c, <) instead of tails. This measure can be used in parametric models where the conditional distribution can be explicitly represented. It can also be easily used in pan-aarmettric models and the measure is equivalent to Choice (2) when those models
supported. While this is obviously not the only plausible interpretation of naturalness, it is useful one for our purpose.
5

---

satisfy the assumption mentioned in Choice (2), since the CDF FVV, (paz) has one-to-one mapping with the CDF F F(U,), i.e., F(v3 pa.) F(u;), when vj = f (naj, nz).
4.1.2 Global Naturalness Criteria
Given a local naturalness measure Jt, we simply define a global naturalness measure for an(A)* as
In (an)
min v)Ean(A)
(ar()))).
That is, In (a)())) returns the smallest local naturalness value among members (A)).. Finally, we can define criterion of e-natural generation to assess whether the counterfactual value and A)) is sufficiently natural.
Definition (e-Natural Generation). Given an SCM containing a set A, let (N(A) contain all ancestors of A and A itseff. A value setting AN(A) an(A)* salisfies e-nalural generation, if and only if, In (n)()))))) and e a small constant.
Obviously, larger value of implies higher standard for the naturalness of the counterfactual value setting an(A) A)" To consider only feasible interventions, we require an( to meet e-natural generation, which a constraint used in FIO.
4.2 Distance Measure to Limit Backtracking
We now turn to the distance measure in Eqn. of the FIO framework. We considered two distinct distance measures in this work. The first prioritizes minimizing changes in the observed causal ancestors of the target variable of the desired change. The second focuses reducing alterations in local mechanisms, regarding them as inherent costs of an intervention. Due to space limitations, we will introduce here only the simpler measure in terms of minimal changes in the observable causal ancestors. A discussion of the other measure can be found Sec. G.
For our purpose, the L norm is good choice, as encourages sparse changes and thus sparse backtracking:
D(an (A), an (m) [an) an(A)ll
where an (A) and an())* represent the actual value and counterfactual value of A's ancestors (N(A) respectively, where A E AA). Because endogenous variables may vary in scale, e.g., normal distribution with a range of (-o0, oo) versus a uniform distribution over the interval [0, 1], standardize each endogenous variable before computing the distance. This normalization ensures a consistent and fair evaluation changes.
Implicitly, this distance metric favors changes in variables that are proximal to A, since altering more remote variable typically results in changes to more downstream variables. the extreme case, when the value an(A) corresponding to do(A a²), i.c., the corresponding to the non-backtracking counterfactual, already meets the e-natural generation criterion, the distance metric D(an( A), an (A)) will achieve the minimal value la a*I. In such case, backtracking is needed and the non- backktaakta counterfactual will be generated. However, If do(A = a²) does not meet the e-natural generation criterion, it becomes necessary backtrack.
4.3 Identifiability of Natural Counterfactuals
As said, we assume we do not have prior knowledge of the functions of the SCM and so noise variables are in general identifiable from the observed variables. However, if assume the SCM satisfies the conditions of the following theorem, then the counterfactual instance resulting from an LBF intervention identifiable.
Theorem 4.1 (Identifiable Natural Counterfactuals), Given the causal graph and the joint distribution over V. suppose V, satisfies the following structural causal model: V; := fa(PA,, U.) for any V, V, assume every Jss though unknown, smooth and strictly monotonic w.r.l. U, for fixed values of PA, Then, given an actual data point V v, with an LBF intervention doi c*) (satisfying the criterion of e-natural generation), the counterfactual instance V v identifiable:
V = = v = c²), =v..

---

This theorem confirms the identifiability of our natural counterfactuals from the causal graph and the joint distribution over the observed variables." Specifically, since do(C c*) satisfies the criterion of e-natural generation, it guarantees that the resulting counterfactual instance falls within the support of the observed joint distribution. Then, building on Theorem from [20], we can demonstrate that
using the actual data distribution allows for the inference of natural counterfactuals without knowing the functions or the noise distributions of the SCM.
5 A Practical Method for Generating Natural Counterfactuals
In this section, we provide practical method for solving (approximately) the FIO problem described in the last section. We assume that we are given data sampled from the joint distribution of the endogenous variables and a causal graph, and that the underlying SCM satisfies the assumptions in Theorem 4.1. We learn a generative model for the endogenous variables from data, serving as an estimated SCM to generate natural counterfactuals: V := fPA,, U;) for i = 1, ...,, N, where fi is assumed to be reversible given PA,, Note that, unlike the functions in the true SCM, these learned functions in general do not generalize well outoof-distribution data, as demonstrated in the experiments, which, recall, is a main motivation for employing natural counterfactuals instead. For simplicity, we assume the noise distribution is standard Gaussian, though the identifiability of natural counterfactuals only requires that noise variables are continuous and do not depend on the specific form of the noise distribution. The specific FIO problem we target plugs the distance measure (Eqn. 4) and the naturalness measure from Choice (3) in Sec. 4.1.1 (or Choice (2), which is equivalent to Choice (3) given the assumptions) into Eqn. 2:
minimize |la - -a(AA)ll (n
s.l.
= a", F(V, = výlpaț) < <1 AN(A).
(5)
Again, in theory, there may be no solution for an(A)* the naturalness criterion demanding. In the extreme case, for example, even when a" if is set so high that the actual instance does not satisfy the condition of e-natural generation, then no solution exists.
We propose to solve this optimization problem using the following approximate method. Since the estimated functions f₃ are assumed to be reversible, we can reformulate the problem of searching for optimal values of the endogenous variables as one of searching for optimal values of the exogenous noise variables. A feasible approach is to use the Lagrangian method [5] to minimize the following objective loss:
C(u ((UAN(A))
Σ UNENANIA)
vzl + W. 18s(pa3, u))
Σ UJ ANEA)
[max(e- mnax(=F((u)).0) 0) max(e + F(uţ) 1,0
s.t. UAA (1=xxaaa")
(6)
where the optimization parameters are the counterfactual values of the noise variables corresponding to A's ancestors, u A and the function max(.) returns the maximum between two values. The first term is the distance measure in the FIO problem, while the second term implements the constraint of e-natural generation. The hyperparameter w, serves to modulate the penalty imposed on non- natural values. Notice that, in order ensure the hard constraint A a', A's noise value u not optimized explicitly, since the value Pa, is fully determined by a* and other noise values. Hence, only noise values other than Uy are optimized. Further details are provided Sec.
For simplicity, we have focused on the case of "full evidence" in presenting our framework and method. But it is straightforward to extend the approach to address cases of "partial evidence" i.c., E + V. In such cases, we can simply sample from p(V)E), treating it as full evidence, and obtain a natural counterfactual from the resulting sample.
$ Given substantial naturalness constraint, there may be one or more solutions for C. or no solution at all. Which of these is the case can always be identified. The above theorem on identifiability of natural counterfactuals is concerned only with the cases in which natural counterfactual exists, meaning that there is at least solution for C. The proof of Theorem 4.1 is provided in Sec. B.
7

---

Table 1: MAE Results on Toy / Toy 4 (Lower MAE is better). To room, also write "do" for "change" for natural counterfactuals.
Dataset
Toy
Toy
Toy 3
Toy
do or change
do(n1)
do(r2)
do(n))
do(()))
do(n2)
do(ns))
do(n:))
do(r2)
Outcome
m2
mg
123
112
72
7h4
74
n4
122
133
13
Non-backtracking
0.477 0.382
0.297
0.315
0.4488
0.472
0.436 0.488
0.230
0.179
0.166
0.446
0.429
Ours
0.434 0.354
0.114
0.303
0.443
0,451
0.423
0.127 0.136
0.137
0.158
0.443
0.327
Evitence Ronbadktackin Ours
2-
my Prediction
An = 2.31 Mr = 0.03
y=x
Nonbacktracking
Curs
mi's Groundtruth
(a) Outcome single sample
(b) Croundtruth-Prediction Scatter Plot
Figure 1: The Visualization Results on Toy / (View the enlarged figure in Fig, 3 in the Appendix).
Experiments
In this section, evaluate the effectiveness of method through empirical experiments four synthetic datasets and two real-world datasets.
We propose using the deviations between generated and ground-truth outcomes as a measure of performance. We expect our natural counterfactuals to significantly reduce errors compared to non-backtracking counterfactuals. This advantage be attributed to the effectiveness of our method in performing necessary backtracking that identifies feasible interventions, keeping counterfactual values within the data distribution, so that the learned functions are applicable. On the other hand, non-backtracking counterfactuals often produce out-of-distribution values [13, 31], posing challenges for generalization using the leameed functions.
6.1 Simulation Experiments
We start with four simulation datasets, which we use designed SCMs to generate. Please refer to the Appendix for more details about these datasets. Let's first look at Toy 1, which contains three variables (M1, n2, ns). is the confounder of n2 and nı, and and n2 cause n2.
Experimental Settings. Again, we assume data and a causal graph are known, but not the ground- truth SCM. We employ normalizing flows to learn generative model of variables (mi, R2, n1a) compatible with the causal order. Given the pretrained causal model and data point from the test set as evidence, we set n. or 122 A and randomly sample values from test dataset counterfactual values of the target variable nı or n22. In our natural counterfactuals, we use Eqn. 6 to determine 10a, while in non- non-backtracking counterfactuals, nı
LBF interventions, with 10-4 and or 72 is directly intervened on. We report the Mean Absolute Error (MAE) between our learned counterfactual outcomes and ground-truth outcomes f22 or/and ny with multiple random seeds. Notice there may be no feasible interventions for some changes, as we mentioned Sec. 5, and thus only report outcomes with feasible interventions, which are within the scope of our natural counterfactuals.
Visualization of Counterfactuals on a Single Sample. We assess the counterfactual outcomes for sample (m) 122, n1)) = (-0.59, 0.71, -0.37), given the desired alteration change(ng = 0.19). For natural counterfactuals, it is necessary to backtrack to n, to realize the change. This step ensures
8

---

that the pair (71.my) remains within high-density area of the data distribution. In essence. our intervention targets the composite variable C (M1, On the other hand, for non- backtrakkkc counterfactual, the intervention simply modifies n2 to 0.19 without adjusting 11, making (n) 122) out of data support. Fig. (a), we depict the original data point (yellow), the nanbbccttaakknng counterfactual (purple), and the natural counterfactual (green) for (M1, n2). The ground-truth support for these variables is shown as a blue scatter plot.
(1) Feasible Intervention VS Hard Intervention. intervention on n2 (do(ng- = 0.19)), shifting the evidence (yellow) to the post-intervention point (purple), which lies outside the support of (121, n2). This shows that direct interventions can result in unnatural values. Conversely, our natural counterfactual (green) remains within the support of (2)) due to backtracking and the LBF intervention on (11, n2).
Non-backtracking counterfactuals apply a hard
(2) Outcome Error. We calculate the absolute error between nı 's model prediction and ground-truth value using either the green or purple point as input for the model ( The error for the green point significantly lower 0.03, compared to 2.31 for the purple point. This lower error with the green point is because it stays within the data distribution after an LBF intervention, allowing for better model generalization than the out-of-distribution purple point.
Counterfactuals on Whole Test Set. In Fig. 1 (b), we illustrate the superior performance of our counterfactual method on the test set, notably outperforming non-backtracking counterfactuals. This is evident as many outcomes from non-backe bon-bbcktracking counterfactuals for n3 significantly diverge from the y line, showing mismatch between predicted and ground-truth values. In contrast, our method's outcomes largely align with this line, barring a few exceptions possibly due to learned model's imperfections. This alignment is attributed our method's consistent and feasible interventions, enhancing prediction accuracy, while non- non-backtracking counterfactuals often lead to infeasible results. Table supports these findings, demonstrating that our approach exhibits an MAE reduction of 61.6% when applied to 122 compared with the non-backtracking method. Furthermore, our method excels even when intervening in the case of n2, rool cause, by excluding points that do not meet the e-natural generation criteria, further demonstrating its effectiveness.
Additional Causal Graph Structures. Our method also shows superior performance three other simulated datasets with varied causal graph structures (Toy 2 to Toy 4), as demonstrated in Table 1.
6.2 MorphoMNIST
As depicted in Fig. 4 (a) of Appendix, (digit stroke thickness) causes both (stroke intensity) and x (images), with i being the direct cause of x in MorphoMNIST. In this experiment, mir- roring those in Section 6.1, we incorporate two key changes. First, we utilize two advanced deep learning models, V-SCM [25] and H-SCM [28]. Although both models are referred to as "SCM," they only learn the conditional distri- butions of endogenous variables and, in theory, do not capture the functional relationships for diitt--disttiuution inputs. Second, due to the
Table 2: Ablation Study on e (Lower MAE is bet- ter)
do(t)
do(i)
Model
CFs
NB
0.336
4.532
0.283
6.556
10 4
0.314
4.506
0.171
4.424
V-SCM
10-a
Ours
0.298
4.486
0.161 4.121
10 2
0.139
4.367 0.145
3.959
NB
0.280 2.562 0.200 3.345
10
0.260
2.495 0.105 2.21T
H-SCM
10-a
Ours
0.245
2.442 0.096 2.091
absence of ground-truth SCM for assessing out- come error, we adopt the counterfactual effec- tiveness metric from [28, 22]. This involves
10-2
0.093 2.338 0.083 2.063
training a predictor on the dataset to estimate parent values ( from a counterfactual image x
generated by model p(z(t, i) with the input (t, i), and then computing the absolute error |t - i| or | i
Ablation Study on Naturalness Threshold Table 2 demonstrates that our error decreases with increasing e, regardless of whether V-SCM or H-SCM is used. This trend suggests that a larger e sets stricter standard for naturalness in counterfactuals, enhancing the feasibility of interventions and consequently lowering prediction errors. This improvement due to that deep-le models are more adept at generalizing to high-frequency data [11].

---

6.3 3DIdentBOX
In this study, employ two prac- tical public datasets from 3DIdent- BOX [4], namely Weak-3DIdent and Strong-3DIdent, where each image contains teapot. Both datasets share the same causal graph, as depicted in Fig. 7 (b) of the Appendix, which includes an image variable x and its seven parent variables, with a single variable b and three pairs of variables: (h,d), (v, B), and (a, ?). where one is the direct cause of the other in each
Table 3: MAE Results on Weak-3DIdent and Strong-3DIdent (abbreviated as "Weak" "Strong" for simplicity). Lower MAE is better. For clarity, we use "Non" to denote the -on-backtracking.
Dataset
h
8
0.27 0.077 0.0042 Ours 0.024 0.018 0.034 0.349 0.221 0.036 0.0041
Non 0.025 0.019 0.035
0.364
Weak
Non 0.100 0.083 0.075 0.387 0.495 0.338 0.0048
Stong
Ours 0.058 0.047 0,050 0.298 0.316 0.139 0.0047
pair. The primary distinction between Weak-3DIden and Strong-3DIdent lies in the strength of the causal relationships between each variable pair, with Weak-3Dldent exhibiting weaker connections (Fig. (c)) compared to Strong-3Dident (Fig. (d)). Our approach mirrors the MorphoMNISST experiments, using H-SCM as the pretrained causal model with = 10
Influence of Causal Strength. As Table 3 reveals, our method outperforms non-backtracking on both datasets, with a notably larger margin in Strong-3 This increased superiority is due to higher incidence of infeasible hard interventions non-backtracking counterfactuals within the Strong-3DIdens dataset.
See Appendix for More Details. Please refer the Appendix for information datasets, generated samples on MorphoMNIST and 3DIdentBox, standard deviation of results, settings of model training and FIO, differences between our natural counterfactuals and related works, and more.
7 Conclusion
Given paraamamttiic SCM learned from data and causal graph, non-backtracking counterfactual inference or generation may be highly unreliable because the corresponding non- -on-backtracking intervention result in scenario that is far removed from the data based which the SCM is learned. To address this issue, we have proposed a notion natural counterfactuals, which incorporates naturalness constraint and aims keep the counterfactual supposition within the support of the training data distribution with minimal backtracking. We also developed practical method for the generation or inference of natural counterfactuals, the effectiveness of which was demonstrated by empirical results.
In case it is not already transparent, we have not shown, nor did we intend to argue, that natural counterfactuals are superior to non-backtracking counterfactuals for all purposes. Non-backtracking counterfactuals, when correctly inferred, have clear advantages in revealing causal relations between events and the effects of specific actions. However, as we showed in this paper, in the context of data- driven counterfactual reasoning, inference generation of non- non-backtracking counterfactuals often go astray due to the challenges of -itto---iittibutiin generalization. Although correct information about coon-bakttracknn counterfactuals has considerable action-guiding values, using unreliable information for that purpose is epistemologically and ethically dubious. We intend our framework of natural counterfactuals to mitigate this kind of risk, though it may appear less elegant theoretically.
Our current method is based on the assumption that the learned functions are invertible. One purpose of using the assumption is to ensure the identifiability of natural counterfactuals when only has access to endogenous variables. If the assumption does not hold, identifiability is not guaranteed. For example, suppose Y XU + U2 where Y and X are endogenous variables and Un and Uz are exogenous noises, then the counterfactual outcome will not be identifiable. However, if we also assume a known distribution of exogenous variables, then method can be generalized without assuming invertible functions or independent exogenous variables.
Finally, we hasten to reiterate that we do not claim that the particular distance measures and natural- ness measures used in this paper are the only choices or among the best. It will be interesting to study and compare alternative implementations in future work.
10

---

Acknowledgment
This material based upon work supported by the RGC Hong Kong under NSF Award No. 2229881, AI Institute for Societal Decision Making (AI-SDM), the National Institutes of Health (NIH) under Contract ROIHL 159805, and grants from Salesforce, Apple Inc., Quris AI, Florin Court Capital, and the MBZUAI-WIS grant.
References
[1] André Artelt and Barbara Hammer. Convex density constraints for computing plausible coun- terfactual explanations. In Artificial Neural Networks and Machine Learning-ICANN 2020: 29th International Conference on Artificial Neural Networks, Bratislava, Slovakia, September 15-18, 2020, Proceedings, Part 29, pages 353-365. Springer, 2020.
[2] Robert B Ash. Information theory. Courier Corporation, 2012.
[3] Solon Barocas, Andrew Selbst, and Manish Raghavan. The hidden assumptions behind counterfactual explanations and principal reasons. In FAT, 2020.
[4] Alice Bizeul, Imant Daunhawer, Emanuele Palumbo, Bernhard Schölkopf, Alexander Marx, and Julia E Vogt. 3didentbox: A toolbox for identifiability benchmarking. 2023.
[5] Stephen Boyd and Lieven Vandenberghe. Convex optimization. Cambridge university press, 2004.
[6] Ruth MJ Byrne. Counterfactual thought. Annual review of psychology, 67:135-157, 2016.
[7] Michelle R Conde. Necessity defined: A new role in the criminal defense system. UCLA Rev., 29:409, 1981.
[8] Saloni Dash, Vineeth N Balasubramanian, and Amit Sharma. Evaluating and mitigating bias in image classifiers: A causal perspective using counterfactuals. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 915-924, 2022.
[9] Morteza Dehghani, Rumen Iliev, and Stefan Kaufmann. Causal explanation and fact mutability in counterfactual reasoning. Mind & Language, 27(1):55-85, 2012.
[10] Amit Dhurandhar, Pin-Y Yu Chen, Ronny Luss, Chun-Chen Tu, Paishun Ting, Karthikeyan Shan- mugam, and Payel Das. Explanations based on the missing: Towards contrastive explanations with pertinent negatives. In NeurIPS, 2018.
[11] Logan Engstrom, Andrew Ilyas, Hadi Salman, Shibani Santurkar, and Dimitris Tsipras. Robust- ness (python library), 2019, License: MIT.
[12] Tobias Gerstenberg, Christos Bechlivanidis, and David A Lagnado. Back on track: Backtracking in counterfactual reasoning. In Proceedings of the Annual Meeting of the Cognitive Science Society, volume 35, 2013.
[13] Negar Hassampour and Russell Greiner. Learning disentangled representations for counterfactual regression. In International Conference on Learning Representations, 2019.
[14] Eric Hiddleston. A causal theory of counterfactuals. Noûs, 39(4).632-667, 2005.
[15] Shalmali Joshi, Oluwasanmi Koyejo, Warut Vijitbenjaronk, Been Kim. and Joydeep Ghosh. Towards realistic individual recourse and actionable explanations in black-box decision making systems. arXiv preprint arXiv: /907,0990000011: 2019.
[16] Amir-Hossein Karimi, Bernhard Schölkopf, and Isabel Valera. Algorithmic recourse: from counterfactual explanations interventions. In Proceedings of the 2021 ACM conference fairness, accountability, and transparency, pages 353-362, 2021.
[17] Diederik Kingma and Max Welling, Auto-encoding variational bayes. In ICLR, 2014.

---

[18] Murat Kocaoglu, Christopher Snyder, Alexandros G Dimakis, and Sriram Vishwanath. Causal- gan: Learning causal implicit generative models with adversarial training. In International Conference on Learning Representations.
[19] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2018
[20] Chaochao Lu, Biwei Huang, Ke Wang, José Miguel Hermández-Lobato Kun Zhang, and Bernhard Schölkopf. Sample-efficient reinforcement learning via counterfactual- data augmentation. arXiv preprint arXiv:20/2.0900) 2020.
[21] Lars Maaløe, Marco Fraccaro, Valentin Liévin, and Ole Winther. Biva: A very deep hierarchy of latent variables for generative modeling. Advances in neural information processing systems, 32, 2019.
[22] Miguel Monteiro, Fabio De Sousa Ribeiro, Nick Pawlowski, Daniel C Castro, and Ben Glocker. Measuring axiomatic soundness of counterfactual image models. arXiv preprint arXiv:3303.01274, 2023.
[23] Ramaravind K. Mothilal, Amit Sharma, and Chenhao Tan. Explaining machine learning classifiers through diverse counterfactual explanations. In FAccT, 2020.
[24] Nick Pawlowski, Daniel Coelho de Castro, and Ben Glocker. Deep structural causal models for tractable counterfactual inference. In NeurlPS, 2020.
[25] Nick Pawlowski, Daniel Coelho de Castro, and Ben Glocker. Deep structural causal models for tractable counterfactual inference. Advances in Neural Information Processing Systems, 33:857-888 869,8699998 2020.
[26] Judea Pearl. Causality. Cambridge university press, 2009.
[27] Rafael Poyiadzi, Kacper Sokol, Raul Santos-Rodriguez, Tijl De Bie, and Peter Flach. Face: fea- sible and actionable counterfactual explanations. In Proceedings of the AAAT/ACM Conference on Al, Ethics, and Society, pages 344-350, 2020.
[28] Fabio De Sousa Ribeiro, Tian Xia, Miguel Monteiro, Nick Pawlowski, and Ben Glocker. High fidelity image counterfactuals with probabilistic causal models. 2023.
[29] Neal J Roese. Counterfactual thinking. Psychological bulletin, 121(1): 1997.
[30] Pedro Sanchez and Sotirios Tsaftaris. Diffusion causal models for counterfactual estimation. In First Conference on Causal Learning and Reasoning.
[31] Bernhard Schölkopf, Francesco Locatello, Stefan Bauer, Nan Rosemary Ke, Nal Kalchbrenner, Anirudh Goyal, and Yoshua Bengio. Toward causal representation learning. Proceedings of the IEEE, 10965)x61-634,, 2021.
[32] Lisa Schut, Oscar Key, Rory MeGrath, Luca Costabello, Bogdan Sacaleanu, Medb Corcoran, and Yarin Gal. Generating interpretable counterfactual explanations by implicit minimisation of epistemic and aleatoric uncertainties. In AISTATS, 2021.
[33] Jonathan Vandenburgh. Backtracking through interventions: An exogenous intervention model for counterfactual semantics. Mind & Language, 38(4):981-999 2023.
[34] Sahil Verma, John P. Dickerson, and Keegan Hines. Counterfactual explanations for machine learning: A review. arXiv preprint, arXiv:2010. 10596, 2020.
[35] Julius von Kügelgen, Abdirisak Mohamed, and Sander Beckers, Backtracking counterfactuals. arXiv preprini urXiv:22/1.00477 2022.
[36] Sandra Wachter, Brent Mittelstadt, and Chris Russell. Counterfactual explanations without opening the black box: Automated decisions and the GDPR. Harvard Journal of Law & Technology, 2018
[37] Jingquan Yan and Hao Wang. Self-interpretable time series prediction with counterfactual explanations. In ICML, 2023.
12

---

71
122
13
71
22
(a) Toy
(b) Toy
(c) Toy 2
(d) Toy 2
(e) Toy 3
(1) Toy
(g) Toy
(h) Toy 4
Figure 2: Causal graphs and Scatter Plot Matrices of Toy 1-4. Figure (a) (c) (e) and (g) show causal graphs of Toy 1-4 respectively. Figures (b) (d) (f) and (h) indicate scatter plot matrices of variables in Toy 1-4 respectively.
A Datasets and More Experimental Results
In this section, we first provide detailed dataset settings and additional experimental results. Subse- quently, we present the standard deviation of experimental outcomes Sec.
A.1
Toy Datasets
We design four simulation datasets, Toy 1-4, and use the designed SCMs to generate 10, 000 data points training dataset and another 10, 000 data points test set for each dataset. Fig. 2 shows causal graphs of Toy 1-4 and scatter plot matrices of test datasets each dataset. The ground-truth SCMs of each dataset are listed below.
Toy I.
7 = U1
21 ~ N/ 1),
1
112 11 U2
U2 NO 1),
3
n1 sin 20.25 (0.5ng n1)] 0.2u3,
U3 (O,
where there are three endogenous variables (n1, n2, na) and three noise variables (u1, U2 Us). is the confounder of n2 and ng. n1 and n2 cause n3.
Toy2 2.
U1 Noo, 1),
71 U1
n/2 = sin (0.2T 2.5)] + 0.2ug,
U2 (0, 1),
where there are two endogenous variables (n1, n2) and two noise variables (u1, u2). nı causes n2.
Toy 3.
11 = U1
81 ~ NN 1),
1
n2 72 + 3 U2
U2 NO0, 1),
ng sin (0. (n2 + 2.0)] + 0.2ug
U3 N/ 1),
n2 = sin 0..55 (na- - ni + 2.0]] + 0.2u4,
#4 ~ ~,,
13

---

where there are four endogenous variables (M,, 12,, na) and four noise variables (u1, U2 u3. Ud). n, the confounder of ng and n4. (n2, n3, (4) a chain, i.e., n2 causes ng, followed by nn.
Toy 4.
((0, 1), Ug ~ M(o,1), 1),
M1
1/2 = - + U2
n3 = sin [0.3T (n2 2.0]] + 0.203,
Ug ~ ((
where there are three endogenous variables (n1, n2, na) and three noise variables (u1, u2 uz). (n1, n2, n3) is a chain, i.e., 11 causes n2. followed by ng.
Table 4: MAE Results on Toy to Toy For simplicity, we use do operator in the table to save room, and when natural counterfactuals are referred to, do means change.
Dataset
Toy
Toy 2
Toy 3
Toy 4
do or change
do(m1)
do(n2)
do(n1)
do(n1)
do(n2)
do(ns))
do(n1)
(d((2))
Outcome
Y12
rig
Y63
712
112
714
714
Y14
/12
Y23
nig.
bon-backtracking
0.477 0.382
0.297
0.315
0..488
0,472
0.436 0.488
0.230
0.179
0.166
0.446
0,429
Ours
0.434 0.354
0.114
0.303
0.443
0.451
0.423
0.127
0.136
0.137
0.158
0.443
0.327
In the main paper, we have explained experiments on Toy in detail. As shown in Table 4, our performance on Toy 2-4 shows big margin compared with backtrcktgg counterfactuals since natural counterfactuals consistently make interventions feasible, while part of hard interventions may not be feasible in non-backtracking counterfactuals.
Visualization Results on Toy 1. Fig. 3, larger figures are displayed, which are identical those shown in Fig. 1, with the only difference being their size.
A.2 MorPhoMNIST
The MorphoMNNIS comes from [25], where there are 60000 images as training set and 10,000 images test dataset. Fig. (a) shows the causal graph for generating MorphoMNIST; specifically, stroke thickness t causes the brightness intensity i, and both thickness t and intensity i cause the digit Fig. 4 (b) show some samples from MorphoMNIST. The ground-truth SCM is as follows:
f 0.5 + Ur,
We r(10, 5),
i 191 (0.5 u, +) 64,
Us ~ NO 1),
SetIntensity (SeThhicnseeiiiesssssssssss uz: i)
22 MNIST,
where ue, us and uy are noise variables, and o is the sigmoid function. SetThickness( and SetIntensity( ()) are the operations set an MNIST digit u, 's thickness and intensity and respectively, and x is the generated image.
Table 5: MorphoMNIST results of change(i) or do(i) using V-SCM
Intersection between Ours and NB
(NC=1, NB=1) (NC=1. NB=0) (NC=0, NB=1) (NC=0, NB=0)
Number of Intersection
5865
3159
975
t's MAE
0.283
0.159
0.460
0.000
0.450
Non-backtracking
i's MAE
6,56
3.97
8.95
0.000
14.3
t's MAE
0.164
0.160
0.171
0.000
0.466
Ours
i's MAE
4.18
4.01
4.49
0.000
14.1
Quantitative Results of change or do(1). We use V-SCM to counterfactual task of change (i)
(where 10-3) do(2) with multiple random seeds test set. In Table 5, the first column shows
14

---

Evidence Nonbacktracking
Ours
2
n2
An3 = 2.31
Ano = 0.03
-2
2
n1
(a) Outcome error on single sample
1.5-
1.2-
y: nn''s Prediction
0.0
-0.5
y=x
Nonbacktracking
1.5
Ours
-1.5
-1.
ass 10
x: Dy's Groundtruth
(b) Groundtruth-Preciicioon Scatter Plot
Figure 3: The Visualization Results on Toy 1.
the MAE of (1, indicating our results outperform that of non-backtracking, since our approach consistently determines LBF interventions.
The Elfectiveness of FIO. Next, we focus on the rest four-column results. both types of counter- factuals, we use the same value in do(i) and change We can calculate which image satisfies e-natural generation. the table, "NC" indicates the set counterfactuals after FIO. Notice that "NC" set does not mean the results of natural counterfactuals, since some results do still not satisfy c-natural generation after FIO. "NC=1" mean the set containing data points satisfying c-natural generation and "NC=0" contains data not satisfying e-natural generation after FIO. Similarly, "NB=1" means set containing data points satisfying naturalness criteria in non-backtracking counterfactuals. (NC=1, NB=1) presents the intersection of "NC=1" and "NB=1*. A similar logic is adopted for the other three combinations. The number of counterfactual data points 10, 000 in two types of counterfactuals.
15

---

t=23;/=145
t= 3:2:/= 229
t
7
2
thicknezz
f-2:1//-10 103
r- 3.6; 226
intensity
9
i
X
thickness
Intersity
(a) Causal Graph
(h) Samples
(c) Scatter Plot Matrix
Figure 4: Causal Graph and samples of MorphoMNIST.
In (NC=1, NB=1) containing 5865 data points, our performance is similar to the non-backtracking, since FIO does not do backtracking when hard interventions have satisfied e-natural generation. In (NC=1, there are 3159 data points, which are "unnatural" points in non- -onbbacktracking counterfactuals. After FIO, this huge amount of data points becomes "natural". Here, our approach significantly reduces errors, achieving 62.8% reduction in thickness and 49.8% in intensity i, the most substantial improvement among the four sets in Table The number of points in (NC=0, NB=1) is zero, showing the stability of our algorithm since our FIO framework will change the hard, feasible intervention into an unfeasible intervention. Two types of counterfactuals perform similarly in the set (NC=0, NB=0), also showing the stability of our approach.
Table 6: Unfeasible solutions per 10,000 instances MorphoMNIST
Unfeasible Solutions
le-4
794
le-3
975
le-2
1166
Ablation Study on In Table 6 provided, we report the frequency of unfeasible solutions per 10,000 instances on MorphoMNIST, following optimization within the V-SCM framework. The data reveals consistent trend: as the value of increases, the frequency of unfeasible solutions also rises. This pattern occurs because higher e corresponds to a stricter standard of naturalness, making it more challenging achieve feasible outcomes.
Visualization of Counterfactual Images. Fig. 5 shows counterfactual images (second row), based on the evidence images (first row), with intended changes on í. The third row illustrates the differences between evidence and counterfactual images. Focusing on the first counterfactual image from non- backtracking and natural counterfactuals respectively, in non- non-backtracking. despite do(i) where thickness value should remain unchanged, the counterfactual image shows reduced thickness, consistent with the measured counterfactual thickness of 2.6. In contrast, natural counterfactuals yield an estimated counterfactual thickness (t* in MS) closely matching the original counterfactual thickness (t" in CF), due to backtracking for LBF intervention the earlier causal variable t, thereby maintaining (t, i) within the data distribution. Observing other images also shows larger errors in non-backtracking counterfactual images.
A.3 3DIdentBOX
The 3DIdemIBOX datasets, first introduced in [4], come with official code for generating customized versions of these datasets. They consist of images created with Blender, each depicting a teapot with seven attributes, such as position, rotation, and hue, determined by seven ground- -ruth variables.
In our experiment with the 3DIdentBOX, which comprises six datasets, we focus on the positions- rotations-hue dataset. We expand this into two datasets, Weak-3DIdent and Strong-3Dldent, Each dataset includes seven variables, besides the image variable x, with specifics outlined in Table Every image features teapot, with variables categorized into three groups: positions (z, y, 2).
16

---

2
4
3
2
1
(
2
4
5
3
2
(
2
4
S
|
3
2
1
(a) Results of Kon-backtracking Counterfactuals
Ale Seoddoie
1e der02 godd
Mir 2
(
2
4
3
2
(
2
3
2
4
S
C
2
4
5
3
2
1
(b) Results of Natural Counterfactuals
Figure 5: Visualization Results on MorphoMNIST: "FF stands for factual values, "CF" for coun- terfactual values, and "MS" for estimated counterfactual values of (1, (At, Ai) represents the absolute eтrors between counterfactual and estimated counterfactual values of (t, i).
Table 7: Details of variables in 3DIndentBOX. Object refers the teapot in each image. The support of each variable is [-1, 1]. The real visual range are listed in the column Visual Range.
Information Block
Variables
Support
Description
Visual Range
Object x-coordinate
Position
[-,
1]
Object y-coordinate
[1,
Object z-coordinate
[1.
1
Spotlight rotation angle
[0", 360
Rotation
[1,
Object a-rotation angle
[0", 360
[11, Object f3-rotation angle
[0", 360"
Hue
[11 1]
Background HSV color
[0%, 360"
rotations (7, or, (), and hue b, representing seven teapot attributes, as depicted (a). Fig. 7 (b) illustrates that both datasets share the same causal graph. It is important to note distinction between Weak-3DIden and Strong-3DIdent. the variables of each pair, shown in Fig. 7 (c), whereas in Strong-3DIdent, the causal relationship stronger, as depicted Fig. 7 (d). The distributions of several parent variables of image æ in these datasets are detailed in Table 8.
Weak-3DIdent, there exists a weak causal relationship between
Visualization on Strong-3Dident. Fig. displays counterfactuals, with the text above the evidence
images (first row) indicating errors for the counterfactual images (second row). In Fig. (a), it is evident that some images (second, third, and fifth images in particular), generated by non-backtracking counterfactuals are less recognizable and have larger emros.. Conversely, our counterfactual images exhibit better visual clarity and more distinet shapes, as our natural counterfactuals consistently ensure feasible interventions, resulting in more natural-looking images.
17

---

Table 8: Distributions in Weak-3DIdent and Strong-3l 3DIdent.N (y. 1) refers to normal distribution truncated to the interval [-1.1] and Nat 1) means normal distribution truncated to the interval [min(1, y + 0.2), max(-1.y 0.2)], where min and max indicate operations that select smaller and bigger values respectively. Na, (o. and (ax 1) are identical to Nur (v. 1) and Na. (v, 1) respectively. U refers to uniform distribution.
Variables
Distribution
Strong 3DIdent Distribution c (Nat 1,
C (r, 2.
(Nwe (y, 1), (-,U-11,1 ((111)
8
(7,
(Nut( ( 1),U -1,-1111 N,,,,, b 6~ U(-1,1)
(Nat 1), (-, Nar U(-1,1)
dd- 0.26, ah 0.24:
Ad 0.07, 20 0.06;
de 20600.22 ABl 40;
dell 0.000 All 0.04;
0.4. Al 0.
dy 0.23:4y 1.02; &e 22 1/25:
By 0.09: by 0.33:
dy 0.00 &y 96:
durm 58: dy 1.12;
da-m 0.70; dym 1.46;
de 0 1.80
be 0.00; 00:
Ber 1.50: 1:15
de 1.58, A8 0.33
(a) Results Non-backtracking Counterfactuals
0v.2 0.02:
0.08:
dr 0.06. 0..77
0.20:
By
Au 0.41:
Aum 0.09; 48 0.29
&e 0.04, 48 0.30
Ae 0.18::0 0.02
An 0.12: 0.05
(b) Results of Natural Counterfactuals
Figure 6: Visualization Results on Stong-3Dident,
A.4 Standard Deviation of Experimental Results
This section presents the standard deviation of all experimental results, demonstrating that the standard deviation for our natural counterfactuals is generally lower. This indicates the increased reliability of our approach, achieved by necessary backtracking ensure counterfactuals remain within data distributions.
Table Standard Deviation of Results on Toy Toy 4.
Dataset
Toy
Toy
Toy 3
Toy 4
do or change
do(n1)
do((2))
do(n))
do(n.))
do(n2)
do(ns)
do(ni)
do(n2)
Outcome
V22
71g
713
T12
7141
71g
Non- Nou-backtracking 0.00184 84 0. 0.0062810 00430000330000 00166 0.00448 0.00686 0.00495| 0.0112 0.00556 0.00142 0005 14 O. 0.00623 000238
Ours
0.00409 0.006840..0 00..00009000009900
0.001 0.00461 0.00201 0000000000000050 0.,0500
0.00531 0.00155
0. 000235 0.005180.00143
18

---

0.02hh
-0.59;
or 0.38 -0.611
bm-d.7
bindl
0.75, 0.46)
v
8
h
(a) Samples
(b) Causal Graph
Weak-3DIdent
(d) Strong-3Dident
Figure 7: Samples, Causal Graph, Scatter Plot Matrices of Weak-3DIdent and Strong-3Dldent
Table 10: Standard Deviation of Results on MorphoMNIST
Intersection between Ours NB
(NC=1, NB=1) (NC=1, NB=0)
(NC=0, NB=1)
(NC=0, NB=0)
Number of Intersection
39.84
81.43
0.00
54.14
t's MAE
0.00322
0.00172
0.00670
0.000
0.0178
Non- Non-backtracking
i's MAE
0.0496
0.0596
0.0508
0.000
0.110
U's MAE
0.00137 0.0359
0.00222 0.0551
0.00157 0.0157
0.000
0.0149
Ours
i's MAE
0.000
0.0853
Table 11: Standard Deviation of Ablation Study on Using MorphoMNIST
do(()
do(i)
Model
CFs
NB
0.000512
0.0172|
0.00322
0.0496
10
0.00159
0.0210
0.00183
0.0561
V-SCM
10
Ours
0.00124
0.0217
0.00137
0.0359
10
0.000954
0.0382
0.000868
0.0556
NB
0.0009955
0.0229
0.000832
0.0245
10
0.000920 0,0178
0.000922
0.0138
H-SCM
Ours
0.000611
0.0206
0.000289
0.0264
10
0.000787 0.0244 0.000431
0.0258
Table 12: Standard Deviation of Results on Weak-3 and Stong-3
Dataset
Counterfactuals
d
h
b
Non-backtracking Ours
3.68e-05
0.000133
0.000226
0,00422
0.00310
0.00357
1.29e-05
Weak-3DIdent
4.27e-05
7.22e-05
0.000249
0.00558
0.00278
0.00136
3.33e-05
Non-backttrcking
0.00233
0.000864
0.00127
0.00933
0.00307
0.00452
1,49e-05
Stong-3DIdent
Ours
0.00166
0.000774
0.000229
0.00908
0.00955
0.00816
2.97c-05
19

---

B Proof for Theorem 4.1
Theorem 4.1 shows that our natural counterfactuals are identiñialle given certain conditions. We unitize Theorem 1 from [20] to assist to prove it, shown as below.
Theorem B.1 (Identifiable Counterfactuals). Given actual distribution p(PA, U,), Suppose V; satisfies the following function:
V J((PA,, U.)
where U. L PA, and assume unknown f. is smooth and strictly monotonic w.E. foror fixed values PA; e. If we have observed V;- Vi and PA, pos, with an intervention do(PA, = pa)), where pag is within the support of pPA.)), the counterfactual oulcome identifiable:
Vildo(PA, =pa)), V, vi. PA, po;
The theorem is not identical to Theorem 1 from [20]. as we incorporate additional assumptions from [20] in our proof to express strietly. The proof is as below.
Proof. Step 1. Identifiahility of AC.. Given an intervention do(C = c²) on the set of variables C, where (C)) denotes the ancestors of C including C itself. By definition of the intervention and the causal graph structure, the value of (CC under the intervention do( = c²) can be uniquely determined. This is because the values of ancestors are not influenced by the intervention, while the values of C are directly determined by the intervention, making them identifiable.
Step 2. Identifiability of V \ (C..). There are two subsets in V \ AN(C). (1) ND: Variables in V (C(C) that are not descendants of (C()). These variables retain their values from the actual observed data since they are not influenced by the changes in AN(C) (C) due to the causal independence. (2) DS: Variables in V (N(C) that are descendants of (C()). The values of these variables might change as a result of the intervention due to their causal dependency on AN(C).
ldentifiability of ND. The counterfactual values of variables in ND remain their observed values in the actual data, as they are causally independent of (C(C). Thus, their values are trivially identifiable.
Identifiability DS. The counterfactual values of variables in DS depend on the values of their parents. For variable set DS; in DS whose parents only come from ADANN( (C), Their counterfactual values are identifiable by Theorem B. based on the fact that the counterfactual values of their parent variables are identifiable and lie within the observed distribution. Now, the the set of identifiable variables is DS, UND J AN C). This allows us to recursively expand the set of identifiable variables. By incorporating variables from DS, whose counterfactual values identifiable, into the identifiable variable set ND U AN(C) (C), this set is iteratively enlarged. This process is repeated until all variables in V are included in the set of identifiable variables, thus establishing the identifiability of their values under the counterfactual scenario induced by the intervention do c*).
Conclusion. By proving the identifiability of both AN(C) and all variables in V \ AC( (parti- tioned into ND and DS), we establish that the entire set of endogenous variables V is identifiable under the intervention do(C c²). This implies that the values of all variables in V can be uniquely determined given the causal graph, the data distribution of endogenous (observed) variables, full evidence, and a single LBF intervention.
C Causal Model Training
Our study focuses on counterfactual inference and we directly use two state-of-the-art deep-learning SCM models to learn conditional distributions among variables using a dataset, i.e., V-SCM [25] and H-SCM [28]. Specifically, we use code of [28] containing the implementation of V-SCM and H-SCM. Take MorphoMNIST as example, in both two models, normalizing flows are firstly trained to learn causal mechanisms for all variables except image x, i.e., i), and conditional VAE used to model image given its parents (l, For V-SCM, the conditional VAE uses a normal VAE framework, while H-SCM uses a hierarchical VAE structure [21] to better capture the distribution of images.
20

---

Toy Experiments. In the case of four toy experiments, we exclusively employed normalizing flows due to the fact that variables are one-dimensional. Our training regimen for the flow-based model spanned 2000 epochs, utilizing batch size of 100 in conjunction with the A optimizer [19]. We initialized the learning rate to 10-3, set 31 to 0.9, f8 to 0.9.
MorphoMNIST. We first train normalized flows to learn causal mechanisms of thickness and intensity (t, i). Other hyper-parameters are similar those of toy experiments. Then, we train two VAE-based models (V-SCM and H-SCM) to learn I given (t, i) respectively. The architectures of the two models are identical [28]. V-SCM and H-SCM underwent training for 160 epochs. We employed a batch size of 32 and utilized the AdamW optimizer. The initial learning rate was set to 1c - 3 and underwent linear warmup consisting of 100 steps. We set 31 0.9, 82 to 0.9, and applied a weight decay of 0.01. Furthermore, we implemented gradient clipping at a threshold of 350 and introduced gradient update skipping mechanism, with a threshold set at 500 based on the T,2 norm. During the testing, i.e. counterfactual inference, we test performance on both models respectively, with the normalized flows.
3DIdentBOX. Similar to experiments on MorphoMNIST, we first train normalized flows. Compared with V-SCM, H-SCM is more powerful to model complex data like 3DIdentBOX, of which the size of the image is 64 x 64 x 4. Then, we train H-SCM to capture the distribution of æ given its parents for 500 epochs with a batch size 32. The hyper-parametee are the same as experiments on MorphoMNIST.
All the experiments above were run NVIDIA RTX 4090 GPUs.
D Feasible Intervention Optimization
To implement particular method, we plug the distance measure (Eqn. and naturalness constraint (we use Choice (3) in Sec. 4.1.1 in the experiments) into Eqn. 2 of the FIO framework:
minimize IIan(A) - an(A)ll
an A)
(8)
s.t.
A a=a
<< F(V, vi < <1-c,V, E AN(A).
According to the properties of reversible functions the causal model, endogenous value (n(A)* is reversible to noise values u AN of AN(A)" corresponding noise variables. Hence, optimizing endogenous value is equivalent optimizing noise value. Then we have:
mininize
Σ
((pm).uz) - vsl
D AN(A)
(9)
u f (pox.aa))
<< F(u)) < <1 <1-e,bU, UAN(A)
We finally use the Lagrangian method [5] to optimize our objective loss to get the counterfactual value given actual value and expected change A a* as below:
((NAA)
Σ (NIIA
(is(1.. vsl+ we " "JEAANIA)
[max(e- F(u)). max(e + F(u)) 1,0)]
s.t. U fA f= aa))
(10)
where the optimization parameters are the counterfactual noise values of A ancestors, u (AN(A)) and the function max(-) returns the maximum value between given values. The first term represents the measure of distance between two distinct worlds, while the second term enforces the constraint of e-natural generation. Here, the constant hyperparameter w, serves to penalize noise values situated in the tails noise distributions. For simplicity, we use A as a subscript as an indicator of terms related to A, instead of a number subscript. Notice that, in order to ensure hard constraint A a*, A noise value u'A is not optimized explicitly, since the value pa is fully determined by a" and other noise values.
Hyper-parameters for Optimization and Judgment for Non-solution Cases. The loss's parameter is thus UAN, which fully determines the value an (n)) using the pretrained causal model, as explained
21

---

in Sec. C. In all experiments, we optimized LA using the Adam W optimizer at a learning rate of 10 for 50, 000 steps. This approach's s effectiveness is validated by the MorphoMNIST experiments, as shown Table 5, As f is not perfectly invertible in practice, it is important to ensure that 1 - fu a²)] (where Vn = f (po a²) )))) remains sufficiently small, ideally on the order of 10- according to our experience, to satisfy the constraint condition in Eqn. 10. In our experience, this deviation Ipa - f (u a
| is typically sufficiently small throughout most of the optimization process. If not, please adjust hyperparameters, such as the learning rate and We, to achieve the desired accuracy. Assuming Ipa's fA (u a a²) is sufficiently small after optimization, if Eu 2ujeuANA [max((- F(u)).0) + max(c + F(u)) - 1,0) > 0 holds, this
indicates that the case does not have solution.
Computational Complexity of FIO. The complexity scales linearly with both the size of the causal graph, specifically the number of ancestors of A, and the dataset size. Formally, for counterfactual query, the overall complexity is O PT), where K is the number of ancestors of A, of parameters in the neural networks, and T is the number of optimization steps. If we use each instance of a whole dataset as evidence and answer one counterfactual query for each instance, the overall complexity becomes linear scaling indicates that FIO is reasonably scalable, making it suitable for large causal graphs and datasets.
the number
where M is the number of data points in the dataset. This
E Differences between Natural Counterfactuals and Non- Non-Backtracking Counterfactuals [26] or Prior-Based Backtracking Counterfactuals [35]
E.1 Differences between Non-backtracking Counterfactuals and Ours
Non-hacktracking counterfactuals only do direct intervention on A, while our natural counterfactuals do backtracking when the direct intervention is infeasible. Notice that when the direct intervention on A is already feasible, our procedure of natural counterfactuals will be automatically distilled to the bon-backtracking counterfactuals. this sense, non-backtracking counterfactual reasoning is our special case.
E.2 Differences between Prior-Based Backtracking Counterfactuals and Ours
(1) Intervention Approach and Resulting Changes:
Prior-based Backtracking Counterfactuals: These counterfactuals directly intervene on noise/exogenous variables, which can lead to unnecessary changes in the counterfactual world. Consequently, the similarity between the actual data point and its counterfactual counterpart tends to be lower. In short, prior-based backtracking counterfactuals may introduce changes that not needed.
Natural Counterfactuals: In contrast, our natural counterfactuals only engage in necessary backtrack- ing when direct intervention is infeasible. This approach aims to ensure that the counterfactual world results from minimal alterations, maintaining a higher degree of fidelity to the actual world.
(2) Counterfactual Worlds:
Prior-based Backtracking Counterfactuals: This approach assigns varying weights to the numerous potential counterfactual worlds capable of effecting the desired change. The weight assigned each world is directly proportional to its similarity to the actual world. it is worth noting that among this array of counterfactual worlds, some may exhibit minimal resemblance to the actual world, even when equipped with complete evidence, including the values of all endogenous variables. This
"To avoid misleading readers, we clarify why we apply the concept of intervention to exogenous variables. Prior-based backtracking counterfactuals do not appeal to interventions in the usual Pearlian sense, for those are restricted endogenous variables and associated with breaking endogenous mechanisms. However, formally speaking, changing the values exogenous variables is analogous intervening on exogenous variables, in the sense that they change those variables without affecting the mechanisms for other variables in the model (the invariance of the mechanisms for other variables is for many the crucial hallmark of intervention). In fact, authors explicitly advocate applying the notion of intervention exogenous variables well [33].
22

---

divergence arises because, by sampling from the posterior distribution of exogenous variables, even highly dissimilar worlds may still be drawn.
Natural Counterfactuals: In contrast, our natural counterfactuals prioritize the construction of coun- terfactual worlds that closely emulate the characteristics of the actual world through an optimization process. As result, in most instances, one actual world corresponds to single counterfactual world when employing natural counterfactuals with full evidence.
(3) Implementation Practicality:
based Backtracking Counterfactuals: The practical implementation of prior-based backtracking counterfactuals can be a daunting challenge. To date, we have been prevented from conducting comparative experiment with this approach due to uncertainty about its feasibility in practical applications. Among other tasks, the computation the posterior distribution of exogenous variables can be a computationally intensive endeavor. Furthermore, it is worth noting that the paper [35] provides only rudimentary examples without presenting comprehensive algorithm or accompanying experimental results.
Natural Counterfactuals: In stark contrast, our natural counterfactuals have been meticulously designed with practicality at the forefront. We have developed a user- useerffiendly algorithm that can be applied in real-world scenarios. Rigorous experimentation, involving four simulation datasets and two public datasets, has confirmed the efficacy and reliability of our approach. This extensive validation underscores the accessibility and utility of our algorithm for tackling specific problems, making it valuable tool for practical applications.
F Observations about the Prior-Based Backtracking Counterfactuals [35]
F1
Possibility of Gratuitous Changes
A theory of backtracking counterfactuals was recently proposed by [35], which utilizes a prior distribution U*) to establish a connection between the actual model and the counterfactual model. This approach allows for the generation of counterfactual results under any condition by considering paths that backtrack to exogenous noises and measuring closeness in terms of noise terms. As result, for any given values of E e and A a", is possible to find sampled value (U=u, U u²) from (UU U*) such that EM(()) = e and A' (u') = a", as described in [35]. This holds true even in cases where V \ E
and V A 0. implying that any combination of endogenous values = e and A a* can co-occur the actual world and the counterfactual world, respectively. In essence, there always exists path (v - - u" - v²) that connects V and V through a value (U- = u, U u²), where v and v* represent any values sampled from P. (V) and PM- (V*), respectively.
However, thanks to this feature, this understanding counterfactuals may allow for what appears to be gratuitous changes in realizing counterfactual supposition. This occurs when there exists a value assignment U u" that satisfies E
in the same world. In such case, intuitively we ought to expect that E- should be maintained in the counterfactual world (as in the factual one). However, there is in general positive probability for E' * e. This is due to the existence of at least one "path" " from E = to any value v sampled from PM- (V+ = a*) by means of at least one value (U u, U u'), allowing E to take any value in the support of P. (E* ====a, a*).
and A AM
In the case where A = 0, an interesting observation is that E can take any value within the support of PM- (E*). Furthermore, when examining the updated exogenous distribution, we find that in Pearl's non- non-backtracking framework, it is given by PM E = e). However, in [35]'s backtracking framework, the updated exogenous distribution becomes Pr E = e) = (UUUUUUPMM e)d(U) * P.M-MUS E e), since using u* sampled from P(U" U u) (where u is any value of U) can result in any value of all endogenous variables V*. Therefore, [35]'s backtracking counterfactual does not reduce to Pearl's counterfactual even when A = 0.
23

---

F.2 Issues with the Distance Measure
In Equation 3.16 of [35], Mahalanobis distance is used for real-valued U R", defined as d(u'. u (u" - - (u" u). However, it should be noted that the exogenous variables are not identifiable. There are several issues with using the Mahalanobis distance in this context.
Firstly, selecting different exogenous distributions would result in different distances. This lack of identifiability makes the distance measure sensitive the choice of exogenous distributions.
Secondly, different noise variables may have different scales. By using the Mahalanobis distance, the variables with larger scales would dominate the distribution changes, which may not accurately reflect the changes each variable fairly.
Thirdly, even if the Mahalanobis distance d(u", u) is very close to O. it does not guarantee that the values of the endogenous variables are similar. This means that the Mahalanobis distance alone may not capture the similarity or dissimilarity of the endogenous variables adequately.
G Another Type of Minimal Change: Minimal Change in Local Causal Mechanisms.
Changes in local mechanisms are the price we pay to do interventions, since interventions are from outside the model and sometimes are imposed on model by us. Hence, we consider the minimal change in local causal mechanisms in A' 's ancestor set AN(A). With L' norm, the total distance of mechanisms in (A(A), called mechanism distance, is defined as:
(Aun((A): Uan(A)-)
Σ
wy (
(11)
1y (l)n uf (uan(A)
where (A) is the value A's exogenous ancestor set U UAN(A) when (N = an(A) in the actual world. (an(A) the value of U UNN(A) when AN(A) = an(A)' the counterfactual world. (Aun)(A): Uan(A)-) represents the distance between actual world and counterfactual world. wy represents fixed weight, and F-)) is the Cumulative Distribution Function (CDF). We employ the CDF of noise variables to normalize distances across various noise distributions, ensuring these distances fall within the range of (0, 1]. noise distributions not identifiable.
Weights in the Distance. The noise variables are independent each other and thus, unlike in perception distance, the change of causal earlier noise variables will not lead to the change of causal later noise nodes. Therefore, if the weight on the difference of each noise variable is the same, the distance will not prefer less change possible, we set a weight wy for each node U,. defined as the number endogenous decedents of V, denoted N ND(V,). j). Generally speaking, for all variables causally earlier than A, one way is to use the number of variables influenced by a particular intervention as the measure of the changes caused by the intervention. Hence, the number of variables influenced by variable's intervention be treated as the coefficient of distance. For example, in a causal graph where where B causes and C is the confounder of A and B. If change A a*), UA and uB 's weight is 1 and respectively, In this way, variables (e.g., Up) with bigger influence on other variables possess bigger weights and thus tend change less, reflecting necessary backtracking.
variables closer to A. To achieve backtracking
little as
G.1 Concretization of Natural Counterfactuals: An Example Methodology
Method Based on Mechanism Distance. We plugin mechanism distance Eqn. 11 into FIO framework Eqn. 2. Below is the equation of optimization:
min Ma)(A)
Σ
F(u)))
u, U,Euen(A): EMan(A)
(12)
s.t. a" = fA fp (
s.t. E< << F(3)) <
e, Vu3 E Uan(A)
Where the first constraint is to achieve change( A a"); the second constraint require counterfactual data point to satisfy e-natural generation given the optional naturalness criteria (3) in Sec. 4. I, and the
24

---

optimization parameter is the value A)n(A) of noise variable set UAN(A) given (N(A) =a(()))n(AA)
For simplicity, we use A as subscript as an indicator of terms related A, instead of number subscript. In practice the Lagrangian method is used to optimize our objective function. The loss is as below:
Σ
W -
uy EuuntA) (H
+ We
Σ
(ex((c-F(uj))) max(c F(u)) 1))
(13)
11 (ll)()):l1) (laa(A)
8.t. U LA (a" ,
In the next section, we use Eqn. 13 for feasible intervention optimization across multiple machine learning case studies, showing that mechanism distance is as effective as perception distance, as discussed in the main paper.
G.2
Case Study
G.2.1
MorphoMNIST
Table 13: MorphoMNIST results of change(i) or do(i) using V-SCM
Intersection between Ours and NB
(NC=1, NB=1) (NC=1, NB=0) (NC=0, NB=1) (NC=0, NB=0)
Number Intersection
5841
3064
1094
t's MAE
0.286
0.159
0.462
0.000
0.471
Non- Non-backtracking
i's MAE
6.62
4.00
8.88
0.000
14.2
t's MAE 0.175
0.159
0,206
0.000
0.471
Ours
i's MAE
4.41
4.00
5.19
0.000
14.2
In this section, study types of counterfactuals the dataset called MorphoMNIST, which contains three variables (t, i, r). From the causal graph shown in Fig. 8 (a), é (the thickness of digit stroke) is the cause of both (intensity of digit stroke) and (images) and is the direct cause of Fig. 8 (b) shows sample from the dataset. The dataset contains 60000 images as the training set and 10000 the set.
We follow the experimental settings of simulation experiments Sec. 6.1, except for two differences. One is that
state-of-the-ard deep learning models, namely V-SCM [25] and H-SCM [28], as the backbones to learn counterfactuals. They use normalizing flows to learn causal relationships among I's parent nodes, e.g., (t) in MorphoMNIST. Further, to learn p(zlt, i), notice that V-SCM uses VAE [17] and HVAE [21]. Another difference is that, instead of estimating the outcome with MAE, we follow the same metric called counterfactual effectiveness in [28] developed by [22], First, trained on the dataset, parent predictors given a value of x can predict parent values, i.e., (4, i)'s, and then measure the absolute eтror between parent values after hard intervention LBF intervention and their predicted values, which is measured on image the pretrained causal model generates given the input of if,
Table 14: Ablation Study on e
do(() t
do(i)
Model
CFs
i
NB
0,336
4..51
0,286
6..2
10
0.314
4.48
0.197
4.90
V-SCM
Ours
0.297
4.47
0.175
4.41
10-2
0.139
4.35
0.151
3.95
NB
0.280
2.54
0.202
3.31
4
0.260
2.49
0.117
2.23
H-SCM
10-3
Ours
0.245
2.44
0.103
2.03
10-2
0..0999
2.34
0.0863
1.87
25

---

2
9
(a)
Causal
(b) Samples
Graph
Figure 8: Causal Graph and samples of MorphoMNIST.
Quantitative Results of change(i) or do(1). We use V-SCM to do counterfactual task of change (where 10-3) do(i) with multiple random seeds test set. In Table 13, the first column shows the MAE of (t. i), indicating our results outperform that of non non-backtracking. Next, we focus on the rest four-column results. In both types of counterfactuals, change(i). Hence, after inference, we know which image satisfies e-natural generation in the two types of counterfactuals. In "NC=1" of the table, NC indicates the set of counterfactuals after feasible intervention optimization. Notice that NC set does not mean the results of natural counterfactuals, since some results do still not satisfy e-natural generation after feasible intervention optimization. "NC=1" mean the set containing data points satisfying e-natural generation and "NC=0" contains data not satisfying e-natural generation after feasible intervention optimization. Similarly, "NB=1" means the set containing data points satisfying naturalness criteria. (NC=1, NB=1) presents the intersection of "NC=1" and "NB=1". Similar logic is adopted the other three combinations. The number of counterfactual data points is 10000 in two types of counterfactuals.
use the same value i in do(i) and
In (NC=1, NB=1) containing 5841 data points, our performance similar the non-backtracking. showing feasible intervention optimization tends to backtrack as less as possible when hard interven- tions have satisfied e-natural generation. In (NC=1, NB=0), there are 3064 data points, which "unnatural" points in non- non-backtracking counterfactuals. After natural counterfactual optimization, this huge amount of data points becomes "natural", In this set, our approach contributes to the maximal improvement compared to the other three sets in Table 13, improving 55.4% and 41.6% on thickness and intensity The number of points in (NC=0, NB=1) zero, showing the stability of our algorithm since our approach will not move the hard, feasible intervention into an unfeasible intervention. Two types of counterfactuals perform similarly in the set (NC=0, NB=0), also showing the stability of our approach.
Ablation Study Naturalness Threshold c. We two models,V-SCM, V-SCM and H-SCM, do counterfactuals with different values of e. As shown in Table 14, our error is reduced as the increases using the same inference model, since the higher will select more feasible interventions.
D C
17
B
h
c-
0-
-1
(a) Causal Graph
(b) Weak Causal Relationship
(c) Strong Causal Relationship
Figure 9: Causal graph of 3DIdent and the causal relationships of variables (d, h) in Weak-3DIder and Strong-3D 3DIdent respectively.
G.2.2 3DIdentBOX
In this task, we utilize practical public datasets called 3DIdentBOX, which encompass multiple datascts [4]. Specifically, we employ Weak-3DIdent and Strong-3DIdent both of which share the
26

---

1.04:
0.52
0.2 1.32.
1.51
Ai 1.4::64- 1.54
Au- 1500; 0.04
de 160 4d 0.64
Ae 1.61.d 0.36
bur 1.73, ad 0.47
(a) Results of Non-backtracking Counterfactuals
0 25: de 0.52:
ddia 0.00 6h 0.0R: dr 0.07: 6.000.04 bar
0d 0.05. &h 0.02: By 0.06, aye 0.51
dd 6 01:16 0.01: dy a 0.13:
i 01: os; Ay 64: 0.14:
A Ay 6.46;
21:02
(b) Results of Natural Counterfactuals
Figure 10:
Visualization Results on Stong-3DIdent.
Table 15: Results on Weak-3DIdent and Stong-3DIdent
Dataset
Counterfactuals
d
7
B
Non-backtracking
0.0252 0.0191 0.0346
0.364
0.266
0.0805
0.00417
Weak-3DIdent
Ours
0.0241 0,0182 0.0339
0.348
0.224
0.0371
0.00416
Non-backtracking
0.104
0.0840
0.0770
0.385
0.495
0.338
0.00466
Stone-3DIdent
Ours
0.0633 0.0512 0.0518
0.326
0.348
0.151 0.0064
same causal graph depicted in Fig. 9 (b), consisting of an image variable denoted as and seven parent variables. These parent variables, denoted (d. h, v). control the depth, horizon position, and vertical position of the teapot in image respectively. Additionally, the variables (7, .3) govern three types of angles associated with the teapot within images, while variable represents the background color of the image. As illustrated in Fig. 9 (a), causal relationships exist among three pairs of parent variables, i.c., (h, d), (v. 3) and (a, 7). Wcak-3DIdes and Strong-3DIdent. In Weak-3DIdent, there exists a weak causal relationship between the variables of each pair, as shown in Fig. (b), whereas in Strong-3Didem the causal relationship stronger, depicted in Fig. (c).
important to note distinction between
We follow the same experimental setup as the MophoMNIST experiments. Using an epsilon value of E = 10 we employ the H-SCM as the inference model. We conduct interventions or changes on the variables (d, () and the results are presented Table 15. In both datasets, our approach outperforms the non-backtracking method, with Strong-3Dident exhibiting more significant margin over the non-backtracking method. This because the non-backtracking method encounters more unfeasible interventions when performing hard interventions using Strong-3DIdent.
27

---

Additionally, we perform visualizations on Strong-3Didenn In Fig. 10, we display counterfactual outcomes in (a) and (b), where the text above each image in the first row (evidence) indicates the eror in the corresponding counterfactual outcome shown in the second row. In Fig. 10 (a), we present counterfactual images that do not meet the e-natural generation criteria in the non-backtracking approach. In contrast, Fig. 10 (b) showcases our results, which are notably more visually effective. This demonstrates that our solution can alleviate the challenges posed by hard interventions in the bon-backtracking method.
28

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Sec.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Sec.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]

---

Justification:
Sec. 4.4 and Sec. B
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Sec. A, C, and D
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
30

---

Answer: [Yes]
Justification: Data are described in Sec. A and The code is available at httts://ggihhn
com/ wttpe//aanaauuaaaoouuutttooaeeee
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (kttps://nips.c cc/
wwwpabbisssooooollioooeoo Policyy)) for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / /nips. cttpcc/ppllc sec/public/pae cubbiccguides: /edeee//ooddummiii wwwssssssssiioooooooo for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Sec. C and D
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Sec. A.4
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
31

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπхо rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Sec. C
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https http:/:://eeeueeeeeeeeeeeeer https//ccccpccccccccccciiiiaeeee
Answer: [Yes]
Justification: The research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations their jurisdiction..
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: The paper focuses on fundamental perspectives in causality and emphasizes conceptual innovation.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
32

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: NA
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We have cited the original paper that produced the code package dataset.
Guidelines:
The answer NA means that the paper does
existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
33

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: NA
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: NA
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: NA
Guidelines:
The answer NA means that the paper does involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
34


Reviews:
[{"review_id": "tRJYswbT7o", "reviewer": "Reviewer_DQof", "summary": "This paper addresses a key limitation of non-backtracking counterfactual reasoning in causal inference. The authors argue that Pearl's framework often generates unrealistic scenarios. To solve this, they propose \"natural counterfactuals,\" which allow controlled backtracking to ensure scenarios remain realistic.They introduce Feasible Intervention Optimization (FIO), a novel framework that generates natural counterfactuals by incorporating a naturalness constraint. This ensures counterfactual instances are plausible given the observed data distribution. The authors also propose distance measures to minimize backtracking while achieving the desired outcome.", "questions": "- How sensitive is the proposed method to violations of the assumptions about the underlying SCM?\n\n- What is the computational complexity of FIO and how does it scale with the size of the causal graph and the dataset?", "limitations": "See weaknesses and questions", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well-written.\n\n- The paper identifies a crucial issue with non-backtracking counterfactuals, highlighting the importance of generating realistic scenarios.\n\n- FIO provides a principled and practical method for generating natural counterfactuals with clear mathematical structure.\n\n- The paper presents convincing experiments on both simulated and real-world datasets, demonstrating the effectiveness of the proposed approach.", "weaknesses": "- FIO involves several parameters and choices (naturalness criteria, distance measures), which may require careful tuning and consideration for different applications.\n\n- The paper mainly focuses on contrasting natural counterfactuals with non-backtracking ones. A more thorough comparison with existing backtracking approaches is missing in the experiment section.\n\n- The paper mainly focuses on contrasting natural counterfactuals with non-backtracking ones. A more thorough comparison with existing backtracking approaches is missing in the experiment section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "***Weaknesses***\n\n**1. FIO involves several parameters and choices (naturalness criteria, distance measures), which may require careful tuning and consideration for different applications.**\n\nThanks for raising this point. We agree that different applications may warrant different choices, as stated in our Conclusion section. Our main purpose is to provide a general and flexible framework for generating natural or realistic counterfactuals with minimal backtracking. The exact choices of parameters and measures can and should be tailored to specific applications, as you suggested.\n\n**2. The paper mainly focuses on contrasting natural counterfactuals with non-backtracking ones. A more thorough comparison with existing backtracking approaches is missing in the experiment section.**\n\nThank you for the suggestion. In light of them, we have added such comparisons to our updated paper. Backtracking Counterfactuals in [30] assume a known joint distribution over factual and counterfactual exogenous variables, which is often impractical due to the non-identifiability of the exogenous distribution. In contrast, we only require access to the observed distribution. In our simulation experiments, for the purpose of implementing Backtracking Counterfactuals, we assume the joint distribution over factual and counterfactual exogenous variables is known. the results show that backtracking counterfactuals can generate infinite counterfactuals for a single query, with higher probability given to those more similar to the actual instance. Conversely, our approach typically produces a single counterfactual closest to the actual instance. Note that an extreme case illustrates that [30] sometimes invokes gratuitous changes. When the counterfactual value $a^*$ equals the actual value $a$, the counterfactual instance should logically match the actual one. However, [30] also assigns positive probability to other instances.\n\nReference:\n\n[30] Julius von Kügelgen, Abdirisak Mohamed, and Sander Beckers. Backtracking counterfactuals. arXiv preprint arXiv:2211.00472, 2022.\n\n***Questions***\n\n**1. How sensitive is the proposed method to violations of the assumptions about the underlying SCM?**\n\nAs widely seen in the literature of causal inference and counterfactual inference, certain assumptions are essential. For Theorem 4, if the assumptions do not hold, identifiability may not be guaranteed. For example, suppose $Y=XU_1+U_2$ where $Y$ and $X$ are endogenous variables and $U_1$ and $U_2$ are exogenous noises, then the counterfactual outcome will not be identifiable.\n\n**2. What is the computational complexity of FIO and how does it scale with the size of the causal graph and the dataset?**\n\nThe computational complexity of FIO scales linearly with both the size of the causal graph (specifically, the number of ancestors of $A$) and the dataset size. Formally, the overall complexity is $O(KPTM)$, where $K$ is the number of ancestors of $A$, $P$ is the number of parameters in the neural networks, $T$ is the number of optimization steps, and $M$ is the number of data points in the dataset. This linear scaling indicates that FIO is reasonably scalable, making it suitable for large causal graphs and datasets.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eVddvrNAhP", "reviewer": "Reviewer_v2As", "summary": "The paper takes the recently developed idea of backtracking counterfactuals and applies it to improve the realistic generation of counterfactuals from data, which is known to be a hard task as the standard, non-backtracking, counterfactuals lie out of the distribution and generative models perform badly on those.", "questions": "Full disclosure: I reviewed a previous version of this paper for last year's conference, and the current version is much much better. Still, there are some issues which the authors should address. Some of them might be mistakes on my part due to my misunderstanding, others are questions for clarification, others are problems that need to be solved. \n\nFirst I present some larger issues, followed by some minor issues as they appear in the order of the paper.\n\n1: I find the motivation rather odd, which is that the supposed benefit of natural counterfactuals is that they are easier to learn. But surely the main focus should be: which counterfactual (backtracking, non-backtracking, partial backtracking) is appropriate/meaningful in a given situation, rather than “which one can we learn”. Simply put, we shouldn't just focus on learning something because it's easy to learn, the thing learned should also be sensible. The paper actually remains silent on this issue, or rather, it seems to imply that non-backtracking counterfactuals are always the only sensible ones, and their natural counterfactuals are therefore just a heuristic. The reason I say this is implied is because in the experiments the non-backtracking one is taken as the ground truth. This is at odds with the related literature cited, which in fact argues that sometimes the backtracking counterfactual is the \"ground truth\", i.e., semantically it is what is meant with a counterfactual query. It would be good if the authors could be more explicit on where they stand on this issue.\n\n2: There are several implicit assumptions that should be addressed. \n\n- It is assumed that there is a unique an(A)*, yet the definitions given do not guarantee unicity. \n\n- Initially it seems as if it is also assumed that an an(A)* always exists, which is not the case. Although this is made clear later, it would be good to flag this early on. \n\n3: 159: This I don’t understand, because it violates the idea that C has to include A. Take the trivial case in which a*=a.  Then you will get the empty LBF intervention, which of course does not include A.\n\n4: Building on the previous point, what if we have A=a, and yet A=a was extremely unlikely (so in the far-end of the tail of the distribution). Now consider the counterfactual a*=a. Will we not get that there is no solution? And if so, isn't that strange? This could be mitigated by combining the distance criterion and the naturalness criterion in a more weighted fashion, rather than having the latter be a necessary criterion and only then apply the distance one. This would bring it more in line with [30], see their 3.17 in particular. \n\n5: Continuing, note that Choice (2) seems to result in a variation of 3.17, except that the distance is here between endogenous variables. Note though that a distance on endogenous variables can easily induce a distance between exogenous variables, by considering the two endogenous states that would result from two exogenous states, and thus this is not an essential difference.\n\n6: Th4.1 is described as being about the identifiability of counterfactuals, but it is not, because it already assumes knowledge of do(C=c*), and that is part of what needs to be identified. Furthermore, it is then said that the theorem confirms that counterfactuals fall within the support of the observational distribution, but this ignores the earlier point that an LBF need not exist, and thus we are only guaranteed to identify counterfactuals in the case that it does. \n\n7: I did not have time to closely examine the experiments, except for the following: all the results are only about those counterfactuals which happen to be in the scope of the natural counterfactuals, and the others are excluded. Yet it seems crucial to know how many were excluded in this manner in order to evaluate how useful in practice this method is, so this should be reported as well.\n\nMinor issues:\n\n81: \"a most recent paper\" I’d say a more accurate description is that [30] was the first paper to formally introduce backtracking counterfactuals within the causal models framework, and thus the current paper builds on that one. (Also, the usefulness for counterfactual explanations is also something that is explicitly discussed in [30].)\n\n100: So the paper is limited to assuming independent and unique exogenous variables. Note that this is not the case in [30].\n\n121: \"In this paper...\" Why? The generalization seems trivial.\n\n175: This assumes that all variables are real-valued. Yet later it is mentioned that not all variables need to be at the same scale. Isn't that inconsistent?\n\n221: Why restrict to a single variable A all of a sudden?\n\nTh4.1: Doesn't the independence of U_i and Pa_i already follow from the independence of the exogenous variables?\n\nTh4.1: Why is there no mention of the distance criterion for do(C=c*)?\n\n\nTypos:\n\n110: \"distribution\" -> \"distributions\"\n\n123: \"date\"\n\n234: \"encourage\"\n\n303: \"datasets, which\"", "limitations": "Yes", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The idea of using backtracking counterfactuals as being more \"natural\", and then invoking them to improve the generation of counterfactuals when the full SCM is not available, is a very good one.", "weaknesses": "There are some technical issues which need to be addressed before the paper can be accepted.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are very grateful for your previous feedback, which helped a lot to improve the paper. Thank you also for the new comments. We will address them and correct the typos you pointed out.\n\n**1: Surely the main focus should be: which counterfactual (backtracking, non-backtracking, partial backtracking) is appropriate/meaningful in a given situation, rather than “which one can we learn”. The reason I say this is implied is because in the experiments the non-backtracking one is taken as the ground truth.**\n\nThank you for this insightful comment, which has helped make our motivation more clearly articulated. It is a consequence of our motivation that our counterfactuals are more learnable than non-backtracking counterfactuals, but being easier to learn is not the primary motivation. We are mainly motivated by observing that thoroughly non-backtracking counterfactual generation, for all its merits, often results in scenarios that are too unrealistic (and so, as a consequence, unreliable when it is based on training data). On the other hand, however, we also think that fully backtracking counterfactuals may not have much implication for guiding actions or decision-making, because they appeal to inventions in unobserved or even unobservable, unidentifable variables. We thus aim to develop a framework that generates sufficiently realistic counterfactuals by finding feasible interventions on observed variables while minimizing the extent of backtracking. In particular, when the intervention set $C$ picked out by our algorithm is sufficiently simple (a special case is that it is identical to the original target set $A$), the counterfactual will remain useful for guiding actions. \n\n*In addition, we hasten to clarify that we did not use non-backtracking counterfactuals as the ground truth.* We apply the same trained models for inferring non-backtracking and natural counterfactuals. In toy experiments, a ground-truth SCM serves as the gold standard to measure the error of inference. Specifically, we compare the ground-truth outcomes with the predicted outcomes after intervening on $A$ in non-backtracking counterfactuals or intervening on $C$ in our natural counterfactuals. In other words, non-backtracking counterfactuals and natural counterfactuals have different ground truths, and the experiments illustrate that the generative algorithm for the former deviates more from its ground truth than our generative algorithm does from the latter's ground truth. In real-world datasets, due to the lack of a ground-truth SCM, we measure how accurately predicted outcomes reflect desired image attributes, which are represented by the input values stated in the antecedents of the counterfactuals, for example, $t = 3$, meaning that thickness should be $3$.\n\n**2: It is assumed that there is a unique $an(A)^\\*$, yet the definitions given do not guarantee unicity.**\n\nThank you for pointing this out. We do not assume that $an(A)^*$ is unique. In fact, $an(A)^*$ can theoretically have multiple solutions (or no solution at all as mentioned in Section 5). We have made this clearer in our updated paper. Even though $an(A)^*$ can have multiple solutions, the inference method remains the same after we sample one value from the available solutions. \n\n**3: 159: This I don’t understand, because it violates the idea that C has to include A. Take the trivial case in which a\\*=a. Then you will get the empty LBF intervention, which of course does not include A.**\n\nWe follow the standard treatment by requiring that even when the counterfactual value $a^*$ is equal to the actual value $a$, an intervention would still be invoked to realize $a^*=a$ (among other things, the links between $A$ and its parents would still be severed). Hence, in the special case when $a^*=a$, $C$ still contains $A$.\n\n**4: Building on the previous point, what if we have A=a, and yet A=a was extremely unlikely (so in the far-end of the tail of the distribution). Now consider the counterfactual a\\*=a. Will we not get that there is no solution?**\n\nExcellent point! In natural counterfactuals, we use a hyperparameter $\\epsilon$ to determine whether a point is considered natural. Therefore, in this situation, there will be no solution if $A = a$ is so unlikely that it conflicts with $\\epsilon$-natural generation. This is a consequence of introducing a parameter to control the degree of naturalness. However, if we only require the lowest level of naturalness, where $\\epsilon$ is zero (i.e., we only require that counterfactuals are within the distribution support), this situation will be eliminated. Otherwise, yes, there could turn out to be no solutions for natural counterfactuals. As you noticed, this may happen in extreme situations, which are not \"natural\" themselves and already deserve extra inspection and attention.  We have made it clearer in the paper.\n\n\n**5: Continuing, note that Choice (2) seems to result in a variation of 3.17, except that the distance is here between endogenous variables.**\n\nThank you for this interesting comparison. Here we do not consider Choice (2) to be a variation of 3.17, because while 3.17 is based on value distance, Choice (2) relies on CDF distance. This distinction reflects a key shortcoming of 3.17: the distribution of exogenous variables is not theoretically identifiable, making it challenging to apply in practice. However, due to the properties of CDF distance, Choice (2) allows us to avoid calculating distances related to exogenous variables. Thus, Choice (2) can be computed without considering the distributions of exogenous variables.\n\n***``Due to space limitations, we have included questions 6 and 7, along with minor issues, in the global response.``***", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "j51RAyHidh", "reviewer": "Reviewer_JBox", "summary": "The paper presents a framework for generating \"natural counterfactuals\" that are more feasible within the support of  the training data distribution. This approach includes controlled backtracking through an optimization method  that uses a \"naturalness\" criterion as a constraint.", "questions": "(1) Line 46: “When interventions lead to unrealistic scenarios relative to the training data,  predicting counterfactual outcomes in such scenarios can be highly uncertain and inaccurate [12].” \nReference [12] addresses selection bias in potential outcome counterfactuals, in particular,  when the treatment group does not match the control group. It is not related to SCM counterfactuals.\n\n(2) Line 48 “This issue becomes particularly pronounced when non-parametric models are employed, as they  often struggle to generalize to unseen, out-of-distribution data [27]. “ \nReference [27] discusses the issue of i.i.d. in machine learning, particularly when the training data distribution differs from the interventional distribution, leading to out-of-distribution scenarios. How is the current paper related  to the discussion in [27]?\n\n(3) Appendix F: “Therefore, [30]’s backtracking counterfactual does not reduce to Pearl’s counterfactual even when $A^∗ =\\emptyset$.”\nThis is incorrect. Neither non-backtracking nor backtracking is a special case of the other. Appendix A of [30] presents a unified framework for counterfactual reasoning that integrates both backtracking and non-backtracking counterfactuals, as suggested by an area chair at that conference. Also, if $A^∗ =\\emptyset$, then there is no counterfactual query.\n\nReferences:\n\n[12] Negar Hassanpour and Russell Greiner. Learning disentangled representations for counterfactual\n      regression. In International Conference on Learning Representations, 2019.\n\n[27] Bernhard Schölkopf, Francesco Locatello, Stefan Bauer, Nan Rosemary Ke, Nal Kalchbrenner,\n      Anirudh Goyal, and Yoshua Bengio. Toward causal representation learning. Proceedings of the\n     IEEE, 109(5):612–634, 2021\n\n[30] Julius von Kügelgen, Abdirisak Mohamed, and Sander Beckers. Backtracking counterfactuals.\n      arXiv preprint arXiv:2211.00472, 2022.", "limitations": "NA", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "Original combination of existing ideas, e.g., Pearl's non-backtracking ( interventional ) counterfactuals, backtracking (observational) counterfactuals, normalizing flows to learn causal mechanisms", "weaknesses": "My first concern is with the term \"natural counterfactual.\" Both Pearl's non-backtracking counterfactuals and backtracking counterfactuals use an algorithm that begins with the prior probabilities of the exogenous variables. In the abduction step, the counterfactual posterior is computed based on the observed facts in the non-backtracking case, or on both the observed facts and observed counterfacts in the backtracking case. However, this paper samples from the observational distribution of the training data. \n\nTheorem 1 builds on prior work $[17]$ where the posterior distribution has been discussed, mentioning that due to the monotonicity assumption of $f_i$ with respect to \n$U$, the posterior distribution is a point mass on $𝑈=u$. However, the experiments in this paper sample from the prior distributions of the structural causal model (SCM), which means it is not a true counterfactual distribution.\n\nThis leads me to the purpose of the paper. It uses optimization to find a point within a high-density area of the data manifold generated by the observational distribution, given a request to change a value. Which use case would require this result?\nFor example, Pearl’s non-backtracking counterfactuals address questions on actual cause, causal necessity, and personalized policy. If performing $do(A)$ is not feasible, it does not imply that a feasible data point within the training data distribution can answer these questions. Further, the concept of \"naturalness\" is not based on the immutability of the variable but on the probability density of the variable given its parents. For example, the variable \"Age\" might meet the criterion based on its probability density, but it remains immutable.\n\nReferences:\n\n[17] Chaochao Lu, Biwei Huang, Ke Wang, José Miguel Hernández-Lobato, Kun Zhang, and\n      Bernhard Schölkopf. Sample-efficient reinforcement learning via counterfactual-based data\n     augmentation. arXiv preprint arXiv:2012.09092, 2020.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "***Weaknesses***\n\n\n**(1) My first concern is ... However, the experiments in this paper sample from the prior distributions of ... counterfactual distribution.**\n\nThanks for raising the concern. To clarify, we sample from the posterior distribution of $U$. First, due to the monotonicity assumption, sampling from the observational distribution amounts to sampling from the distribution of the exogenous variables. More importantly, during inference, the only difference is that non-backtracking counterfactuals use interventions on $A$, while natural counterfactuals use interventions on $C$, with observed evidence considered in both. Hence, we are sampling from posterior distribution of $U$. We have made this clearer in updated manuscript. \n\nYour further feedback would be appreciated.\n\n**(2) This leads me to the purpose of the paper. ... answer these questions.**\n\nThis is a good point that we had in mind before starting the project. We have revised the presentation to highlight our motivation and explain why we prefer feasible interventions for addressing counterfactual queries. We are motivated by observing that non-backtracking counterfactual generation, for all its merits, often results in scenarios that are not realistic (and so, as a consequence, unreliable when it is based on training data). On the other hand, we also think that fully backtracking counterfactuals do not have much implication for guiding actions, because they appeal to intervention on unobserved or even unobservable, unidentifiable variables. We thus develop a framework that generates realistic counterfactuals by finding feasible interventions on observed variables while minimizing backtracking. In particular, when the intervention set $C$ picked out by our algorithm is sufficiently simple (a special case is that it is identical to original target set $A$), the counterfactual will remain useful for guiding actions. \n\nAs demonstrated by our experiments, generating a non-backtracking counterfactual could be very unreliable, when it is not \"natural\" in our sense. So one use of our framework is to check on the reliability or feasibility of non-backtracking counterfactuals, and when dubious, to generate a reliable surrogate by adjusting the antecedent of the counterfactual. We think that reliable information about the adjusted counterfactuals will be more useful than probably unreliable information about the original, non-backtracking counterfactuals.\n\n**(3) Further, the concept of \"naturalness\" ... but it remains immutable.**\n\nThanks for raising the issue. As noted in the footnote on page 4, \"naturalness\" can have various interpretations. In this paper, we provide a general framework for natural counterfactuals without considering the meanings of specific variables. Hence, we do not discuss all possible constraints for various real-world scenarios that deserve to be labeled \"naturalness\". It is possible and in some situations important to incorporate more constraints, such as the relative ease of intervention, into this framework, though we did not attempt that in this paper. We will consider using another term if \"naturalness\" is perceived as inapt or misleading. We have included a remark on your good point in updated paper. \n\n***Questions***\n\n**(1) Line 46: Reference [12] addresses ... ... It is not related to SCM counterfactuals.**\n\n\nYes, [12] is based on the potential outcome framework, but we believe that one of its points is aligned with our statement: \"The fact that counterfactual outcomes are unobservable (i.e., not present in any training data) makes estimating treatment effects more difficult than the generalization problem in the supervised learning paradigm.\" Still, we are reconsidering whether this reference is more helpful than misleading in our context. Thanks for this helpful comment. \n\n**(2) Line 48: How is the current paper related to the discussion in [27]?**\n\nThanks for the question. One of our concerns about non-backtracking counterfactuals, despite their novelty and elegance, is precisely their frequent requirement to do out-of-sample generalization. This is why we believe the discussion in [27] of the difficulty of the out-of-distribution problem is very relevant. If you do not think this is convincing to you, please let us know.\n\n**(3) Appendix F: “[30]’s backtracking counterfactual does not reduce to Pearl’s ... no counterfactual query.**\n\nThis statement of ours is a bit confusing. Thanks for pointing it out! We did not mean to claim that backtracking counterfactuals in [30] are or should be a special case of Pearl’s counterfactuals. What we meant was that $A^* = \\emptyset$ represents a special though extreme case of a counterfactual query, and intuitively, when $A^* = \\emptyset$, the counterfactual instance should be exactly the same as the actual instance, for nothing is supposed in the antecedent of the counterfactual. Pearl's counterfactuals agree with this intuition. However, [30]'s backtracking counterfactual does not reduce to this result in this special case, as it allows changes in the exogenous variables even in the case of a null antecedent. Indeed, it would in the special case allow any value setting of the exogenous variables, for they are all compatible with the null antecedent. \n\nNote that another extreme case could illustrate that backtracking counterfactuals proposed in [30] seem to invoke gratuitous changes sometimes. In general, when the counterfactual value $a^*$ is supposed to be equal to the actual value $a$, Pearl's theory will return the exact same instance as the actual one, which is intuitively reasonable, but the theory in [30] will give positive probability to other instances as well. One may stipulate that $a^*$ must be different from $a$ in order to count as a counterfactual query, but to our knowledge, the literature usually understands counterfactuals as just subjunctive conditionals, which also includes subjunctive conditionals with true antecedents as special cases.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "urKZ7Xiyly", "reviewer": "Reviewer_fYWn", "summary": "The authors propose a novel approach for generating causally valid and *natural* counterfactuals. These counterfactuals are *natural* in that they are close to the observed data manifold. This is achieved by allowing a certain amount of backtracking, which involves tracing back upstream causal effects on variables.\n\nThe authors essentially propose a more flexible version of Judea Pearl’s $do$-calculus, by replacing the explicitly non-backtracking $do$-operator with a $change$-operators that represents the notion of a *least-backtracking feasible intervention* (LBF). Intuitively, LBF interventions take into account the effects of causal ancestors instead of considering the intervention itself as entirely exogenous.\n\nIn order to compute LBF interventions, the authors propose a *feasible intervention optimization* that targets minimal changes to the causal ancestors of some variable $A$ while satisfying certain naturalness constraints. These constraints ensure that the generated counterfactual remains anchored in the observed data manifold. Through experiments involving both synthetic and real-world datasets, the authors demonstrate that their proposed method yields more “natural” counterfactuals than the non-backtracking alternative approach.\n\nThe proposed method has some clear links to the literature on counterfactual explanations, which are mentioned but not thoroughly explored here. This is an interesting avenue for future research and this work constitutes a solid first step in that direction. Overall, the paper seems to make a strong contribution to the field of Causal Inference, although I am not a subject expert.", "questions": "### Link to CE\n\n- How does this approach compare to existing approaches to causal algorithmic recourse (e.g. [MINT](https://arxiv.org/abs/2002.06278))?\n- How do the proposed *naturalness* constraints compare to *plausibility* (e.g.  [Artelt and Hammer](https://arxiv.org/abs/2002.04862), [REVISE](https://arxiv.org/abs/1907.09615)) and *feasibility* (e.g. [FACE](https://arxiv.org/abs/1909.09369)) constraints in CE/algorithmic recourse? Equation 2 in your paper looks quite similar to Equation 5 in [Artelt and Hammer](https://arxiv.org/abs/2002.04862).\n\n### Other questions\n\n- The sentence spanning from line 81 to 86 is very long and it’s easy to get lost in it. Consider splitting this into 2/3 sentences.\n- Figures and annotations on page 8 are very small (annotations are barely legible).\n- Could you highlight the single point from Figure 1 (a) also in panel (b)? To illustrate that in (b) you essentially repeat the experiment over all test samples.", "limitations": "The authors mention that the assumption of invertibility is a small limitation, but not theoretically problematic. The work may have other limitations that I have failed to recognized, as I’m not a subject expert.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- Interesting and natural extension of non-backtracking counterfactual generation.\n- Empirical results for both synthetic and real-world datasets show improved performance when LFB is used.", "weaknesses": "- No standard errors reported in Tables, so difficult to assess how substantial the performance differences are after accounting for noise.\n- Link to CE is mentioned, but the authors could have gone into more detail. In particular, you propose that the literature on CE could benefit from incorporating your definition of *naturalness* (which I don’t disagree with) but you fall short of comparing it to existing definitions in CE. See also my related questions.\n- The figures and annotations are too small (and in general, both figures and tables currently make the paper look a little crammed).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "***Weaknesses***\n\n**1. No standard errors reported in Tables, so difficult to assess how substantial the performance differences are after accounting for noise.**\n\nThank you for asking this. We have included the standard errors of all experiments in Tables 9-12 in Section A.4 of the Appendix. As you can see, the standard deviation for our natural counterfactuals is generally lower.\n\n\n**2. The figures and annotations are too small (and in general, both figures and tables currently make the paper look a little crammed).**\n\n\nThank you. We have made modifications according to your helpful suggestions.\n\n***Questions Linked to CE***\n\n**1. How does this approach compare to existing approaches to causal algorithmic recourse (e.g. MINT)?**\n\nThank you for this great question. We would like to mention that the proposed work has different motivations and purposes. Our natural counterfactuals aim to provide a principled inference method to ensure that counterfactual scenarios remain realistic relative to the regular causal mechanisms,  with the actual (conditional) data distribution (using a limited extent of backtracking) as a surrogate of the mechanisms. Given an expected change or intervention, we offer a framework for inferring counterfactual outcomes. In contrast, causal algorithmic recourse focuses on identifying interventions to achieve a pre-specified desirable counterfactual outcome. In summary, similar to Pearl's counterfactuals, natural counterfactuals aim to perform counterfactual inference to obtain counterfactual outcomes, while causal algorithmic recourse seeks interventions to achieve a given desirable outcome.\n\nHowever, if we assume the aim is to find intervention variables to achieve $A = a^*$ in our framework, the task becomes more similar to finding interventions that cause a given outcome, as in causal algorithmic recourse. Still, our framework does not operate in exactly this way. Instead, we identify a group of intervention variables $C$ that includes $A$, which does not imply that intervening on variables other than $C$ causes $A$. This means that we perform interventions on all variables in $C$, including $A$, simultaneously. In other words, our aim is to find, so to speak, companion interventions in order to make the result ``natural'' in our sense.\n\n\n**2. How do the proposed naturalness constraints compare to plausibility (e.g. Artelt and Hammer, REVISE) and feasibility (e.g. FACE) constraints in CE/algorithmic recourse? Equation 2 in your paper looks quite similar to Equation 5 in Artelt and Hammer.**\n\n\nThank you for recommending these useful works. We have discussed them in our updated manuscript, including the paper mentioned in Question 1, summarized below. \n\n1. Our proposed naturalness constraints share a similar spirit with the feasibility constraints in counterfactual explanation (CE)/algorithmic recourse, as existing methods may overlook real-world constraints. However, there are two key differences:\n\n   - **Different Motivations:** We propose naturalness constraints for counterfactual inference, which generates counterfactual outcomes that are natural with respect to the actual world. In contrast, the feasibility constraints of CE/algorithmic recourse serve to identify possible interventions that can lead to a given desired outcome.\n\n   - **Scope of Application:** While feasibility constraints may consider more detailed constraints for a specific application, we propose a general framework without considering the meanings of specific variables. Of course, when applying our framework in a specific situation, users can and should consider special constraints relevant to that situation.\n\n2. The reason why Equation 2 in our paper looks similar to Equation 5 in [Artelt and Hammer](https://arxiv.org/pdf/2002.04862) is that both involve optimization with constraints related to the observed distribution. However, Artelt and Hammer focus on CE, and the two papers are fundamentally different:\n\n   - **Different Purposes:** CE is similar to algorithmic recourse in that it seeks to find the input for a target output. Our ultimate aim is to obtain a feasible output given an input.\n\n   - **Framework Differences:** Unlike algorithmic recourse, CE does not build on the SCM framework, making it difficult to explain the similarity between actual and counterfactual values in CE. Additionally, CE often studies anti-causal problems, such as classifying an image $x$ into a category $y$, where $y$ is the cause of $x$ instead of being the causal outcome. Although CE tries to minimize the difference between actual and counterfactual values, it is hard to claim that the distance is minimized from a causal perspective.\n\n\n\n***Other questions***\n\n**The sentence spanning from line 81 to 86 is very long and it’s easy to get lost in it. Consider splitting this into 2/3 sentences.** and **Figures and annotations on page 8 are very small (annotations are barely legible).**\n\nThank you for the feedback. We have implemented your helpful suggestions to modify lines 81-86 and updated the figures and annotations on page 8.\n\n**Could you highlight the single point from Figure 1 (a) also in panel (b)? To illustrate that in (b) you essentially repeat the experiment over all test samples.**\n\nThank you for your excellent suggestion. In addition to your suggestions, we have sampled two more data points. Consequently, we have displayed both the non-backtracking counterfactual values and our natural counterfactual values for three data points. Specifically, we randomly select three data points that require backtracking in our natural counterfactuals. Moreover, when a hard intervention is feasible, our natural counterfactuals and non-backtracking counterfactuals yield the same results, so we do not sample these cases. The updated PDF file with the related figures has been uploaded under the global response.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely thank all the reviewers for their time dedicated to reviewing this paper and their valuable comments. We are encouraged by and grateful for the comments that say our paper was described as  ``"well-written"``  (DQof) and ``"interesting"`` (fYWn), and was considered to``"make a strong contribution"`` (fYWn). Additionally, we appreciate the recognition of our approach as ``"a principled and practical method"`` with a ``"clear mathematical structure"`` (DQof), and as ``"a very good one"`` (v2As) and ``"interesting"`` (fYWn).
The acknowledgment of our empirical results demonstrating ``"convincing experiments"`` (DQof) and ``"improved performance"`` (fYWn) is particularly exciting.

We have carefully responded to all important questions and hope to properly address any remaining concerns.

---------------

***Below are additional responses to questions 6 and 7, as well as minor issues, for Reviewer v2As.***

**6: Th4.1 is described as being about the identifiability of counterfactuals, but it is not, because it already assumes knowledge of do(C=c\*), and that is part of what needs to be identified. Furthermore, this ignores the earlier point that an LBF need not exist, and thus we are only guaranteed to identify counterfactuals in the case that it does.**

Thank you for this valuable comment. When discussing the identifiability of natural counterfactuals, we assume that a natural counterfactual exists, meaning that $C$ has at least one solution. Therefore, we do not address the distance criterion for $do(C=c^*)$ and the solutions of $C$ directly. Theoretically, $C$ may have one or more identifiable solutions, or no solution at all, and this can always be identified. For multiple solutions, we sample one value from these solutions, and its counterfactual is identifiable according to Theorem 4.1.

We have clarified this in our updated manuscript.

**7: I did not have time to closely examine the experiments, except for the following: all the results are only about those counterfactuals which happen to be in the scope of the natural counterfactuals, and the others are excluded. Yet it seems crucial to know how many were excluded in this manner in order to evaluate how useful in practice this method is, so this should be reported as well.**

We present this result in Table 6 on Page 14, where we report the frequency of unfeasible solutions per 10,000 instances in the MorphoMNIST dataset. The data reveals a consistent trend: as the value of $\epsilon$ increases, the frequency of unfeasible solutions also rises. This occurs because a higher $\epsilon$ corresponds to a stricter standard of naturalness. In theory, when $\epsilon = 0$ (i.e., we only require that counterfactuals are within the distribution support), there are no unfeasible solutions.

| $\epsilon$ | Unfeasible Solutions |
|:------------:|:---------------------:|
|     1e-4     |          794          |
|     1e-3     |          975          |
|     1e-2     |         1166          |



***Minor issues***

**81: "a most recent paper" I’d say a more accurate description is that [30] was the first paper to formally introduce backtracking counterfactuals within the causal models framework, and thus the current paper builds on that one. (Also, the usefulness for counterfactual explanations is also something that is explicitly discussed in [30].)**

Thanks for this suggestion. We will add an accurate description of [30], as the first paper to formally introduce (fully) backtracking counterfactuals within the SCM framework. At the same time, we wish to add that our work differs significantly in both the motivation and the approach. We aim to find feasible interventions to make counterfactuals more realistic by intervening on endogenous variables. In contrast, [30] assumes a known exogenous distribution and always intervenes on exogenous variables. We explain the key differences between our work and [30] in Section E2 of the Appendix.

**100: So the paper is limited to assuming independent and unique exogenous variables. Note that this is not the case in [30].**

We completely agree. One purpose of using the independence assumption is to ensure the identifiability of natural counterfactuals when we only have access to endogenous variables. If we also assume a known distribution of exogenous variables, our work can be generalized.


**121: "In this paper..." Why? The generalization seems trivial.** and **221: Why restrict to a single variable A all of a sudden?**

We agree. $A$ could be a set of variables. In fact, in our experiment on the 3DIdentBox dataset, we perform interventions on a set of three variables. In the paper, for the sake of simplicity, we initially assume $A$ is a single variable. We have modified our presentation to explicitly state that $A$ can be a single variable or a set of variables.

**175: This assumes that all variables are real-valued. Yet later it is mentioned that not all variables need to be at the same scale. Isn't that inconsistent?**


We meant to say that all variables are continuous, and scale refers to the range of support. For example, $V_1$ follows a normal distribution with a range of $(-\infty, \infty)$. $V_2$ follows a uniform distribution over the interval $[0, 1]$. We have clarified this point in our updated manuscript.


**Th4.1: Doesn't the independence of U_i and Pa_i already follow from the independence of the exogenous variables?**

That is true. We have updated the presentation accordingly. Thank you.

**Th4.1: Why is there no mention of the distance criterion for do(C=c\*)?**

Please see our response to Q6 above. We have clarified it in the updated manuscript.

Meta-review: The reviewers and authors held a vigorous discussion. Overall the consensus is that the paper presents a novel approach to an important problem. The approach is especially promising in the era of big generative model, as the problem of generating "natural counterfactuals" comes up in many applications of such models. During the discussion multiple comments were made that would improve the paper, including its motivation and presentation - I trust the authors will follow through on these in the finalized version.
Decision: Accept (poster)