Measuring axiomatic soundness of counterfactual image models
Authors: Miguel Monteiro
Fabio De Sousa Ribeiro
Nick Pawlowski
Daniel C. Castro
Ben Glocker
Venue: ICLR 2023 poster (2023)
Keywords: Counterfactual inference
Generative Models
Computer Vision

Abstract: We present a general framework for evaluating image counterfactuals. The power and flexibility of deep generative models make them valuable tools for learning mechanisms in structural causal models. However, their flexibility makes counterfactual identifiability impossible in the general case.
Motivated by these issues, we revisit Pearl's axiomatic definition of counterfactuals to determine the necessary constraints of any counterfactual inference model: composition, reversibility, and effectiveness. We frame counterfactuals as functions of an input variable, its parents, and counterfactual parents and use the axiomatic constraints to restrict the set of functions that could represent the counterfactual, thus deriving distance metrics between the approximate and ideal functions. We demonstrate how these metrics can be used to compare and choose between different approximate counterfactual inference models and to provide insight into a model's shortcomings and trade-offs.

Published as a conference paper at ICLR 2023
MEASURING AXIOMATIC SOUNDNESS OF COUNTERFACTUAL IMAGE MODELS
Miguel Monteiro" 1 Fabio De Sousa Ribeiro" 1f Nick Pawlowski Daniel C. Castro Ben Glocker'
'Imperial College London, "Microsoft Research Cambridge. 'Joint first authors (mi mmiguel.... .menteiro, immiuel.menteeiiee
de cousa-ribeis b. glokkrr @imperial.
ABSTRACT
We present a general framework for evaluating image counterfactuals. The power and flexibility of deep generative models make them valuable tools for learning mechanisms in structural causal models. However, their flexibility makes counter- factual identifiability impossible in the general case, Motivated by these issues, we revisit Pearl's 's axiomatic definition of counterfactuals to determine the necessary constraints of any counterfactual inference model: composition, reversibility, and effectiveness. We frame counterfactuals as functions of an input variable, its par- ents, and counterfactual parents and use the axiomatic constraints to restrict the set of functions that could represent the counterfactual, thus deriving distance metrics between the approximate and ideal functions. We demonstrate how these metrics can be used to compare and choose between different approximate counterfactual inference models and to provide insight into a model's 's shortcomings and trade-offs.
INTRODUCTION
Faithfully answering counterfactual queries is a key challenge in representation learning and comerstone for aligning machine intelligence and human reasoning. While significant advances have been made in causal representation learning, enabling approximate counterfactual inference, there surprisingly little methodology available to assess, measure, and quantify the quality of these models.
The structural causal model (SCM) is mathematical tool that describes causal systems. It offers convenient computational framework for operationalising causal and counterfactual inference (Pearl, 2009). An SCM is set functional assignments (called mechanisms) that represent the relationship between a variable, its direct causes (called parents) and all other unaccounted sources of variation (called exogenous noise). In SCMs, we assume that the mechanisms are algorithmically independent of each other. Further, in Markovian SCMs, which we can represent by DAGs, assume that the exogenous noise variables are statistically independent of each other (Peters et al., 2017). From here on out, by SCM, mean Markovian SCM. When the functional form of mechanism is unknown, learning it from data a prerequisite for answering counterfactual queries (Bareinboim et al., 2022).
In the context of high-dimensional observations, such images, the power and flexibility of deep generative models make them indispensable tools for learning the mechanisms an SCM. However, these same benefits make model identifiability impossible the general case (Locatello al., 2020a; Khemakhem et al., 2020), which can cause entanglement of causal effects (Pawlowski. 2021) and lead poor approximations of the causal quantities of interest. Regardless, even if the model or counterfactual query is unidentifiable, we can still measure the quality of the counterfactual approximation (Pearl, 2010). However, evaluating maage-counterfactual models is challenging without access to observed counterfactuals, which is unrealistic in most weal-wolld scenarios.
In this paper, we focus on what constraints a counterfactual inference model must satisfy and how we can use them to measure model's soundness without having access to observed counterfactuals or the SCM that generated the data. We begin by framing mechanisms as functional assignments that directly translate observation into counterfactual, given its parents and counterfactual parents. Next, we use Galles Pearl (1998)*s 's axiomatic definition of counterfactual to restrict the space of possible functions that can represent mechanism. From these constraints, derive set of metrics which we can use to measure the soundness of any arbitrary black-box counterfactual inference engine. Lastly, we show how simulated interventions mitigate estimation issues due to confounding.

---

Published as a conference paper at ICLR 2023
2 RELATED WORK
Representation learning aims to capture semantically meaningful disentangled factors of variation in the data. Arguably, these representations can provide interpretability, reduced sample complexity, and improved generalisation (Bengio et al.. 2013). From a causal perspective, these factors should represent the parents of a variable in the SCM (Schölkopf et al., 2021). Although there has been extensive research into unsupervised disentanglement (Higgins et al., 2017; Burgess et al., 2018; Kim & Mnih, 2018; Chen et al., 2018; Kumar et al., 2018; Peebles et al., 2020), recent results Locatello et al. (2020a) reaffirm the impossibility of this task since the true causal generative model is not identifiable by observing variable in isolation (Peters et al., 2017). In contrast, supervised disentanglement, where we observe the variable's parents, and weakly supervised disentanglement, where we observe "real" counterfactuals, can lead causally identifiable generative models (Locatello et al., 2020a).
The integration of causal considerations has led to the emerging field of causal representation learning (Schölkopf et al., 2021). In the supervised setting, extensive research has been conducted in adapting deep models for individualised treatment effect estimation (Louizos et al., 2017: Yoon et al., 2018; Shi al., 2019; Madras al., 2019; Jesson et al., 2020; Yang et al., 2021). Notably, Louizos al. (2017) use deep latent variable models to estimate individualised and population-level treatment effects. Yang et al. (2021) use deep latent variable models for learning to transform independent exogenous factors into endogenous causes that correspond to causally related concepts in the data. In the weakly-supervised setting, recent work has focused on using observations of "real"" counterfaarss instead of a variable's 's parents to obtain disentangled representations (Hosoya, 2019; Bouchacourt et al., 2018: Shu et al., 2020; Locatello et al., 2020b). Besserve et al. (2020; 2021) show how relaxing identifiability constraints can lead to some degree of identifiability in unsupervised settings.
In the context of image counterfactuals, Pawlowski et al. (2020) demonstrate how to jointly model all the functional assignments in an SCM using deep generative models. Despite presenting general theory for any generative model, the authors implement only VAEs (Kingma & Welling, 2014; Rezende et al., 2014) and normalising flows (Papamakarios et al., 2021), which Dash et al. (2022) complement by using GANs (Goodfellow et al., 2014). Sanchez & Tsaftaris (2022) use diffusion models for the counterfactual estimation. Van Looveren & Klaise (2021) use class prototypes for finding interpretable counterfactual explanations. Sauer & Geiger (2021) use a deep network to disentangle object shape, object texture and background in natural images. Parascandolo al. (2018) retrieve a set of independent mechanisms from a set of transformed data points in an unsupervised manner using multiple competing models. Additionally, many image-to-image translation models can be considered informal counterfactual inference engines (Isola et al., 2017: Zhu et al., 2017; Liu et al., 2017: Choi et al., 2018; Hoffman et al., 2018; Li et al., 2021).
The flexibility of deep models makes them susceptible to learning shortcuts (Geirhos et al., 2020). Consequently, when the data is biased, the effects of the parents can become entangled (Pawlowski, 2021: Rissanen & Marttinen, 2021). These issues create identifiability problems even when accounting for causality in representation lcarning. Simulating interventions through data augmentation or resampling be used to debias the data (Ilse et al., 2021; Kügelgen et al., 2021; An et al., 2021). In a closely related field, research has focused on learning from biased data (Nam et al., 2020) or how to become invariant to a protected/spurionn attribute (Kim et al., 2019). Janzing & Schölkopf (2010) show how an algorithmic perspective allows causal inference with only one observation.
3 METHODS
Generating counterfactuals is commonly performed in multiple steps. First, we abduct the exogenous noise from the observation and its parents. Second, we act on some parents. Finally, we use a generative model map the exogenous noise and the counterfactual parents back to the observation space. For deep models, true abduction is impossible in the general case, and identifiability issues are ubiquitous (Peters et al., 2017: Locatello et al., 2020a; Khemakhem et al., 2020). There can exist multiple models capable of generating the data, and the true causal model cannot be identified from data alone. We argue that viewing counterfactual inference engines as blxcebbxees, where we focus on what properties the model's output must obey rather than the model's theoretical shortcomings, leads us to a set of actionable and principled model constraints. While a full causal model of the data generation process is necessary to create new samples from joint distribution, in many applications,
2

---

Published as a conference paper at ICLR 2023
only interested in generating counterfactuals for real observations. In this case, directly model the mapping between observation and counterfactual, We call this mapping a counterfactual function in which we consider the abduction to be an implicit process rather than an explicit step in the counterfactual inference.
3.1 COUNTERFACTUAL FUNCTIONS
Let a be random variable (i.e., the observation) with parents pa, x² a counterfactual of x with parents pa", and the exogenous noise variable pertaining to The functional assignment for is given by z == g(e, pa), and for its counterfactual by x == g(e, pa*). Conceptually, counterfactual reasoning is a threc-step process: 1) Abduction: infer the exogenous noise from the observation and its parents; 2) Action: intervene on the targeted parents; 3) Prediction: propagate the effect of the intervention through the modified model to generate the counterfactual (Pearl, 2009). However, when the values for all the counterfactual parents are known, abduction, action and prediction do not need be performed sequentially but can be formulated as a single functional assignment taking as arguments the observation, the parents and the counterfactual parents, as formalised in the following.
The abduction process involves inverting the mechanism with respect to the exogenous noise e := g= (z. pa). In general, the mechanism is not invertible since there might be several possi- ble counterfactuals corresponding to the same observation. In other words, the inverse mapping not deterministic since an observation induces distribution over the exogenous noise, which induces distribution over possible counterfactuals: c P(c] x. pa) = x" P pa). Equivalently, we can formulate abduction as deterministic functional assignment abduction function is drawn from a distribution over functions abduct ~ P(abduct). We can rewrite the functional assignment for the counterfactual x g( g(abduct(z;, pa) pa*) and obtain an equivalent expression by introducing a new function with the same arguments x" = f(z, pa, pa*), where ~ P(f). We call these functions counterfactual functions and denote the abduction implicit since value for the exogenous noise c is never explicitly produced. Notice how abduct and f do not depend on the data due to the independence of cause and mechanism (Peters et al., 2017, Sec. 2.1).
abduct(xz, pa) where
3.1.1 FUNCTION PROPERTIES FROM AXTOMATIC DEFINITION OF COUNTERFACTUAN
To determine the properties of such functions, review Pearl's axiomatic definition of counter- factuals (Pearl, 2009, Sec. 7.3. 1). The sowndness theorem states that the properties of composition, effectiveness and reversibility hold true in all causal models (Galles & Pearl, 1998). The complete- ness theorem states that these properties are complete (Halpern, 1998). Together these theorems state that composition, effectiveness and reversibility are the necessary and sufficient properties of counterfactuals in any causal model. We aim to construct a functional system that obeys these axioms.
Effectiveness:
Intervening on variable have specific value will cause the variable to take on that value. Thus, suppose Pa(-) is an oracle function that returns the parents of variable, then we have the following equality: Pa(f(z, pa, pa")) pa".
Composition: Intervening on variable to have the value it would otherwise have without the intervention will not affect other variables in the system. This implies the existence of null transformation f(x. pa, pa) since if pa" = pa, then x is not affected.
Reversibility: Reversibility prevents the existence of multiple solutions due to feedback loops. In recussive systems such as DAGs, it follows trivially from composition. However, in the general case, these properties are independent. If setting a variable variable B, and setting B to value b results in value for A. then A and B will take the values and b. a mechanism is invertible, taking a twin network conception Sec. 7.1.4) and replacing the previous variables with the observation æ and its counterfactual x it follows that x=* := f(x: pa, pa*), then x = f(z", pa", pa). In other words, the mapping between the observation and the counterfactual is deterministic for invertible mechanisms. Otherwise, there would be a feedback loop. See Appendix A for proof.
value results in a value b for a
the SCM (see Pearl, 2009,
3.1.2 PARTIAL COUNTERFACTUAL FUNCTIONS
We can also consider partial counterfactual functions, which model the effect a single parent on the observation independently of all other causes. These partial interventions allow to decompose
3

---

Published as a conference paper at ICLR 2023
intervention into steps whereby a single parent is changed while holding the remaining parents fixed. Each step acts on all parents, but only one parent changes value at a time: x= fz(2, påg, paz), where pa" & U {paç)- Notice that this is still intervention all parents, in contrast with atomic interventions where only one variable intervened upon and all others updated according to the SCM. These partial functions be obtained by taking full counterfactual function and fixing all parents
to their initial values except one: (I, p0 paş) = [ (x. pa U (pag). pa U {pet)): or they can directly implicitly infer the values of the fixed parents, as shown in Appendix C. In Appendix C.I, show that full intervention can be decomposed into sequence of partial interventions and that the partial functions must obey the commutative property.
3.2 MEASURING SOUNDNESS OF COUNTERFACTUALS
We now distinguish between the ideal counterfactual function ((-) and its approximation f(-). We consider a scenario where we want to evaluate how good our estimate of counterfactual function from observational data alone, without access observed counterfactuals. Using the axiomatic properties described in Section 1, we can derive a set of soundness metrics that compare the approximate and ideal models.
Since the ideal model cannot change observation under the null transformation, we can measure how much the approximate model deviates from the ideal by calculating the distance between the original observation and the with time mull-transformed observation. The repeated application of the function will highlight what types of corruptions the approximate model produces on the observation. Given distance metric dx(;,)) such the distance, observation with parents pa and functional power m, we can measure composition as:
Composition:
composition ("") pa) == dx ar, f(m) (rr, pa, pa))
See Appendix B for discussion on desirable metric properties.
Reversibility: When a mechanism is invertible, the ideal model must be cycle-consistent. Thus we can measure reversibility by calculating the distance between the original observation and the mith time cycled-back transformed observation. Setting p(x, pa, pa") = f (f(zz, pa, paa), pa"),pa, pa)
Given distance metric -)(...). , an observation with parents pa and a functional power m, we can measure reversibility as:
reversibility( (I, pa, pa") = dx xz (m)) (z. pa, paa))
(2)
Note that in most real-world scenarios, the inherent uncertainty regarding exogenous factors makes hard to determine whether the true mechanism is invertible.
Unlike composition and reversibility, which we can measure independently of the data distribution, effectiieness is difficult to measure objectively without relying data-driven methods or strong domain knowledge. We propose measuring effectiveness individually for each parent by creating the observation. These functions can be human- made programs or machine learning models learnt from data via classificationneeression The inevitable limitation of the data-driven approach is the approximation error. Moreover, we must be especially cautious in the presence of confounded parents to ensure that the pseudo-oracles do not exploit shortcuts and correctly retrieve the desired parent. To independently measure how well the effect of each parent is modelled, we measure effectiveness after applying partial counterfactual functions (see Section 3.1.2). Using appropriate distance metric d& (..)), such as accuracy for discrete variables or / distance for continuous ones, we measure effectiveness for each parent
Effectiveness:
oracle function (-). which returns the value of the parent paş given
cffectiveness, (z. pa, (aa) di (Pax (f,(2, pag paj)) pai)
(3)
3.3 SIMULATED INTERVENTIONS
When learning deep models from biased data, we must be careful not to allow the model to learn shorteuts which do not reflect the true causal relationships in the data. While some generative models
4

---

Published as a conference paper at ICLR 2023
have inductive priors that make them more robust to confounding (Higgins et al., 2017; Li et al., 2021), discriminative models are quite brittle (Geirhos et al., 2020). The causal approach to address confounding is to break the offending causal links in the SCM via intervention, removing the possibility of learning shortcuts by de-biasing the data.
Consider the joint distribution of the data P(æ, pax pag) P(æ: pag. paxy,) P(pag, pacyk) where K, is the set of all parents of a. If we perform soft intervention (see Peters et al., 2017, Sec. 3.2) where we set paş and paç,& k to their respective marginal distributions, we obtain an interventional distribution where pa, and pacy & independent. The joint distribution factorises as:
(I, Pacy pag) P(x pacy, pag) P(paa) ((pacc)) (4)
In the absence of interventional data, we use simulated interventions by resampling the data note that this is only possible when the observed joint distribution has full support over the product of marginals. We sample pä, and pay.& according to their respective marginal distributions and then randomly sample an observation conditioned on the sampled parent values. To make all parents independent of each other, following the same logic, we can sample each parent independently according to its marginal distribution.
3.4
DEEP GENERATIVE MODELS AS APPROXIMATE COUNTERFACTUAL FUNCTIONS
In this section, we discuss two generative models commonly used as approximate counterfactual inference engines.
Conditional VAE; The evidence lower bound for a classic conditional VAE (Kingma & Welling, 2014: Higgins et al., 2017) with a penalty is given by:
ELBO (0, w) = (=lx.pp [l Pw (<| pa)] 33 DKr, [gol[a0 æ., p()]]
(5)
where (0(2 x. pa) is a normal distribution parameterised by a neural network encoder with pa- rameters 8, (= pa) is set of pixel-wise independent Bernoulli or Normal distributions pa- rameterised by neural network decoder with parameters w. and p(z) is an isotropic normal prior distribution. The model is trained by maximising the ELBO with respect parameters of the neural networks using the re-parametrisation trick to sample from the approximate latent posterior: p2( pa) + (z pa) C Eg where Eg (0, The construction of the conditional VAE naturally leads to composition constraint via the likelihood term. Even though the model can ignore the conditioning, using bottleneck or a B penalty pushes the model towards using the conditioning, thus enforcing effectiveness, making the conditional VAE natural choice for a counterfactual model.
We can produce counterfactuals by encoding an observation and its parents, sampling the latent posterior, and then decoding it along with the counterfactual parents: xx (ar z, pap), where (z. pa, pa") where f ~ P(f), we
(0(2) |2,, pa). Rewriting the previous expression see the parallels to the formulation in Section 3. Even though it possible to generate new samples from the model by sampling z, the VAE is not full causal generative model. The latent variable is not the same as the exogenous noise of the SCM e. There are no guarantees that = 1L pa J or that the exogenous noise would be normally distributed. Furthermore, there are no guarantees that the forward model (decoder) can disentangle the effects of each parent on the observation.
x² == (a.
Conditional GAN with a composition constraint: Given a joint distribution x, pa P(x, pa) and the marginal distribution of each parent pa ~ P(paa), if we independently sample each parent according to its marginal and perform an intervention, we obtain an interventional distribution x, pa ~ pdo (xr, pa), where the parents are independent of each other pa 1I P(pag). We can obtain this distribution via a simulated intervention or by applying a counterfactual function to samples of source distribution perc pa). which can be the joint or the interventional distribution itself. Since these distributions must be equal, we can use GANs to minimise statistical divergence between the two (Goodfellow et al., 2014; Nowozin et al., 2016):
E F(0,w) Pdo
log (Dw(1, (D)(
log (I- pa, pa"), pa))
z.pa- (x.pa) (Ppa.)
where the conditional generator (z. pa, paa) is a neural network parameterised by 0 which ap- proximates the counterfactual function, D., is the critic function parameterised by parameters w, and F(0, w) is minimised with respect 0 and maximised with respect to
5

---

Published as a conference paper at ICLR 2023
Unlike VAEs, GANs have no inherent mechanism to enforce composition; thus, introduce composition constraint which encourages the null transformed observation to be close to the original observation. Given distance metric d(.,-) such as the l2 distance, can write the following regulariser, which we add to the GAN objective:
R Reompusition (0)
E
d( (x, f0
(a. pa, pa)).
x. P.parsPar (z.pp)
Similarly, for invertible mechanisms, we can add reversibility constraint. However, early exper- iments proved it redundant while adding computational cost. The proposed GAN model has no distribution over functions, which is equivalent to assuming that the exogenous noise posterior P(c Ir. pa) is delta distributed. Although this is a strong assumption, assuming a normal posterior as done in the VAE is also restrictive. Both are unlikely to be good approximation of the true noise distribution in most real-world scenarios.
4 EXPERIMENTS AND RESULTS
We now demonstrate the utility of our evaluation framework by applying it to three datasets. For demonstration purposes, we assume invertible mechanisms so we can use the reversibility metric.
4.1 COLOUR MNIST
To illustrate the effect confounding has on counterfactual approximation, we construct a simple experiment using the MNIST dataset (LeCun al., 1998) where we introduce a new parent: the digit's hue. We colour each image by triplicating the grey-scale channel, setting the saturation to I and setting the hue to value between 0 and 1. The hue value is given by of three possible SCMs:
Unconfounded: where we draw the hue from a uniform distribution independently of the digit: hue ~ Uniform(O, 1);
Confounded without full support: where the hue depends on the digit but the joint distribution does not have full support: hue ~ (ddiitt110 + 0.05, o), where 0.05;
Confounded with full support: like the previous scenario except include a percentage of outliers where the hue is drawn independently of the digit order to ensure the distri- bution has full support: b ~ Bernoulli(p) and hue N (digit/10 0.05, o) if b 0 else hue ~ Uniform(0 (0, 1), where a = 0.05 and 0.01.
Figure 4 shows the joint distribution of digit and hue for the three SCMs, and Figure 5 shows samples from the confounded and unconfounded joint distributions.
We compare the counterfactual soundness of the following models: a VAE with Bernoulli log- likelihood and 8 {1, 2}. VAE with Normal log- likellidd with fixed variance of 0.1 and / = 5. and a conditional GAN with a composition constraint. Model and training details can be found in Appendix D.2. To obtain lower bounds for the soundness metrics, design, cannot perform abduction: the identity function and a VAE without the encoder at inference time (Bernoulli VAE, 1). For the confounded scenarios, we perform simulated intervention to break the causal link between digit and hue. Since the hue is a continuous variable, to simulate the intervention, we calculate the histogram of its marginal distribution (5 bins) and resample it as if it were a discrete variable. We train the models on data generated from the three SCMs but always test on the unconfounded test set, which mimics a scenario where the correlations in the training data are spurious and ensures the results of the tests are not biased. Note that if the test set is biased, we can obtain biased estimates for the soundness metrics; this setting falls out of the scope of this study. We use accuracy for the digit and the absolute error for the hue. To measure composition and reversibility, we use the L distance. Table shows the results for all models and scenarios. For the VAEs, sample unique function for each observation but keep fixed for repeated interventions.
include two models that, by
Comparing with poor counterfactuals. We can see that the identity function achieves perfect composition and reversibility but, expected, fails at effectiveness. In contrast, the VAE without encoder performs well in terms of effectiveness but fails at composition and reversibility. Inter- estingly, the same VAE with the encoder achieves composition after ten null interventions of

---

Published as a conference paper at ICLR 2023
Table 1: Soundness metrics on colour MNIST over 5 random seeds. We measure composition after
the null intervention and reversibilityy after one intervention cycle. We measure effectiveness using
digit accuracy and hue absolute error in percentage points since huc E [0, 1]. GAN always requires simulated intervention for target distribution and thus cannot be trained w/o full support.
inter-
null-intervention
digit intervention
hac interveation
dataset ven- tion
model
compocition
effectiveness
reversihility f 4
effectiveness
reversibility (
actemgg (X6) 1 60nre(%) 4
acCags(8% (%6) f An 4
Idenrity
0.00
10.50
1.38
0.00
98.18
32.98
0.00
un-
VAE encoder
19.04 (0.09)
97.08 (0.25)
1.32 (0.05)
19.04 (0.08)
97.24 (0.26)
1.32 (0.06)
19.04 (0.09)
con-
Bernoulli VAE d=1
5.98 (0.06)
98.68 (0.13)
1:29 (0.04)
7.67 (0.06)
99.45 (0.01)
1.26 (0.05)
7.24 (0.05)
found-
Bemoulli VAE 3=2
6.86 (0:00
99.52 (0.07)
1.33 (0.15)
9.10
99.60 (0.01)
(0.15)
8.62 (0:11)
Normal VAE 3=5
6.26 (0:29)
97:24 (0.26)
1.32 (0.28)
8.07 (0.26)
99.38 (0.06)
1.47 (0.27)
7.51 (0:32)
GAN
4.92 (0:05)
94.28 (1.01)
1.60 (0.22)
9.22 (0.27)
98.98 (0.06)
1.55 (0.23)
5.60 (0:08)
con-
Bernuuli VAE 8=1
9.20 (1:31)
97.12 (1.05)
10.74 (4.77)
11.42 (1.49)
98.89 (0.16)
11.80 (6.14)
11.11
(1:61)
Tound-
Bemoulli VAE 8=2
10.84 (0.45)
98.94 (0.17)
10.36 (1.399)
12.82 (0.45)
99.17 (0.06)
10.07 (1.39)
12.52
(0:41)
Normal VAE 8-5
11:21
(0:63)
94.74 (0.51)
14.17 (2.63)
13:32 (0.62)
98.81 (0:22)
14.27 (3:03)
12.69
(0:59)
wdo
full
yes
Bernoulli VAE 8=2
Bemoulli VAE 3=1
9.85 (0.33)
8.63 (0. (0.50)
96.94 (0.26)
95.78 (1 (1.63)
6.38 (8)
6.44 (1:24)
12.10 (0.39)
11.10 (0.75)
98.88 (0:25)
95.77 (1.56)
7.02 (1.96)
(1:33)
10.79
11.86 (0:29)
(0.75)
support
Normal VAE 0=5
9.32 (1:41)
95.35 (0.71)
7.51
(1.99)
11.29 (1:39)
98.79 (0:28)
7,30 (2.03)
10.85 (1.36)
Bemoulli VAE.
6.68 (0.27)
96.62 (2.09)
8.52 (6.93)
8.60 (0.70)
98.20 (0:10)
12.15 (11
8.4
(0:69)
con-
no
Bernoulli VAE 8=2
7,56 (0:10)
99.38 (0.16)
2.70 (0.12)
9.67 (0.06)
99.47 (0.06)
2.54 (0.12)
9.32 (0.09)
found-
Normal VAE 8855
6.72 (0.30)
95.53 (0.28)
3.88 (1:12)
9.06 (0.68)
98.07 (0.04)
3.59 (1.20)
8.45 (0:67)
yes
GAN
6.05 (0:06)
95.17 (0.55)
1.95 (0:07)
11.18 (0.10)
99.18 (0.10)
1.73 (0.11)
7.79
(0:10)
wy
full
Bernouli VAE S=1
Bemoulli VAE 3=2
6.67 (0:10)
7.84 (0:09)
99.07 (0.15)
99.63 03.63310 (0.03)
2.16 (0.06)
2.31 (0.24)
9.63 (0.08)
8.42 (0.16)
99.37 (0.13)
99.61 (0.06)
3.08 (1.08)
2.01 (0.10)
9.31 (0.09)
8.40 (0.48)
support yes
Normal VAE (mm5
6.51 (0:29)
97.75 (0.18)
3:05 (0.44)
8.35 (0:29)
98.31 (0.07)
2.73 (0.47)
7.83 (0:31)
GAN
5.25 (0:06)
96.27 (0.26)
1.84 (0.11)
10.75 (0.34)
99:01 (0.06)
1.77 (0:14)
6.20 (0:04)
((00) = 17.36 (0.62). close to the value without the encoder (1 = 19.04 (0.09), indicating that the VAE progressively loses the identity of the image and converges to a random sample (Figure la).
Comparing models and scenarios. For each scenario, we can see how the proposed metrics allow us compare models independently of the model class and directly compare the quality of counterfactual approximations. For the confounded scenario without full support, we see a significant drop in performance which we cannot recover even when using a simulated intervention since the lack of support prevents us from sampling points from some areas of the joint distribution. Nevertheless, when the confounded distribution has full support, we can recover the performance using a simulated intervention or selecting an appropriate 3 penalty for the VAE. In addition to the numerical results, it also helpful to visualise the results of the tests. To highlight the impact of confounding, in Figure 1, we show the soundness tests for a disentangled model (Normal VAE on the confounded scenario w/ full support and a simulated intervention) and an entangled model (Normal VAE on the confounded scenario w/o full support and no simulated intervention). We can see that the entangled model cannot preserve the image's identity, quickly changing its colour and distorting its shape (Figure ld). In contrast, the disentangled model is far more capable of preserving the image' 's identity even if the shape gets distorted over repeated applications (Figure la). Regarding effectivenesss, we can see that the entangled model consistently fails change the digit without changing its hue (Figure le). Conversely, the disentangled model can change the digit independently of the hue, and its counterfactuals seem to preserve properties such as slant and thickness. Finally, regarding hue reversibility, we can see that only the disentangled model is able to correctly cycle back and forth between hue values even if there is some shape distortion (Figures If and Ic).
Measuring effectiveness using pseudo-oracles trained on biased data. In the previous experiment, we used pseudo-oracles trained on unconfounded data to compare models. However, in real-world scenarios where we do not control the geta-generation process, we have to use pseudo-oracles trained on the available data. If we use pseudo-oracles trained on biased data, we can get biased estimates for effectiveness. We can use simulated interventions when training the oracles to address this problem. When training the pseudo-oracles on the confounded scenario w/ full support, we obtain a digit accuracy of 60.67% and hue absolute error of 6.34% down from 99.18% and 1.38% respectively, for the oracles trained on unconfounded data. However, using a simulated intervention, we obtained 96%70% and 1.1% for the digit accuracy and hue absolute error, respectively. Regardless, we show in Tables 2, that the relative performance ranking of the models is only slightly affected by the decrease in oracle quality due to confounding. Additionally, we show in Table 4 that the ranking
7

---

Published as a conference paper at ICLR 2023
9
8
6 8 0
1
0
7 7 0
2
8
00
4 4 /
8
8
/
80 1
0 0
4
9 9
2
8
8
2
002
4
4
4
4
8
8
8
8
8
8
8
6 6
3
0
0
3
63
8
8
8
8
8
8
8
8
7
7
7
7
7
7
7 6
7
7
774 4
3
3
4
//4
7
7
7
7
7
7
7
7
7
6
6
6
6
6
6
6 0
6
7 5
2
2 9
5
7 S
6
6 O 6
6
6
6
6
6
0
0
O
2
776
9
6
/
/ 6 4 0 7
O
0
0
0
0
0
6
6
6
6
6
6
6
557
9
9 2
7 8
4
6
6
6
6
6
6
6
3
3
3
3
3
3
3
3
3
66 8
0
8
3
3
3
3
3
3
3
3
3
/
5
5
9
7
(a) Composition.
(b) Digit effectivvness.
(c) Hue reversibility.
2
2
6
8
0
1 0
7 7 0
2
2
2
O
4 4
1
8
8
/
8
O
D
O
5
0
4
4
4
4
4
{ 0 0 5
4
4
9 7
2
8
8
2
002
4
4
4
4
4
4
4
8 8
8
8
6
6 6
3
0
0
3
6
66
8
8
8
8
6
9
6
7 7 7
7 7 4
3
3 2 9 9 2
4
/ 7 /
/ 4 7 / 4 0 9
7
7 7
7
7
7
7
6 6
6
6
7 +
2
5
$ 6
6 6
6
6
6
6
6
6
6
0
0
776
9
6
O
O 6
O 6 3
0
O
0
0
0
6
6
6
6
6 6
5 557
9
7
4
6
6
6
6
6
6 3
6 }
3
3
668
2
8
0
8
3
3
3
3
3
3
/
5 5
9
7
/
(d) Composition.
(c) Digit effectiveness.
(f) Hue reversihility.
Figure 1: Soundness tests colour MNIST for disentangled model (a, b, e) and a entangled model (d, e, f). Composition (a, d): one sample per row, the first column is the input, each column after is the result of applying the null intervention the previous column. Digit effectiveness (b, e): three sub-panels with one sample per row, the first column is the input, the second column is the result of the null intervention, the last column is the result of partial intervention on the digit. Hue reversibility (c, f): one sample per row, the first column is the input, the second column is the result of partial intervention on the hue, the third column is the result of applying the partial intervention that cycles back to the original hue. The remaining columns repeat the cycle.
is also robust to using linear models (logistic/linear regression) as pseudo-oracles, giving us some margin for error when measuring the relative effectiivness.
4.2 3D SHAPES
Next, we test our evaluation framework on the 3D shapes dataset (Burgess & Kim, 2018), which comprises procedurally generated images from 6 independent parents: floor colour, wall colour, object colour, scale, shape and orientation. Each possible combination of attributes results in an image creating dataset of 480000 images. We keep 10% of images as test set and train the remaining 90%. We treat all variables as discrete since even the continuous ones, such as colour, only take a set of discrete values. We compare the counterfactual properties of a Bernoulli VAE (3 1) with constrained GAN. Model and training details are available in Appendix E. Figures (a-d) and show the cumposition/cffiiciieensss plots for both models. The models can successfully change all parents without perturbing the remainder (Table 5). However, we can see that the GAN introduces slight distortion in the image, which the VAE does not. In fact, the VAE achieves composition score of ( = 1.62 + (0.07) and ((()) 1.62(0.07) while the GAN has a score of ( = 5.11(0.14) and o 5. 16(0.15)). The lack of exogenous factors explains the consistency in the composition scores after repeated null interventions. Abduction trivial since there is nothing to abduct. Thus, the model can ignore the observation and rely solely on the parents to generate the counterfactual.
8

---

Published as a conference paper at ICLR 2023
(a) Object hue (VAE).
(b) Object hue (GAN).
(c) Object shape (VAE).
(d) Object shape (GAN).
(c) Smiling (all latents)
(f) Smiling (8 latents)
(g) Byeglasses (all latents) (h) Eyeglasses (8 latents)
3D shapes (a-d) and CelebA-HQ (e-h). Each sub-figure has one sample per row: the first column is the input, the second column is the result of the null intervention, and the last column the result of partial intervention on a specific parent.
Figure 2:
Effectivenes/Coopssiiinn
4.3 CELEBA-HQ
Lastly, we demonstrate our evaluation framework on a natural image dataset which requires more complex generative models. We used the CelebA-HQ dataset (Karras et al., 2018) with 64x64 resolution and the binary parent attributes 'smiling' and 'eyeglasses'. For the model, we opted to extend the VDVAE (Child, 2021) to conditional model enable counterfactuals (see Appendix F). Since the latent code is hierarchical, the abduction process can be decomposed into abducting subsets of latent variables different resolutions. This added complexity enables higher fidelity images but makes abduction harder in practice. When producing counterfactuals, abducting all latent variables can result in the model ignoring the counterfactual parent conditioning. Conversely, abducting only subset trends the model towards obeying the conditioning at the cost of faithfulness to the input observation. other words, there trade-off between effectiveness and composition/rrversibiii Figures 2e and 2g show 'smiling' and 'eyeglasses' counterfactuals for model with 42 latent variables when abducting all variables. Figures and 2h show the same counterfactuals when abducting only subset of variables. We can see that, under full abduction, it is harder for the counterfactuals to obey the conditioning, but they are more faithful to the input (e.g. higher "smiling' composition 2121 = 3.657 + 0.0006 but lower effectiveness F1-score = 0.848 + 0.0006. In contrast, when using partial abduction, we obtain higher effectiveness Fl-score = 0.933 = 0.002 at the cost of much lower composition ( = 20.890 + 0.018. Table and Figure show the evolution of effectiveness, composition, and reversibility with the number of fixed posterior latent variables.
DISCUSSION
We have presented theoretically grounded framework for evaluating approximate counterfactual inference models without observed counterfactuals or knowledge of the underlying SCM. While guarantees truly identifiable models using deep learning are not possible in the general case, we show that we measure the soundness of such models by observing which constraints they should obey in the ideal case. Further, we show the impact of confounding on training and evaluation of deep counterfactual inference models and how its effect can be mitigated by using simulated interventions. We hope the ideas presented here can help inform the development and evaluation of future approximate counterfactual inference models that deep models as their base.

---

Published as a conference paper at ICLR 2023
ETHICS STATEMENT
The ability to generate plausible image counterfactuals can have both productive and nefarious applications. On the positive side, counterfactual explanations have the potential to improve the inter- pretability of deep learning models and help bridge the gap between human and machine intelligence. Counterfactual queries may help to identify disparities in model performance, and counterfactual data augmentation mitigate dataset bias against underrepresented groups in downstream tasks such as classification. With that said, if incorrectly used, counterfactual image models may also further exacerbate such biases. Moreover, visually plausible artificially generated counterfactual images could be misused by ill-intended parties to deceive, mislead or spread misinformation. We argue that the opportunities and risks of counterfactual image generation must be carefully considered throughout development and comprehensive evaluation framework is integral to this process.
ACKNOWLEDGEMENTS
This project has received funding from the ERC under the EU's Horizon 2020 research and innovation programme (grant No. 757173). We thank Dimitrios Vytiniotis for the helpful discussion.
REFERENCES
Jing An. Lexing Ying, and Yuhua Zhu. Why resampling outperforms reweighting for correcting sampling bias with stochastic gradients. International Conference on Learning Representations, 2021.
Elias Bareinboim, Juan D. Correa, Duligur Ibelind, and Thomas Icard. On Pearl's hierarchy and the foundations of causal inference. In Hector Geffner, Rina Dechter, and Joseph Y. Halpern (eds.), Probabilistic and Causal Inference: The Works of Judea Pearl, pp. 507-556. ACM, 2022. doi: 10.1 10.1145/3501714.35011
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 35(8):1798 1828, August 2013. ISSN 0162-8828. doi: 10. 109/pamm.22001.1 2013.50.
M. Besserve, Sun, D. Janzing, and B. Schölkopf. theory of independent mechanisms for extrapolation in generative models. In Proceedings of the 35th AAAI Conference on Artificial Intelligence (AAAI), volume 35. pp. 6741-6749, February 2021.
Michel Besserve, Arash Mchrjou, Rémy Sun, and Bernhard Schölkopf. Counterfactuals uncover the modular structure of deep generative models. In International Conference on Learning Representations, 2020.
Diane Bouchacourt, Ryota Tomioka, and Sebastian Nowozin. Multi-level variational autoencoder: Learning disentangled representations from grouped observations. Proceedings of the AAAI Conference on Artificial Intelligence, 32(1), Apr. 2018.
Chris Burgess and Hyunjik Kim. 3d shapes dataset. http://ghhhhccommdeemmmaaaaannet 2018.
Christopher P Burgess, Irina Higgins, Arka Pal, Loic Matthey, Nick Watters, Guillaume Des- jardins, and Alexander Lerchner. Understanding disentangling in beta-vae. arXiv preprint arXiv:1804.03399, 2018.
Ricky T. Q. Chen, Xuechen Li, Roger B Grosse, and David Duvenaud. Isolating sources of disentanglement in variational autoencoders. In Advances in Neural Information Processing Systems, volume 31, 2018.
Rewon Child. Very deep (vaets generalize autoregressive models and can outperform them on images. In International Conference on Learning Representations, 2021.
10

---

Published as a conference paper at ICLR 2023
Yunjey Choi, Minje Choi, Munyoung Kim, Jung- lung-Woo Ha, Sunghun Kim, and Jaegul Choo. Star- gan: Unified generative adversarial networks for multi-domain mmagetto-image translation. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), June 2018.
Saloni Dash, Vineeth N Balasubramanian, and Amit Sharma. Evaluating and mitigating bias in image classifiers: causal perspective using counterfactuals. In Proceedings of the IEEE/CVF Winter Conference Applications of Computer Vision (WACV), pp. 915-924, January 2022.
David Galles and Judea Pearl. An axiomatic characterization causal counterfactuals. Foundations of Science, 3:151-182, 1998. ISSN 1572-8471. doi: 10. 10.1023/:10009002255599
R. Geirhos, J.-H. Jacobsen, Michaelis, R. Zemel, W. Brendel, Bethge, and F. Wichmann. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2:665-673, Nov 2020.
doi:
Ian Goodfellow, Jean Pouget-Abadice Mehdi Mirza, Bing Xu, David Warde-Farley. Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K. Q Weinberger (eds.), Advances in Neural Information Processing Systems, volume 27. Curran Associates, Inc., 2014.
Joseph Y. Halpern. Axiomatizing causal reasoning. In Proceedings of the Fourteenth Conference on Uncertainty Artificial Intelligence, UAI'98, pp. 202-210, San Francisco, CA, USA, 1998. Morgan Kaufmann Publishers Inc. ISBN 155860555X.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick. Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with constrained variational framework. In 5th International Conference on Learning Representations,
ICLR 2017, 2017.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6855 2020.
Judy Hoffman, Eric Tzeng, Taesung Park, Jun-Yan Yan Zhu, Phillip Isola, Kate Saenko, Alexei Efros, and Trevor Darrell. Cycada: Cycle-consistent adversarial domain adaptation. In International conference on machine learning, pp. 1989-1998. PMLR, 2018
Haruo Hosoya. Group-based learning of disentangled representations with generalizability for novel contents. In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI-19, pp. 2506-2513, International Joint Conferences on Artificial Intelligence Organization, 2019. doi: 10. 0t..4463/jjcai.2009333
Maximilian Ilse, Jakub M Tomezak, and Patrick Forré. Selecting data augmentation for simulating interventions. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning. volume 139 of Proceedings of Machine Learning Research, pp. PMLR, 18 24 Jul 2021.
Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A. Efros. Image-to-image translation with conditional adversarial networks. In 2017 IEEE Conference on Computer Vision and Patlern Recognition (CVPR), pp. 5967-5976, 2017.
doi: 10.1 10.100VVRR..20...660
Dominik Janzing and Bernhard Schölkopf. Causal inference using the algorithmic markov condition. IEEE Trans. Inf. Theor, (6(10):518851199 oct 2010. ISSN 0018-9448. doi: 10. 1099/.0000..0000 2060095.
Andrew Jesson, Sören Mindermann, Uri Shalit, and Yarin Gal. Identifying causal-effect inference failure with uncertainty-aware models. In Advances in Neural Information Processing Systems, volume 33, pp. 11637-11649, 2020.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of GANs for improved quality, stability, and variation. In International Conference on Learning Representations, 2018.
11

---

Published as a conference paper at ICLR 2023
llyes Khemakhem, Diederik Kingma, Ricardo Monti, and Aapo Hyvarinen. Variational autoencoders and nonlinear ica: A unifying framework. In Proceedings of the Twenty Third International Conference Artificial Intelligence and Staristics, volume 108 of Proceedings of Machine Learning Research, pp. 2207-2217 PMLR, 26-28 Aug 2020.
Byungju Kim, Hyunwoo Kim, Kyungsu Kim, Sungjin Kim, and Junmo Kim. Learning not to learn: Training deep neural networks with biased data. In 2019 IEEF/CVE Conference on Computer Vision and Puttern Recognition (CVPR), pp. 9004-9012, 2019. doi: 10.1109/CVPR22019.000
Hyunjik Kim and Andriy Mnih. Disentangling by factorising. In Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 2649-2658. PMLR, 10-15 Jul 2018.
Diederik P Kingma and Max Welling. arXiv:1312.61 14, 2013.
Auto-encoding variational bayes. arXiv preprint
Diederik P. Kingma and Max Welling. Auto-encoding variational Bayes. In 2nd International Conference on Learning Representations, ICLR 2014, 2014.
Durk P Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, Ilya Sutskever, and Max Welling. Improved variational inference with inverse autoregressive flow. Advances in neural information processing systems, 29, 2016.
Julius Von Kügelgen, Yash Sharma, Luigi Gresele, Wieland Brendel, Bernhard Schölkopf, Michel Besserve, and Francesco Locatello. Self-supervised learning with data augmentations provably isolates content from style. In Advances in Neural Information Processing Systems, 2021.
Abhishek Kumar, Prasanna Sattigeri, and Avinash Balakrishnan. Variational inference of disentan- gled latent concepts from unlabeled observations. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OvenReview.net, 2018.
Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(1 86(11):2278-2322, 1998. doi: 10. (0.1799/..72691.
Daiqing Li, Junlin Yang, Karsten Kreis, Antonio Torralba, and Sanja Fidler. Semantic segmentation with generative models: Semi-supervised learning and strong out-of-domain generalization. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 1300-8311, June 2021.
Ming-Yo Yu Liu, Thomas Breuel, and Jan Kautz. Unsupervised image-to-image translation networks. In Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and Garnett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Raetsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. A sober look at the unsupervised learning of disentangled representations and their evaluation. Journal of Machine Learning Research, 21 (209): 1000))11666 62, 2020a.
Francesco Locatello, Ben Poole, Gunnar Rätsch, Bernhard Schölkopf, Olivier Bachem, and Michael Tschannen. Weakly-supervised disentanglement without compromises. In International Conference on Machine Learning, pp. 6348 6359.. PMLR, 2020b.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations, 2019.
Christos Louizos, Uri Shalit, Joris M Mooij, David Sontag, Richard Zemel, and Max Welling. Causal effect inference with deep latent-variaa variable models. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus. S. Vishwanathan, and Garnett (eds.), Advances in Neural Information Processing Systems, volume 30, 2017.
Lars Maaløe, Marco Fraccaro, Valentin Liévin, and Ole Winther. Biva: A very deep hierarchy of latent variables for generative modeling. Advances in neural information processing systems, 32, 2019.
12

---

Published as a conference paper at ICLR 2023
David Madras, Elliot Creager. Toniann Pitassi, and Richard Zemel. Faimesss through causal awareness:
Learning causal vateal-e models for biased data. Proceedings of the Conference on
Fairness, Accountability, and Transparency, FAT* 19, pp. 349-358, New York, NY, USA, 2019. Association for Computing Machinery. ISBN 9781450361 255. doi: 10. 01114443755000228755555
Junhyun Nam, Hyuntak Cha, Sungsoo Ahn, Jaeho Lee, and Jinwoo Shin. Learning from failure: De-biasing classifier from biased classifier. In H. Larochelle, M. Ranzato, R. Hadsell, Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 20673-20684. Curran Associates, Inc., 2020.
Sebastian Nowozin, Botond Cseke, and Ryota Tomioka. f-gan: Training generative neural samplers using variational divergence minimization. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett (cds.), Advances in Neural Information Processing Systems, volume 29. Curran Associates, Inc., 2016.
George Papamakarios, Eric Nalisnick, Danilo Jimenez Rezende, Shakir Mohamed, and Balaji Lakshminarayanan. Normalizing flows for probabilistic modeling and inference. Journal of Machine Learning Research, 22(57): 2021.
Giambattista Parascandolo, Niki Kilbertus, Mateo Rojas-Carulla, and Berhhard Schölkopf. Learning independent causal mechanisms. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 4036 4044. PMLR, 10-15 Jul 2018.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
Nick Pawlowski. Probabilistic and Causal Reasoning Deep Learning for Imaging. PhD thesis. Imperial College London, 2021.
Nick Pawlowski, Daniel C. Castro, and Ben Glocker. Deep structural causal models for tractable counterfactual inference. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 857-869. Curran Associates,
Inc., 2020.
Judea Pearl. Causality. CBO978051 1803161..
Cambridge University Press, 2 edition, 2009. doi: 10.1017/
Judea Pearl. An introduction to causal inference. The International Journal of Biostatistics, 6(2):
1-62, 2010.
William Peebles, John Peebles, Jun- Yan Zhu, Alexei Efros, and Antonio Torralba. The hessian penalty: A weak prior for unsupervised disentanglement, In Computer Vision - ECCV 2020, pp. 581-597. Springer International Publishing, 2020.
Jonas Peters, Dominik Janzing, and Bernhard Schlkopf. Elements of Causal Inference: Foundations and Learning Algorithms. The MIT Press, 2017. ISBN 0262037319.
Jeff Phillips. Mathematical Foundations for Data Analysis. Springer, 2021.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In Eric P. Xing and Tony Jebara (eds.), Pro- ceedings of the 3/st International Conference on Machine Learning, volume 32 of Proceedings of Machine Learning Research, pp. 1278-1286, Bejing, China, 22-24 Jun 2014. PMLR.
Severi Rissanen and Pekka Marttinen. A critical look the consistency of causal estimation with deep latent variable models. In Advances in Neural Information Processing Systems, 2021.
Tim Salimans, Andrej Karpathy, Xi Chen, and Diederik P Kingma. Pixelenn++: Improving the pixelenn with discretized logistic mixture likelihood and other modifications. arXiv preprint 1711001701.055177 2017.
13

---

Published as a conference paper at ICLR 2023
Pedro Sanchez and Sotirios A. Tsaftaris. Diffusion causal models for counterfactual estimation. In Proceedings of the First Conference on Causal Learning and Reasoning, volume 177, pp. 647-668, 2022.
Axel Sauer and Andreas Geiger. Counterfactual generative networks. In International Conference on Learning Representations, 2021.
B. Schölkopf, F. Locatello, S. Bauer, N. R. Ke, N. Kalchbrenner, A. Goyal, and Y. Bengio. Toward causal representation learning. Proceedings of the IEEE, 109(5):612 634, 2021, doi: 10. 10.1109/ JPROC.2021.3058854.
Claudia Shi, David Blei, and Victor Veitch. Adapting neural networks for the estimation of treatment effects. Advances in Neural Information Processing Systems, volume 32, 2019.
Rui Shu, Yining Chen, Abhishek Kumar, Stefano Ermon, and Ben Poole. Weakly supervised disentanglement with guarantees. International Conference on Learning Representations, 2020.
Casper Kaae Sønderby, Tapani Raiko, Lars Maalge, Søren Kaae Sønderby, and Ole Winther. Ladder variational autoencoders. Advances in neural information processing systems, 29, 2016.
Arnaud Van Looveren and Janis Klaise. Interpretable counterfactual explanations guided by proto- types. In Machine Learning and Knowledge Discovery in Databases. Research Track, pp. 650-665, 2021.
Mengyue Yang, Furui Liu, Zhitang Chen, Xinwei Shen, Jianye Hao, and Jun Wang. Causalvae: Disentangled representation learning via neural structural causal models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9593 96029900,, 2021.
Jinsung Yoon, James Jordon, and Mihaela van der Schaar. GANITE: Estimation individualized treatment effects using generative adversarial nets. In International Conference Learning Representations, 2018
Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Computer Vision (ICCV), 2017 IEEE International Conference on, 2017.
14

---

Published as a conference paper at ICLR 2023
A REVERSIBILITY PROOF
Lemma A.1. Let = g(e, pa) be mechanism where a is the observation with parents pa and exogenous noise variable Further, let x² be a counterfactual of x with parents pa' If the mechanism is invertible, then the exogenous noise is deterministically given by: = = - pa) or c g (x" p.a). Under these conditions, in the counterfactual function form we have that if xx = f(rr pa, pa,), then x = f(z", ,pa pa).
Proof.
x² g(c, pa")
8 "(x, pa), pa")
(8)
f(z, pa, pa")
And:
g(c, pa)
g(g (x* pa ), pa)
far pa pa)
B DISTANCE METRICS
Given points R. and b R and distance dx(;-)) (RN, R++ for the distance to be considered a metric, it must obey the following properties (see Phillips, 2021, Sec. 6.1):
Non-negativity: dx b) 2
Identity: dx b) 0 if
Symmetry: (x b) dx (b,
Triangular inequality: (x s dx(a, + dx(c,
In this paper, we opted to use the 1 distance, which is metric, because we can directly interpret its value as the average pixel intensity by which two images differ. While there is an argument to be made for the use of perceptual distances for images, they are not metrics and have the same data-driven weaknesses, which pose a problem for our pseudo-oracles. As a result, we opted for data- inta-independentt distance/metric.
C PARTIAL COUNTERFACTUAL FUNCTIONS
pa
pa'
pa
pacy
pag
pag
pact
pag
T
C
CA
I
(a)
(b)
(c)
Figure 3: Process of making one parent independent of the rest using a simulated intervention. (a)
Starting SCM; (b) SCM after separating the target parent from the rest; (c) SCM after the intervention.
Figure 3a shows a causal graph for single mechanism, where there can exist unknown arbitrary causal links between the variable's parents. Suppose take the endogenous causes of and convert all but one into exogenous causes resulting the causal graph in Figure 3b. The bidirectional arrow between the endogenous parent pa, and the now exogenous parents paхı denotes the unknown causal relationships between them. This manipulation results in "partial" mechanism x: gi((kR, pag)
15

---

Published as a conference paper at ICLR 2023
where only parent is endogenous and thus susceptible to action. The remaining parents and the previous exogenous noise are grouped into a new noise variable Çb: = 8k (z, pag)). However, due to the dependence between the target parent pag and the unobserved exogenous parents pa abducting ÇA is impossible. The path between Ça and paş, breaks the assumption of independent exogenous noise variables, making it impossible to separate the effects of endogenous and exogenous causes. The mechanism induced by ÇE now depends the cause pa, since Çk K pag. violating the principle of independent causal mechanisms.
To preserve the principle of independent causal mechanisms, must first make pag, independent of the remaining parents pa Since the causal direction between pay and paçy & is unknown, we intervene both these variables, thus guaranteeing their link is severed. The modified SCM is shown in Figure 3c, where abduction is now possible since the mechanism no longer depends on the cause. Because there are backdoor paths between påg and Çs. using do-caleulus (Pearl, 2009), we have that P(Ck z. (o(a))) )) P(Ça I.. pag). Since x" depends on r via Ç& the distribution over potential counterfactuals P(x" xr, pag) remains unchanged after the intervention. After the intervention, we can learn the partial counterfactual function without confounding being an issue. The partial function changes one parent while holding the remainder fixed since these are now part of the exogenous
noise.
C.I PARTIAL COUNTER FUNCTION DECOMPOSITION
Using the process described in Section 3. we can write a partial counterfactual function as: x² == & (ar, pa paç. where f P(fx). Since the output of (-) is counterfactual of all the properties from Section 3.1.1 still apply. Repeating the process for each parent results in a set of independent partial functions allowing us to write the full counterfactual function as composition of multiple independent partial functions:
= [fxc pag, paj) o... f2(., paz, paş) fi (., pa, paj)] (x)
(10)
By construction, each partial function models the effect of a single parent on the observation independently of other parents. As result, the decomposition in equation 10 must be commutative.
Lemma C.1. Let = g(e, ; paz) be the generative model for an observation æ with parents pa, and pag and exogenous noise variable c. Further, let fç (r. pag pat) and f2 pan, paş) be two partial functions operating on pa, and pa2 respectively. Then, the composition of fi (-) and fz(-) is commutative:
G (12(1, p82, pat). pag. paç) f2(fi (z,, p8g, par), p82, paa)
(11)
Proof. Since any counterfactual of x can be generated by the generative model g(.), we have:
f (x. pa paï) = g(e, paj, pag) fg (z. pag; paj) = g(e, pa, pa.)
(12)
Substituting on the left-hand side of equation 11: f (I2(2,, p42, pag), pa pai) = [ (g(e, pa,, paz), pai, pai) g(c, pai, pa5)
(13)
And on the right-hand side:
(2 (Ir, рaq, paï), pay, pao) = (g(e,, paj, paz), pag, pai)
(14)
g(e, pai, paς)
16

---

Published as a conference paper at ICLR 2023
D COLOUR MNIST
D.1 DATASET DETAILS
(a) Unconfounded joint distribution.
(b) Confounded joint distribution without full support.
(c) Confounded joint distribution with full support.
Figure 4: Colour MNIST joint distribution of digit and hue for different SCMs.
0
6
0
1
1
|
2
2
2
/
/
(
/
2
2
2 3
2
2
2
2
2
2
2
2
2
3
333 3
2
2
2
2
2
2
2
3
3
3
3
3
3
3
3
3
3
4
4
4
4
4 5
44
3
3
3
3
4
4
y
4
4
4
4
4
4
4
5
S
5
5
S
5
S
5
5
5
6
4
5
5
5
5
5
5
5
5
5
S
6
6
L 7
6
6
6
6
6
6
6
6
6
7
7 8
7
6 6 6
6
6
6
6
6
6
6
7 8
7
7
7
7
7
7
7
7
8
8
8
&
8
8
8
8
7
7
7
8
8
8
8
8
8
8
8
8
9
9
9
4
9 9
8 8
9
8
9
9
9
9
9
9
(a) Samples from the unconfounded joint distribution.
(b) Samples from the confounded joint distribution.
Figure Colour MNIST samples.
D.2 EXPERIMENT DETAILS
We describe the architectures of the different components of the model using pseudo-code snippets. These snippets close the JAX framework but should enable reproducing the same architectures in all popular deep learning frameworks.
D.2.1 PSEUDO-ORACLES.
Architecture.
The architecture of the pseudo-oracles is:
pseudo_orade _orcoleorcleeeeoooo serial
Conv couttohannaceeee filter_shape (4, 4), strides- (2, 2)),
LeakyRelu,
Conv (out_ (out_chan=64, filter_shape- (4, 4), strides= (2, 2)),
LeakyRelu,
Flatten,
Dense (out_dim=128),
LeakyRelu,
Dense wwwt_tiinnuuummmmeaaa clm_sssses if classification else 1)
hat pseudo_oracla (image)
We trained the pseudo-oracles for 2000 steps with a batch size 1024 using the AdamW (Loshchilov & Hutter, 2019) optimiser with learning rate of 0.0005, 31 0.9, 8r 0.999
Training details.
17

---

Published as a conference paper at ICLR 2023
and weight_ decay 0.0001. For data augmentation used random translations where the amount of horizontal and vertical translation is sampled uniformly from 0 to 10% of the width/height of the image.
D.2.2 VAE
The following image encoder and decoder used in both the GAN and VAE:
Generic image encoder and decoder.
image_encoder serial {
Conv (out_chan=64, filter_shape-(4 (4, 4), strides= (2, 2)),
LeakyRelu, Conv (out_chann664, filter_shape- (4, 4), strides- (2, 2)),
LeakyRelu,
Flatten,
Dense (out cout_dim=128), LeakyRelu)
tiaggeddecode serial Dense (7 7 64),
LeakyRelu,
Reshape ( (-1, 7, 7, 64)),
Resize (-1, 14, 14, 64), method='linearr) Conv (ont_cchaca filter_shape- (4, 4), strides= (1, 1), padding='SAME'), SAME'),
LeakyRelu,
Resize ( (-1, 28, 28, 64), method='linear") Conv (ouu_chaan=64, filter_shaper (4, 4), strides= (1, 1), padding= "SAME") LeakyRelu,
Conv (out_chan,,, filter_shape- (3, 3), strides (1, 1), padding=' SAME' ))
Architecture.
The VAE architecture is as follows:
#encoder encoded_ Smage__ncodee (image)
tmp Dense pout_dim-1288 (concat imaag,, parents))
mu = Dense fout_dim==t (tmp)
sigma softplus (Dense (out_ dim=16) (tmp))
#decoder
sample_from_standrree counterfactual Silage_decoder (concat (2, counterfactual_pareeee
(mu, sigma)
Training details. We trained the VAE for 20000 steps with a batch size of 512 using the AdamW
(Loshchilov & Hutter, 2019) optimiser with learning rate of 0.001, B1 0.9, ,3 0.999 and weight_decay = 0.0001. The data-augmentation the same as in Section D.2.1.
D.2.3 GAN
Architecture.
The architecture of the GAN generator and discriminator are as follows:
#generator
gan_decoder serial(
Dense (out_dim=128) LeakyRelu,
image_decoder
Tanh
pecoded_image image_encoder (image)
18

---

Published as a conference paper at ICLR 2023
tmp concat sencoded_imace parents, counterfaactall_paeettt.
counterfactual gan gan_decoder (tmp)
# critic
critic serial
Conv (out_chan--64, filter_shape (4, 4), strides (2, 2), padding "SAME')
LeakyRelu,
Conv (outt (out_chann,,,, filter_shaper (4, 4), strides= (2, 2), padding= "aainn=sSAMMM" SAME
LeakyRelu,
Flatten,
Dense (out_dim=128),
TeakyRelu,
Dense (out_dim=128))
LeakyRelu
Dense (1)
critic_input concat (image, broadcast_to_shape (parents, image. shape) logits = catticceritic_ipput
Training details. We trained the GAN for 20000 steps with a batch size of 512 using the Adam
(Loshchilov & Hutter, 2019) optimiser with learning rate of 0.0001, 81 0, 82 0.9 and weight_ = 0.0001. The learning rate is multiplied by 0.5 at 12000 steps and again multiplied by 0.2 16000 steps. The aumeentgmtnttoinn is the same in Section D.2.1.
D.3 EXTRA RESULTS
Table 2:
Effectiveness results on colour MNIST when using pseudo oracles trained from biased data without using simulated intervention. We measure effectiveness using digit accuracy and hue absolute error in percentage points since hue E [0, I]. The average model ranking change compared Table (oracles trained from unbiased data) was 0.8(3) for the dataset without full support and 0.75 for the dataset with full support.
inter-
digit intervention effectiveness
hue intervention effectiveness
dataset
model
avg.
original
abs. dift'
tion
acCaigit (%) f
8e (%)
aCCaiai (%)
(%)
rank
rank
found- Bemoulli VAR 8=1
53.77
(9.88)
11.43 (4.56)
53.68 (6.08)
11.80 (5.85)
3.5
0
Bemoulli VAE 8=2
48.31
(2.27)
10.42 (1.64)
49.32 (2.01)
10.15 (1.62)
2.5
1.5
ed
Normal VAE 3=5
57.65
(8.41)
13.92 (3.38)
60.69 (6.96)
13.63 (3.51)
3.5
5.5
2
w/o
Bemoulli VAE 8=1
49.83 (2.52)
(U.87)
50.92 (2.07)
9.51 (1.35)
2.25
full
support yes
Berouuli VAE 8=2
46.48
(1.22)
8.51 (0.87)
47.30 (1.34)
8.53 (0.87)
3.5
3.25
0.25
Normal VAE 3=5
43.86 (1.28)
8.28 (1.38)
47.19 (1.10)
7.97 (1.34)
4
0.5
Bemoulli VAR A=1
87.71
(0.92)
10.90 (6.53)
90.15
(0.48)
14.02 (10.59
6.5
0.5
con-
no
Bemoulli VAE 8=2
90.05 (0.44)
(0.20)
90.64 (0.52)
5.92 (0.23)
5.25
3.25
2
found-
Normal VAE 3=5
82.98 (0.91)
(0.73)
80.23 (0.19)
(0.65)
7
0.5
yes*
GAN
87.03 (0.62)
4.71 (0.16)
92.51 (0.36)
5.28 (0.34)
4.25
1.25
w/
Bemoulli 8=1
92.07 (0.25)
(0.19)
92.19
(0.35)
6.15 (0.59)
3.75
4
0.25
full
Beroulli 8=2
93.98 (0:26)
(0.25)
93.76 (0.15)
5.82 (0.23)
2
0.5
support
Normal VAE 8=5
88.68 (0.33)
(0.46)
91.85 (0.12)
5.87 (0.47)
4.75
0.25
GAN
87.40 (0.14)
4.78 (0.18)
90.90 (0.21)
4.90 (0.16)
3.5
4.25
0.75
19

---

Published as a conference paper at ICLR 2023
Table 3: Effectiveness results colour MNIST when using pseudo oracles trained from biased data using a simulated intervention. We measure effectiveness using digit accuracy and hue absolute emror percentage points since hue E [0, 1]. The average model ranking change compared Table (oracles trained from unbiased data) was 0.8(3) for the dataset without full support and 0.625 for the dataset with full support.
inter- ven- model tion
digit intervention effectiveness
hue intervention effectivencss
dataset
avg.
original
abs. diff.
rank
rank
DCCaigit (%) f (0how 4
BCCdigh (%6 1 aC (%) 4
con-
Bemoulli VAE 8=1
57.09 (10.01)
11.17 (188)
57.14
(6.63)
11.42 (6.06)
3.5
3.5
0
found-
nо
Bermoulii VAE 8=2
53.43 (2.44)
10.08 (1.57)
54.03 (2.57)
9.73 (1.55)
4
2.5
1.5
ed
Normal VAE 3=5
62.93 (8.26)
13.81 (3.27)
67.23 (7.21)
13.55
(3.38)
3.5
5.5
2
w/o
Bemoulli VAR 8=1
53.72 (2.61)
8.57 (1.00)
55.36
(2:37)
8.88 (1.45)
2.25
0.75
full
Bemoulli VAE 8=2
50.96 (0.76)
7.93 (0.96)
51.79 (0.72)
7.89 (0.90)
3.5
3.25
0.25
support
Normal VAE 3=5
47.89 (1.75)
7,83 (1.58)
51.23 (2.01)
7.42 (1.59)
3.5
4
0.5
Bemoulli VAË 8=1
93.25 (1:48)
8,93 (7.08)
96.57 (0.42)
12.36 (11.61
6.5
1
Bemoulli VAE 8=2
96.90 (0.40)
3.17 (0.11)
96.94 (0:20)
2.98 (0.14)
4.25
3.25
1
found-
Normal VAE 3=5
90.37
(0.45)
4.16 (1.07)
95.92 (0.25)
3.85 (1.11)
7.5
7
0.5
ed
yes
GAN
93.30 (0.69)
2.36 (0.19)
97.47 (0.23)
2.52 (0.09)
3.25
4.25
1
w/
Berouuli VAE 8=1
97.36 (0.18)
2.80 (0.14)
97.70 (0:25)
3.67 (0.83)
3.5
4
0.5
full
Bemoulli VAE 8=2
98.74 (0.11)
2.68 (0.17)
98.84 (0.16)
2.61 (0.16)
2
0
support yes
Normal VAE 5
94.68 (0.39)
3.26 (0.24)
97.50 (0.21)
3,20 (0.25)
4.5
4.75
0.25
GAN
93.90 (8)
2.38 (0.16)
96.58 (0.15)
2.44 (0.17)
3.5
4.25
0.75
Table 4: Effectiveness colour MNIST when using linear/logistic regression pseudo oracles trained from unbiased data. We measure effectiveness using digit accuracy and hue absolute error in percentage points since hue [0, 1J. The average model ranking change compared Table (oracles trained from unbiased data) was 0.91(6) for the dataset without full support and 0.6875 for the dataset with full support.
inter-
digit intervention effectiveness
hue intervention effectiveness
dataset ven- model tion
avg.
original abs. rank
rank
diff.
aCGdigis (%) f 80 (%) |
aCGiiii (%)
aChom (%) $
con-
Bernoulii VAB 8=1
80.83 (1.91)
17.77
(4.12)
81.79
(1.42)
17.28 (4.14)
4.75
3.5
1.25
found- nO
Bemoulli VAR 8=2
86.18
(0.63)
16.17
(1.32)
86.03 (0.52)
15.91 (1.27)
2.5
2.5
0
ed
Normal VAE 8-5
78.67
(1.32)
18.97
(1.56)
82.43 (3.80)
18.37 (1.22)
5.25
5.5
0.25
w/o
Bemoulli VAE 8=1
82.23 (0.81)
15.22
(0.63)
82.31
(1.01)
14.24 (0.74)
2.75
2.25
0.5
Berouuli VAE 8=2
80.52 (2.11)
15.93 (1.34)
79.37
(2.49)
15.74 (1:41)
4.25
3.25
1
support
Normal VAE 3=5
82.50 (1.61)
14.17
(1.01)
85.20 (1.13)
13.82 (1.08)
1.5
4
2.5
Beroulli VAE 3=1
82.26 (2.97)
18.05
(4.15)
84.23 (1.16)
18.73
(5.29)
7.25
6.5
0.75
con-
Bemoulli VAR A==
87.55 (0.47)
14.75 (0.19)
87.00 (0.28)
14.80 (0.20)
3.5
3.25
0.25
found-
Normal VAE 3=5
81.78 (0.93)
15.14 (U.39)
$4.90 (0.59)
14.36 (0.444
5.25
7
1.75
ed
*
GAN
80.01 (1.28)
14.71 (0.08)
87.09 (0.84)
14.87 (0.17)
4.25
0.75
w/
Bemoulli VAR A=l
87.17 (0.64)
14.88 (0.11)
86.09
(1.73)
14.37 (0.64)
3.75
4
0.25
full
Bemoulli VAB 8=2
89.87 (0.51)
14.70 (0.20)
89.32 (0.34)
14.56 (0.16)
1.5
2
0.5
support yes
Normal VAE 8=5
85.43 (0.59)
15.00 (0.17)
87.23
(0.53)
14.84 (0.25)
4.5
4.75
0.25
GAN
80.39 (0.56)
14.77 (0.15)
85.51
(0.51)
14.65 (0.12)
5.25
4.25
1
20

---

Published as a conference paper at ICLR 2023
E 3D SHAPES
E. EXPERIMENT DETAILS
For the 3D Shapes dataset we used the same architectures and training regimes as in Section D.2 except we did not use aata-auumentation and modified the image encoder and decoder as follows:
image_encoder = serial
Conv (out_chan=64, filter_shaper (4, 4), strides= (2, 2)),
LeakyRelu,
Conv Conv(oout_cha-64,, filter_shape (4, 4), strides= (2, 2)),
LeakyRelu,
Conv (out (out_chan=-44, shater_shapetiee (4, 4), strides- (2, 2)),
LeakyRelu, Flatten,
Dense (out_dim=128),
LeakyRelu)
image__deee decoder serial( Dense (8 * 64),
TeakyRelu,
Reshape (-1, 8, 8, 64)),
Resize (-1, (-11, 16, 16, 64), method-'lineaz" Conv(out_chan-644 filter_shape- (4, 4), strides- (1, 1), padding " SAME"),
LeakyRelu,
Resize (-1, 32, 32, 64), method='linear') Conv (out_chaaa filter_shape (4, 4), strides= (1, 1), padding='SAME")
LeakyRelu,
Resize (-1, 64, 64, 64), bethod='lineat Conv (out (outtoouunn_)) filter_ (4, 4), strides- (1, 1), padding SAME'),
LeakyRelu, Conv (out_chan=3, filter_shaper (3, 3), strides= (1, 1), padding= SAME
E.2 EXTRA RESULTS
Table 5: Results of quality tests on 3D shapes: we measure composition after the null intervention and reversibility after one intervention cycle. Effectiveness is measured using accuracy.
effectiveness (%)
revensihility
intervention model
(-)
(10)
BCC
BCCahsectooientatiin
aCCabyertehhge
aCCyalllon
VAF
100.00
100.00
100.00
100.00
100.00
100.00
1.62 (0.07) 1.62 (0.07)
floor huc
GAN
100.00
100.00
99.90 (0.16)
99.37 (0.13)
99.30 (0.16)
100.00
5.23 (0.15)
5.25 (0.15)
object
VAE
100.00
100.00
100.00
100.00
100.00
100.00
1.62 (0.07)
1.62 (0.07)
GAN
100.00
99.99 (0.01)
99.91 (0.16)
99.29 (0.16)
99.16 (0.25)
100.00
5.33 (0.16)
5.35 (0.16)
object
VAE
100.00
100.00
100,00
100.00
100.00
100.00
1.62 (0.07)
1.62 (0.07)
urientation
GAN
100.00
100.00
99.88 (0.16)
99.07 (0.15)
99.00 (0.22)
100.00
5.32 (0.14)
5.35 (0.14)
object
VAE
100.00
100.00
100,00
100.00
100.00
100.00
1.62 (0.07)
1.62 (0.07)
scale
GAN
100.00
100.00
99.91 (0:15)
98. (0.27)
99.19 (0.27)
100.00
5.30 (0.15)
5.31 (0.15)
object
VAE
100.00
100.00
100.00
100.00
100.00
100.00
1.62 (0.07)
1.62 (0.07)
shape
(0:12)
98. (0.27)
(0.16)
(0.15)
(0.15)
VAE
100100
100.00
100.00
100.00
100.00
100.00
1.62 (0.07)
1.62 (0.07)
wall hue
(0:20)
(0.
(0:19)
100.00
(0.16)
5.29
21

---

Published as a conference paper at ICLR 2023
(a) Floor huc (VAE).
(b) Floor hue (GAN).
(c) Wall huc (VAE).
(d) Wall hue (GAN),
(e) Orientation (VAE). (I Orientation (GAN).
(g) Object scale (VAE).
(h) Object scale (GAN).
3D shapes. Each sub-figure has sample per row: the first column is the input, the second column is the result of the null intervention, and the last column is the result of partial intervention on a specific parent.
Figure 6: Effectiveness/Compoiitie
22

---

Published as a conference paper at ICLR 2023
F CE
F.1 DATASET DETAILS
For our CelebA-HO (Karras et al., 2018) experiments, we randomly split the 30,000 examples into 70% for training, 15% for validation and 15% for testing. We selected the 'smiling' and "eyeglasses' binary parent attributes for conditioning our generative model. After careful consideration, we opted for these attributes for three main reasons: (i) they are gender neutral which mitigates bias and allows us to use the entire dataset rather than a smaller subset; (ii) they are more objective than other attributes like "attractive' and 'young": (iii) we can train reasonably accurate pseudo-oracles for our counterfactual evaluation on these attributes, which is not the case for some others due to label noise/ambiguity and significant class imbalance.
F.2 EXPERIMENT DETAILS
F.2.1 GENERATIVE MODEL
Architecture. For the experiments on the CelebA-HQ dataset, we follow the general very deep VAE (VDVAE) setup proposed by (Child, 2021), and introduce some modifications to accommodate both parent conditioning and our compute constraints. The architecture is based on the ResNet VAE proposed by (Kingma et al.. 2016) but is much deeper and uses bottleneck residual blocks. The VDVAE has several stochastic layers of latent variables which are conditionally dependent upon each other, and are organised into L groups = {zu, Zi...., } These latents are typically output as feature maps of varying resolutions, whereby Zo consists of fewer latents at low resolution up to many latents z at high resolution. The conditioning structure is organised following the ladder structure proposed by (Sønderby al., 2016), where both the prior po (z) and approximate posterior (z) x) generate latent latent variables in the same top-down order. As in previous work, the prior and posterior are diagonal Gaussian distributions and the model is trained end-to-end by optimizing the usual variational bound on the log-likelihook (ELBO) (Kingma & Welling, 2013; Maaløe et al., 2019). We found that a naive application of VDVAE to produce high resolution counterfactuals leads to ignored counterfactual parent conditioning. Although the setup we describe next worked well enough in our experiments, understanding and overcoming this issue the general case likely warrants further investigation.
For our experiments, we used a VDVAE with stochastic latent variables spanning resolution scales up to the 64 x 64 input resolution: {13, 42, 82, 162, 322 642), where each latent variable has 16 channels. We used the following number of residual blocks per resolution scale: {4, 4, 8, 12, 12, 4}. resulting in total of 42 stochastic latent variable layers and 19M trainable parameters. Additionally, we modified the original architecture from a fixed channel width (e.g. 384) across all resolutions to the following custom channel widths per resolution: {32, 64, 128, 256, 512, 1024). We found that reducing the number of channels at higher resolutions and increasing them for lower resolutions performed well enough in our experiments, whilst reducing both memory and runtime requirements significantly.
Conditional VDVAE. For our counterfactual generation purposes, we augmented the original prior and posterior tow-down conditioning structure include x's parents pa as follows:
1.
Pa (z pa) = Po (Z() II pa(z; (z: Zi pa), qo (1/z/ pa) (so xx) II (z, pa). (15)
We call this the Conditional VDVAE. In practical terms, we simply expand and concatenate the parent conditioning attributes to the latent variables at each stochastic layer, then merge them into the downstream via a 1 x 1 convolution. Although there are many other conditioning structures one could consider, this one proved good enough for our experiments. In order to produce counterfactuals, we perform abduction by passing images and their parent attributes through the encoder and retrieving the posterior latent variables at each stochastic layer. We then fix these latents and propagate them through the decoder along with the counterfactual parent conditioning. We also found it beneficial to replace the original diagonal discretized logistic mixture likelihood (Salimans et al., 2017) used in VDVAEs with diagonal discretized Gaussian likelihood (Ho al., 2020), as it produced visually sharper
23

---

Published as a conference paper at ICLR 2023
counterfactuals in our experiments. Importantly, model trained with a 3 penalty (Higgins et al., 2017) of 5 which discouraged from focusing mostly on maximising the likelihood term the ELBO and ignoring counterfactual conditioning at inference time. We found that this introduces trade-off between reconstruction quality and obeying counterfactual conditioning.
Training Details. We trained our VDVAE for 1.7M steps with batch size of 32 using the Adam (Loshchilov & Hutter, 2019) optimiser with an initial learning rate of le-3, B1 0.9, 82 0.9 and weight decay of 0.01. The learning rate was linearly warmed-up from 0 to le-3 over the first 100 steps then reduced to 1.5c-4 175K steps and again to 1.5c-5 at 900K steps. We set gradient clipping to 220 and gradient L2 norm skipping threshold to 380, and observed no significant instabilities during training. The final artefact rate of 0.999 which we use at inference time. For lata-augmentation, since the initial dataset was 128 128 resolution, during training applied random cropping 120 120 and resized to 64 64 before applying random horizontal flipping with probability 0.5 and finally scaling pixel intensities [-1, 1]. At test time simply resize to 64 x 64.
exponential moving average of the model parameters with
F.2.2 PSEUDO-ORACLES
Architecture. In order to evaluate the effectiveness of our CelebA-HQ counterfactuals, trained the following simple classifier for each of the 'smiling' and "eyeglasses' binary attributes:
class ConvNet (an.Module])
def
_nnit_(seff)= super () ()__init________________________ ()
self. nn. S
nn. Cn.Con2 (3, 16, 3, 1, 1),
(m.BatchNorm2d(16)) nn An.ReLU(),
(16, 32, 3, 2, 1), nn. m..attchNorm2d1333 (32),
R).Ue nn CnCon 32, 3, 1, 1), nn m.BatcchNor22d (32), nn.ReLU() nn. (32, 64, 3, 2, 1), BatchhhNorm22 nn. nn.Retu
C6Co2 64, 3, 1, 1) nn (at)
nn (),U(),
nn Conv2d 128, 3, 2, 1), nn. (12))) nn Re nn. im.Akeptiieevvpoo12220
self.fc nn. ww..equential
nn Linear (128, 128), nn. nn.BatchNormld (128), Rn. (), Lineir (128, 1)
def (orwar x)
return self. (self. cnn (x) sqqueee ())
Training Details. Our attribute classifiers were trained for 300 epochs with batch size of 32 using the AdamW (Loshchilov & Hutter, 2019) optimizer with initial learning rate of 1e-3, $1 0.9. B2 = 0.999 and weight decay of 0.01. The learning rate followed plateau schedule on the validation set FI-score, where it halved after 50 epochs of improvement. The final artefact is
24

---

Published as a conference paper at ICLR 2023
exponential moving average of the model parameters with rate of 0.99 which time. To find the best binary classification threshold between and 1, we evaluated the validation set at threshold intervals of 0.001 and took the best performing threshold in terms of Fi-score. We ran training from random seeds and the final test set F1-score for the 'smiling' attribute is 0.9342 + 0.00188 and for the "eyeglasses' attribute it is 0.9758 + 0.0037. For data augmentation, the VDVAE setup but also included the torchvision. www.sstnnnnsss..mmmm Auton AutoAutouutt Augment)) augmentation module from PyTorch (Paszke al., 2019) during training strengthen regularisation. Since the CelebA-HQ binary attributes can be highly imbalanced, we also used weighted random data sampler (with replacement) during training, which ensures minibatches have balanced representation of each class at each step.
inference
followed
F.3 EXTRA RESULTS
Compesition
Reversibility
50
0.94
Efeciiennss (P1
distance
0.92
80
0.88
Null Intervention
Countorfactual
10
Cycle
10
20
10
20
30
40
Abducted latent variables
Abducted latent variables
(a) Effectiveness ('smiling)).
(b) Composition & Reversibility ('smiling').
Composttion
Reversibility
Broctvenees (F3 (3-3ccr)))
0.850
50
0.925
1osstance
Null Intervention Counterfectual
0.900
Reverse Cycle
80
0.825
10
0.800
10
20
40
10
20 Abducted latent varia bles
30
40
Aboucted latent variables
(c) Effectiveness ('eyeglasses")
(d) Composition & Reversibility ("eyeglasses") ").
Figure 7: Measuring CelebA-HQ test set effectiveness, composition and reversibility as a function of the number of latent variables abdueted. Notice how counterfactual effectiveness drops rapidly as more (posterior) latent variables are abducted, unlike the null intervention and reverse cycle case.
25

---

Published as a conference paper at ICLR 2023
Eftectvenest (F1 (11-soore)
23
0.92
0.90
Intervention
Counterfactual
Composition Reversibility
Reverse Cycle
Temperature
Temperature
(a) Effectiveness ('smiling").
(b) Composition Reversibility ('smiling') )
0.95
venee (F) (2002c)))
0.90
distance
22
23
0.75
Null Intervention Collterfactual
0.70
Composition Reversibility
Reverse Cycle
0.65
0-4 Prior Temperature
1.0
Prior Temperature
(c) Effectiveness (' ("eyeglasses").
(d) Composition Reversibility ('eyeglasses")
Figure 8: Measuring CelebA-HQ test set effectiveness, composition and reversibility as a function of the temperature used when sampling from the prior. this example, the first 8 (posterior) latent variables were abducted and the remaining 34 were randomly sampled from the prior. Notice how low prior temperatures affect the effectiveness of cycled back images the most (Figure (a)).
Table 6: Soundness metrics on the CelebA-HQ test set over 3 random seeds, with different subsets of abducted latent variables from our conditional VDVAE model. Composition is measured via the null
intervention and reversibility after one intervention cycle. Effectiveness of test set counterfactuals is measured using Fi-score given by our siilling//eyeglaasssss attribute classifiers.
smiling intervention
eyeglasses intervention
latents
composition (())
reversibility g(1)
effectiveness
composition z |
reversibility ((9)
effectiveness
abducted
F1-score
F1-score f
60.100 (0.127)
59.521
(64)
0.984 (0.0005)
60.273 (0.144)
59.263
(0.013)
0.979 (0.002)
55.521 (0.068)
54.360 (0.092)
0.982 (0.0008)
55.443 (0.036)
54.314
(0.055)
0.982 (0.005)
29.489 (0.018)
30.705 (0.030)
0.957 (0.001)
29.491 (0.048)
30.758
(0.025)
0.976 (0.005)
20.890 (0.018)
22.604 (0.025)
0.933 (0.002)
20.896 (0.011)
22.526
(0.005)
0.932 (0.006)
16
15,997 (0.003)
18.269 (0.021)
0,921 (0.003)
16.312 (0.009)
18.323
(0.009)
0.932 (0.019)
6,606 (0.001)
9,880 (0.017)
0.886 (0.006)
8.092 (0.002)
9.290 (0.006)
0.872 (0.003)
32
4.160 (0.002)
6.189 (0.015)
0.858 (0.001)
4,451 (0.0006)
5.606 (0.010)
0,865 (0.023)
42 (all)
3.657 (0.0006)
5.701 (0.017)
0.848 (0.0006)
3.656 (0.001)
4.808 (0.004)
0.854 (0.006)
26

---

Published as a conference paper at ICLR 2023
Table 7: Ablation of 8 penalty training composition, reversibility and effectiveness. Soundness metrics were calculated on the CelebA-HQ test set using the 'smiling' attribute, over random seeds. We used our conditional VDVAR with diagonal discretized logistic mixture likelihood. Notice how no penalty (8=1) yields the best likelihood, composition and reversibility the cost of low effectiveness (i.e. ignored counterfactual conditioning under full abduction).
B
latents
NLL
composition ((1)
reversibility 2 4
effectiveness
penalty
abducted
bits per dim
Fl-score
42
S 2.874
1.794 (0.0002)
2.482 (0.002)
0.536 (0.014)
3
42
< 3.556
4.335 (0.001)
5.333 (0.002)
0.720 (0.002)
5
42
S 3,931
6.362 (0.002)
7.775 (0.013)
0.877 (0.002)
7
42
4.1889
10.020 (0.008)
8.040 (0.0006)
0.910 (0.003)
10
42
S 4.446
12,209
(0.003)
10.050 (0.002)
0.952 (0.001)
27

---

Published as a conference paper at ICLR 2023
Nill
Cyseed
Inteeenttn
Oyted
HOM NORNAN
MHE WONNINNI
ENG DO
PAS NO KOONON
PIG MONNIN
DMC CORROO
Observatto
Nill Intervention
Counterfectual
Cyteed Back
Obse vation
Intervention
Counteriocual
Cyded Bock
MEC SHiPS
weing
Eie no soraring
Int siiling
PSt (18li
PSt miln
SHE Pnili
Figure 9: CelebA-HQ test counterfactuals of 'eyeglasses' (Top) and 'smiling' (Bottom) attributes. 'Null Intervention' refers to standard reconstruction using observed parents and 'Cycled Back' refers performing abduction an observation's counterfactual and re-conditioning back on the observed parents. The x-axis shows the predicted label by the respective attribute classifier for each image.
28

---

Published as a conference paper at ICLR 2023
de(smiling, eyeglesses)
dojsmiling, eyeglassesi
Ubservation
(0,0)
(0.1)
(1,0)
Observation
<0,03
(0.1)
(1.0)
O.1
e
Iw0.1
1w0.1
t-0
t-03
L-0.7
Le0.7
t-0
b-0.
Observation
Observation
tw0.1
tw0
U
tw0
tw0.7
bw0.7
tw0.
U
US
b-0.9
Figure 10: Traversing counterfactual latent space under different interventions and prior temperatures t (le-5, 0.1, 0.3. 0.5, 0.7, 0.9}. We are visualising CelebA-Hi test set 'smiling' and 'eyeglasses' counterfactual combinations under partial abduction of the first 8 latent variables of our conditional VDVAE, where the remaining latents are sampled from the prior different l's per row.
29

---

Published as a conference paper at ICLR 2023
Figure 11: Visualising the evolution of 'smiling' CelebA-HO counterfactuals from full abduction (42
latents) to partial abduction (32-1 latents) and finally random samples (0 latents fixed).
30

---

Published as a conference paper at ICLR 2023
PD
PL
Figure 12: Visualising the evolution of 'eyeglasses'' CelebA-HQ counterfactuals from full abduction
(42 latents) to partial abduction (32-1 latents) and finally random samples (0 latents fixed).
31

---

Published as a conference paper at ICLR 2023
Figure 13: Visualising CelebA-HQ test set 'smiling' reversibility. First column from the left contains the observations. the second column contains 'smiling' counterfactuals under full abduction (42 latents), and the third column contains the first cycled back reversal. The process is repeated for the remaining columns, showing progressive degradation away from the original observation.
32


Reviews:
[{"review_id": "tuCm7BBtm_s", "reviewer": "Reviewer_segb", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes an approach to learn counterfactual functions based on the properties that the axiomatic definition of counterfactuals require (composition, reversibility, effectiveness). Evaluating the degree to which this properties are satisfied allows to assess the distance between an ideal counterfactual and its approximation.", "summary_of_the_review": "The paper offers a good contribution which may be of interest for anyone working on causality and causal learning.", "strength_and_weaknesses": "+ The paper considers a relevant problem from an original perspective\n+ The proposed framework is grounded in SEM theory\n+ The framework is applied in representative experiments and datasets.\n+ Experiments, ranging from simpler to more realistic examples are well designed, discussed and commented, providing an illustration of the application of the author's frameworks.\n\n- It may have been interesting to have a short discussion (maybe in the Appendix) on the choice of metrics.\n- If feasible, comparison with other counterfactual deep models would have been instructive.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "Ln0m7zegJh", "reviewer": "Reviewer_o4KV", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes some metrics to evaluate counterfactual generation models, based on Galles and Pearl's axiomatization of counterfactuals that are interpreted according to (a certain class of) structural causal models. Corresponding to each of the three axioms -- effectiveness, composition, and reversibility -- a metric (family) is developed in the paper. Using these metrics, the authors examine the empirical behaviors and performance of some popular generative models, and demonstrate in experiments the potential of these metrics for indicating the quality of counterfactual generation. The extent to which simulated intervention can be used to mitigate confounding is also explored in the experiments. ", "summary_of_the_review": "This is a novel and interesting paper with quite a few merits. But I am also puzzled by several conceptual aspects of the paper, as indicated in my comments on the weaknesses. (I will raise the score if those comments prove to be my confusions in the subsequent discussions.) ", "strength_and_weaknesses": "Strengths:\n\n1. The central idea of designing evaluation metrics based on Galles and Pearl's axioms is novel. \n2. The development of the metrics is well argued and explained.\n3. The empirical studies are interesting and seem to provide good evidence for the potential usefulness of the proposed metrics.\n4. The demonstration of the utility of simulated intervention is clever and fairly convincing.\n\nWeaknesses:\n\n1. Galles and Pearl's axioms are proposed to characterize a certain class of structural causal models, i.e., those models that imply that the endogenous variables are fully determined by the exogenous variables (including the exogenous noises). This class is on the one hand more general than the usually assumed recursive models and on the other hand quite special among non-recursive models. This point is only alluded to but not made sufficiently clear in the paper. It should be explicitly discussed to properly qualify the intended applications of the proposed metrics. In particular, if more general feedback mechanisms that allow multiple equilibria are of interest, composition and reversibility will need to be weakened to accommodate them. \n2. It is not clear what \"counterfactual identifiability\" exactly means in this paper. If it means identifiability of counterfactual queries from observational data (or perhaps also interventional data?), I don't see how the soundness and completeness results imply that \"composition, effectiveness and reversibility are necessary and sufficient for counterfactual identifiability.\" It is possible to have a causal model that does not satisfy some of these properties (because it is outside of the class mentioned above), but still allows some counterfactuals to be identified, isn't it? Conversely, when data underdetermine the underlying causal model, even if all the possible candidates are assumed to satisfy these properties, counterfactuals may still not be identified. It seems to me that either the authors intend to use the term \"counterfactual identifiability\" in a very special sense, or there is some misunderstanding of what the completeness theorem amounts to.\n3. The proposed metric for reversibility seems to be much more about the invertibility of a causal mechanism than about reversibility. In particular, I don't see exactly how the proposed metric derives from applying (something like) reversibility to \"a twin network conception of the SCM.\" If it is really just a metric for invertibility, it is doubtful whether it is appropriate to require a counterfactual inference model to respect it (because invertibility may be too strong a requirement in many contexts). \n4. The proposed metric for effectiveness is apparently a metric to evaluate a combination of a counterfactual inference model and a pseudo-oracle function, rather than of the counterfactual inference model by itself. It is hence a little unclear in principle whether it is suitable for evaluating the latter. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "A67AqnFvt9k", "reviewer": "Reviewer_cqtT", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a method that evaluates counterfactuals.  The method does not focus on the algorithm (or models) for the inference of counterfactuals but on properties the inferred outputs must obey. Based on Pearl’s axiomatic definition of counterfactuals, the authors propose to check three properties: Effectiveness, Composition, and Reversibility. Experimental results demonstrate the utility of the proposed evaluation method. ", "summary_of_the_review": "The reviewer recommends to accept this paper. The reviewer believes the evaluation of counterfactuals is an important research topic and is worth discussed in the conference. ", "strength_and_weaknesses": "Strong points:\nAs mentioned by the authors, there is little methodology available to assess the quality of inferred counterfactuals. The reviewer believes the assessment of counterfactuals is very important and methodologies of the counterfactuals assessment should be discussed more vigorously. The idea to focus on the properties the outputs must obey is novel and the list of the three properties spearheads research on counterfactual inference.\n\n\nWeak points:\nIt should be more discussed on the properties the distance metric, dx, shown in Equations (1) and (2) should satisfy. It is sure that distance is equal to zero if the counterfactual function is ideal. In case the counterfactual function is not ideal, the distance would be larger than zero and the inference quality is measured by the distance metric quantitatively. The reviewer believes more discussion on the distance between given two images is needed.\n\nFor the check of Effectiveness, the method uses a pseudo-oracle function that infers the parents of a variable. An image generative model often generates artificial images that do not obey the probability density distribution of (real) images. When a generated counterfactual image deviates from the distribution of real images, the reviewer thinks, the pseudo-oracle function should not output any parents of a given image and should degrades the rating of Effectiveness.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster