From Identifiable Causal Representations to Controllable Counterfactual Generation: A Survey on Causal Generative Modeling
Authors: Aneesh Komanduri
Xintao Wu
Yongkai Wu
Feng Chen
Venue: Accepted by TMLR ()

Abstract: Deep generative models have shown tremendous capability in data density estimation and data generation from finite samples. While these models have shown impressive performance by learning correlations among features in the data, some fundamental shortcomings are their lack of explainability, tendency to induce spurious correlations, and poor out-of-distribution extrapolation. To remedy such challenges, recent work has proposed a shift toward causal generative models. Causal models offer several beneficial properties to deep generative models, such as distribution shift robustness, fairness, and interpretability. Structural causal models (SCMs) describe data-generating processes and model complex causal relationships and mechanisms among variables in a system. Thus, SCMs can naturally be combined with deep generative models. We provide a technical survey on causal generative modeling categorized into causal representation learning and controllable counterfactual generation methods. We focus on fundamental theory, methodology, drawbacks, datasets, and metrics. Then, we cover applications of causal generative models in fairness, privacy, out-of-distribution generalization, precision medicine, and biological sciences. Lastly, we discuss open problems and fruitful research directions for future work in the field.

Published in Transactions on Machine Learning Research (05/2024)
From Identifiable Causal Representations to Controllable Counterfactual Generation: A Survey on Causal Generative Modeling
Aneesh Komanduri
akomandu@uark.nm edu
Department of Electrical Engineering and Computer Science University of Arkonsas
Xintao Wu
zintaounuuurrk edu
Department of Electricol Engineering and Computer Science
Universily of Arkansas
Yongkai Wu
yongkans@ddemson.cde
Department of Electrical and Computer Engineering Clemson University
Feng Chen
Seng. www.chhhhlllllaa...mommooo. edu
Department of Computer Science University of Teras at Dallas
Reviewed on OpenReview: https: // opwprevieeeeeeee.emmmmoe forum? PUx
Abstract
Deep generative models have shown tremendous capability in data density estimation and data generation from finite samples. While these modols have shown impressive perfor- mance by learning correlations amoug features the data, some fundamental shorlcomings are their lack of explainability, tendency induce spurious correlations, and poor out-of- distribution extrapolation. To remedy such challenges, recent work has proposed a shift toward causal generative models. Causal models offer several beneficial properties to deep generative models, such distribution shift robustness, fairness, and interpretability. Struc- tural causal models (SCMs) describe data-generating processes and model complex causal relationships and mechanisms among variables in a system. Thus, SCMs can naturally be combined with deep generative models. We provide a technical survey on cansal genera- tive modeling categorized into causal representation learning and controllable counterfactual generation methods. We focus on fundamental theory, methodology, drawbacks, datasets, and metrics. Then, we cover applications of causal generative models in fairness, privacy, generalization, precision medicine, and biological sciences. Lastly, discuss open problems and fruitful rescarch directions for future work in the field.
1 Introduction
With the growing interest in using deep learning for domain-specifh tasks in the natural and medical sciences, deep generative models have become popular area of rescarch. Deep generative models are class of methods that either explicitly estimate the probability density or implicitly learn to sample from some data distribution. Such models have demonstrated impressive performance, especially in generating high-quality image samples. Stato-of-the-art deep learning models such as GPT-4 AOpenAI, 2023) and CLIP (Radford et al., 2021) have taken the advances in deep generative modeling new heights. Recent research efforts generative modeling focus important properties of the data-generating process that lead to interpretable

---

Published in Transactions on Machine Learning Research (05/2024)
representations and controllable generation (Locatello et al., 2019b). Despite such advances, most current methods neglect the fact that systems in the real world are complex, dynamic, and may have underlying causal dependencies. The failure to account for this has made deep learning models prone to shortcut learning and spurious associations (Arjovsky et al., 2020).
Many have argued that we must go beyond learning mere statistical correlations toward cousal models that capture the causal-effect relationship between influential variables in system. In the 1950s, Alan Turing asked what it would take for computer to think like human (Turing, 1950). Turing proposed what many today refer to the "Turing Test." However, more practical version of this test would be to ask the following question: how can machines represent causal knowledge of the world in a way that would enable them to access the necessary information quickly, answer questions accurately, and do so with ease (Pearl & Mackenzie, 2018)? Pearl's Causal Hierarchy (Pearl, 2019; Bareinboim et al., 2022) lays out the three levels of causal models that provide a mechanism to progressively get closer to modeling human-level reasoning: obscrvations (sceing), interventions (doing), and counterfactuals (imagining). Intervening on causal variables of system leads to the ability to reason about hypothetical scenarios in the world (counterfactuals).
Deep generative models such as variational autoencoders (VAE), generative adversarial networks (GANs), normalizing flows, and diffusion models have been at the forefront of machine learning for approximating complex data distributions. A traditional deep generative model would perform quite well in generating new data for an application such as generating novel images from brain Magnetic Resonance Imaging (MRI) data, However, understanding what and how underlying factors truly influence the generative process can only be captured by modeling the causal mechanisms between variables. If certain markers or labeled factors causally influence cancer diagnoses, incorporating them into causal model will more robustly encode the system's 's information and its complex relationships. If we want to know the effect of changing certain factors on the structure of the brain MRI, we can easily perform interventions to infer such relationships. Further, one could "simulate" patients by generating counterfactual instances of tumors under different interventions that lic outside the support of the training data. This would greatly reduce the cost associated with collecting and annotating patient data with constraints such as privacy. Thus, a shift lowards causal modeling is imperative to learn more robust models that reflect the complexities in the real world.
To improve the interpretability and robustness of generative modcls, recent rescarch has focused on learning causal generalive models using Pearl's structural causal model (SCM) formalism (Pearl, 2009). The SCM inherently describes a lata-generating process consisting of a set of variables that influence each other through causal mechanisms. The SCM describes a generative model of the world and, as such, can be naturally utilized in deep generative modeling. Recent work has shown the effectiveness of neural models in parameterizing causal mechanisms between variables (Kocaoglu et al., 2018; Xia et al., 2021). Causal models offer several beneficial properties deep generative models, such as distribution shift robustness, fairness, and interpretability (Scholkopf et al., 2021). The interpretability of models can be viewed in terms of both learning semantically meaningful representation of system and controllably generating realistic counterfactual data. In the context of representation learning, SCMs can be used to model the causal dependencies among encoded latent variables. In the context of controllable generation, SCMs can be used to model the mechanism that generates high-dimensional observational data as a function of causal variables to enable counterfactual inference.
We take this opportunity to survey the exciting and ever-growing field of causal generative models based on the Pearlian theory of structural causal models, discuss the benefits and drawbacks of different approaches, and highlight open problems and applications of causal generative models in variety of domains for real- world impact. Unlike other surveys (Kaddour et al., 2022; Zhou et al., 2023), in addition to covering the methodology of causal generative models, we cover foundational identifiability theory in causal represen- tation learning with a careful emphasis on intuition. We also outline several challenging open problems from the perspective of both application and theory. We focus our survey on generative models that utilize fundamental principles of causality to enable causal representation learning (CRL) and controllable counter- factual generation (CCG). We do not cover causality-inspiree frameworks causal discovery in this survey. Although other causal frameworks exist, such as Potential Outcomes (PO) (Rubin, 2005), we note that most methods in CRL and CCG utilize the SCM framework since they are concerned with explicitly modeling mechanisms among causal variables.
2

---

Published in Transactions on Machine Learning Research (05/2024)
CausalVAE, 2021
DEAR, 2021
SCM-VAE, 2022
ICM-VAE, 2023
Multi-domain CRL, 2023
G-CaRL, 2023
Data Paradigm
Static
Multi-view CRL, 2023
Causal Representation Learning (CRL)
Observational
Temporal
LEAPP, 20122
TDRL, 2022
NCTRL,, 2023
Interventional CRL, 2022
Linear CRL (linear), 2023
Linear CRL (general), 2023
Discrepancy-VAE 2023
Nonparametric CRL, 2023
Static
Score-based CRL, 2023
Interventional
Causal Generative Modeling
Temporal
CITRIS, 2022
Mechanism Sparsity, 2022
iCTTRIS, 2023
BISCUIT, 2023
SSL Data Augmentation, 2021
Counterfactual
L.M,, 2022
Parudigm
CausalGAN, 2017
Controllable Counterfactual
GAN-based
CGN, 2021
Generation (CCG)
CONIC, 2023
DSCM, 2020
Flow/VAE-based
Causal-HVe HVAE,
Diffusion-based
DHT-SCM,
Figure 1: Taxonomy of Causal Generative Modeling
3

---

Published in Transactions on Machine Learning Research (05/2024)
Causal representation learning (CRL) is concerned with learning semantically meaningful causally related latent variables and their causal structure from high dimensional data. We break down causal representa- tion learning into the three different classes from Pearl's causal hierarchy based on the assumptions on the data-generating process to achieve identifiability. That is, one assumes access to either purcly observational data, interventional data, or counterfactual data. Identifiability is an important property of representations learned by generative models since ensures that we can provably recover the true generative factors. This is important for representation learning since this theoretical guarantee implies the model enjoys robustness and generalizability properties. Achieving identifiability in causal representation learning also enables iso- lated interventions on causal variables to observe causal effects downstream, which is often of great interest in domain-specific applications. Controllable counterfactual generation (CCG) methods focus more on mod- eling known causal variables (such as labels) and learning a mapping to the observed data. We break down controllable counterfactual generation methods by GAN-based, Flow/ Flow/VAE-Be Vaseba and Diffusion- based mod- cls. One of the main assumptions in CCG methods is access to causal variables in the form of labels, which makes modeling the relationships between causal variables and high-dimensional data convenient, efficient, and flexible. However, in causal representation learning, the causal variables and their structure are often unobserved and we must disentangle the factors before being able to generate counterfactual images. We emphasize that CRL methods are also capable of controllable generation through latent space manipulations and decoding. However, the focus of CRL is primarily on learning robust and interpretable representations for downstream tasks. Our survey focuses on feature-level causal variables and not on more general causal modeling schemes. The taxonomy of methods shown in Figure 1.
Contributions. (1) We taxonomize causal generative modeling work into the identifiable causal represen- tation learning and controllable counterfactual generation tasks. (2) We further categorize causal represen- lation learning methods according to the lype of data they assune access to from Pearl's Causal Hierarchy and break down controllable generation approaches based on the type of generative model utilized. (3) We discuss the existing evaluation metrics, benchmark datasets, and results of causal generative models. (4) Finally, we lay out current limitations and fruitful research directions in both causal representation learning and controllable counterfactual generation.
Structure of survey. The survey is structured follows. In Section 2, discuss the background necessary to understand the rest of the paper, including formal definitions of SCM, common deep generative models, and disentangled ion learning. In Section 3, we formulate the causal representation learning problem and discuss methods categorized according to the type of data they assume access to from Pearl's Causal Hicrarchy. Then, in Section 4, we discuss the controllable counterfactual generation task and methods from GAN-based, Flow Flow/VAF-baseee
and Diffusion-based frameworks. most common metrics used in causal generative modeling- In Section 6, we discuss synthetic and real-world datasets that are often used in causal representation learning and controllable generation tasks. In Section 7, highlight applications of causal generative models in trustworthy and robust AI, precision medicine, and biology. In Section 8, we outline several open research directions in causal generative modeling. Finally, we conclude the survey in Section 9.
Section 5, we give an overview of the
2 Background
In this section, we introduce notation and key concepts in generative modeling and representation learning important for understanding the rest of the paper, including Pearl's structural causal model, variational autoencoders, normalizing flows, generative adversarial networks, diffusion probabilistic models, and disen- tangled representation learning. A list of the main notation used in the paper is provided in Table 1.
2.1 Structural Causal Model
All the methods discussed in this survey utilize the structural causal model (SCM) formalism from Pearl (2009), which formally describes the relationship among set of variables. The SCM induces hicrarchy of causal reasoning referred to as Pearl's Causal Hierarchy.
4

---

Published in Transactions on Machine Learning Research (05/2024)
Table 1: Main notation used for causal representation learning
Notation
Description
2
observed data
&
endogenous latent causal variables exogenous noise variables labels (domain, class, time etc.)
y X' C RRd
domain of observational data
Z IR"
domain of causal variables domain of noise variables domain of labels
& S 1
y
A
causal DAG adjacency matrix causal parents of variable 2 causal mechanism deriving variable 2; reduced form mapping noise to causal variables new mechanism replacing fo after intervention set of all causal mechanisms counterfactual data
paa
fi
fRF
fi
F
E
post-intervention causal representation post-intervention noise variables structural causal model
C
Č
intervened structural causal model perfect hard intervention mixing function joint distribution set of variables composition of functions Hadamard product
do(-)
g())
p(=) (or p=)
1 Note that the notations p(z) and Ps are used interchangeably throughout the survey and are equivalent
Definition 1 (Structural Causal Model) A structural causal model (SCM) is defined by tuple C (2, F,Pe), where Z is the domain of n endogenous causal variables = fz,...., ,, h E is the domain of n exogenous noise variables e = fci,...,ca) En), and F (fI...., is a collection of n cousal mechanisms of the form
2, fi(Ei, 2
(1)
where Vi, fo : E E ElIpp 2 Z; are causal mechanisms that determine each causal variable as a function of the parents and noise, Epa, the parents of causal variable 2;; and pc(c) is probability measure of E.
An SCM where the exogenous noise variables are jointly independent (no hidden confounders) is known Markovian model. The assumption of no hidden confounding is also referred to as causal sufficiency. We depict the causal structure of by causal directed acyclic graph (DAG) G with adjacency matrix A E 10.1) A 2-varialle SCM is shown in Figure 2a. The distribution over endogenous variables pf(2) is called the entailed distribution. We have that the joint distribution of endogenous variables can be written as the following causal Markov factorization
p (2) = II p (x)
(2)
Interventions. An SCM enables one to go beyond observational distributions to construct interventional distributions that reflect changes in a system at the population level. Interventions enable one to answer the question: "What would be if we change 2,?" We can construct interventional distributions by making modifications to an SCM C and considering the entailed distribution.
Let E, F. P.e be an SCM and (2) be its entailed distribution. We replace any number of structural assignments to obtain SCM C. Suppose we replace the mechanism that generates causal variable z;
5

---

Published in Transactions on Machine Learning Research (05/2024)
61
61
21
21
E2
do = c)
E2
f()))2)
12
E2)
22
(a) 2-variable SCM
(b) do-intervention (hard)
(c) soft (imperfect)
Figure 2: Types of Interventions
by
21 [ipin 6))
(3)
We call the entailed distribution of the SCM an interventional distribution, which is denoted by
o (z) y Cidola (eom:
(4)
The two main types of interventions are hard interventions, which remove dependency from parents, and soft interventions, which alter the dependency on direct parents, as depicted in Figure 2.
Hard intervention. A hard intervention, or do do-interventton, modifies an SCM by deterministically replacing for subset of causal variables the variable 2; with a constant c, denoted do(2, = c). such that the causal variable is no longer dependent its parents. The interventional distribution given hard intervention on variable 2 can be expressed as:
p(21,..., 2n do = =c) = II
p(2, 1pa
(5)
Jah
Soft intervention. Unlike hard interventions, which remove all direct causal edges of the intervened target, soft intervention, or mechanism change, simply changes the dependency of the causal variable on its direct parents, where the variable can still be dependent on subset of the original parents.
We introduce two additional subclasses of interventions that are of special interest in the causal generative modeling literature: perfect and imperfect interventions (Peters et al., 2017).
Perfect intervention. A perfect intervention modifies an SCM by replacing for a subset of causal variables the causal mechanism fs with new mechanism f. that is no longer dependent the original parents. With perfect interventions, all causal dependencies between the intervened target and its causes are removed, but the variable can still depend on its corresponding noise term, and its value may still be random. Thus, perfect interventions can be seen as a generalization of hard interventions where stochasticity allowed.
Imperfect intervention. An imperfect intervention modifics an SCM by replacing for a subset of causal variables the causal mechanism fi with a different causal mechanism ft that preserves the dependence on the causal variable's original parents. We note that imperfect interventions are a special case of soft interventions where the variable is still dependent all of its original parents.
Beyond the ability to compute causal effects from interventional distributions, SCMs enable reasoning about counterfactuals. A counterfactual is query at the unit level that asks the question: "Given an arbitrary unit, what would zj have been if z; were different."" Counterfactuals, unlike interven- tions, are relative to a factual observation. That is, given an observed set variables that reflect reality, counterfactuals aim infer what would happen the original observations were intervened upon while keeping the original noise terms. Let C = (Z, & F.Pe) be an SCM over variables 2. Given some observations 2, Pearl (2009) outlines the following three-step procedure to perform counterfactual inference:
Counterfactuals.

---

Published in Transactions on Machine Learning Research (05/2024)
L3 Counterfactual
L2 Interventional
L Observational
Figure 3: Pearl's Causal Hierarchy
Abduction: Infer noise terms compatible with observations a (i.e., p(e[z))
Action: Perform an intervention do(2, = c) corresponding to the desired manipulation, which results in a modified interventional SCM Č Cdo(s.=c)
Prediction: Compute the counterfactual quantity based on the modified SCM and p(elz)
Definition 2 (Pearl's Causal Hierarchy (Pearl, 2019)) The observational (I1), interventional (Iz)), and counterfactual (La) distributions entailed by the SCM form a hierarchy in the sense that L C. L2 C La, where each level encodes richer information that the previous level cannot express. The three layers are collectively referred to Pearl's Causal Hierarchy (PCH).
On Pearl's Causal Hierarchy. L1 (observational) is concerned with modeling mere statistical correlation and does not in itself provide a causal perspective. L2 (interventions) focuses on manipulating system of causal variables to observe downstream effects on other variables. Interventions are often performed using the do-operator and are population-level operations. Finally, L3 (counterfactuals) is the most complex layer and deals with retrospective thinking. That is, counterfactuals are unit-level queries that imagine alternative situations contrary factual observations under a specific intervention. For instance, question such "Would T have gotten A grade my exam had studied for more hour?" would be counterfactual query. An illustration of Pearl's causal hierarchy is shown in Figure 3.
2.2 Variational Autoencoders
The variational autoencoder (VAB) (Kingma & Welling, 2013), shown in Figure 4a, is framework for optimizing latent variable models inspired by the theory of variational inference and energy-based models. Similar to a standard autoencoder, the VAE is composed of both an encoder and decoder and is trained to minimize the reconstruction error. However, instead of encoding an input as a single point, the VAE encodes it as a distribution over the latent space. Formally, suppose the observed data x is mapped to some low-dimensional latent space via an encoder X + and reconstructed by a decoder g Z + X. The goal of the VAE is to approximate the likelihood of the observed data, namely Pe(œ), parameterized by 0. Given some prior over the latent variable po(z), can introduce variational distribution (p(2|>) with parameters 6 that approximates an intractable posterior p=(2)z). Since cannot tractably estimate the exact likelihood, we alternatively maximize the evidence lower bound (ELBO), which is lower bound to the log-likelihood the data formulated as follows:
(ogp 2 (xla)) log - D (=x(((((((((()))((((
(6)
where the first term corresponds to the reconstruction loss (likelihood) and the second term encourages the learned posterior to match a suitable prior of the latent variables.
2.3 Normalizing Flows
Normalizing flows (Tabak & Turner, 2013; Tabak & Vanden-Eijnden 2010; Papamakarios al., 2021). shown in Figure 4b, are expressive and general way of constructing flexible probability distributions over

---

Published in Transactions on Machine Learning Research (05/2024)
x
aΦ (2|x)
2
(x(x22)
8
(x)
T(x)
8
(a) VAE
(b) Normalizing Flow
((z)
&
0
t t=7
(Noise)
x
((x)
0/1
X0
X2
XT
Ps
(c) GAN
(d) Diffusion
Figure 4: Overview of generative models (a) Variational Autocncoder (VAE), (b) Normalizing Flow, (c) Generative Adversarial Network (GAN), and (d) Diflusion models
continuous random variables. Flow-based models can transform simple base distribution, e.g., Gaussian, into a complex distribution that resembles the distribution of the underlying data. Unlike VAEs and GANs, flow-based models explicitly learn the data distribution and thus the loss function is simply the negative log- likelihood. Consider the observed data æ and suppose, similar to the VAE, that we would like estimate the likelihood of this data, In flow-based models, we express as an invertible transformation of some real-valued variable follows:
x T(2)
(7)
where 2 is sampled from some distribution p(2), known as the base distribution of the flow-based model. Further, we require that. T is a diffeomorphic function and that = is the same dimension as x. The densily of well-defined and can be estimated exactly by a change of variables:
p(z) = p(=) det JT()))
(8)
equivalently,
p(x) = -0 det JT(T
(9)
The Jacobian, which practice can be efficiently computed by restricting it to a triangular structure, is a d matrix consisting of all partial derivatives the transformation T
dz
3zd
(2(2) =
(10)
8Ta
8Ta
Ba1
8ag
Flows can be thought of as warping space to mold the base density p(z), which is usually a simple distribution such Gaussian, into complex distribution p(x)). The determinant of the Jacobian is the volumetric change in due to the transformation T.. Flow-based models are capable of both generating new samples from the data distribution by sampling from the base distribution and computing the forward transformation T and exact density estimation by computing the inverse transformation 7-1, Normalizing flows are easier to converge but less expressive when compared to GANs and VAEs bocause the transformations noed to be invertible with a tractable Jacobian determinant.
2.4 Generative Adversarial Networks
Generative adversarial networks (GANs) (Goodfellow et al., 2014), shown in Figure 4c, are an example of implicit generative models, which are capable of sampling from probability distributions but cannot learn the likelihoods for data points. Thus, GANs are quite different than VAEs in their applications. GANs utilize an adversarial training procedure to produce realistic samples from distributions over a very high dimensional space, such as images. To learn the generator's distribution Pg over data æ, prior is defined input noise

---

Published in Transactions on Machine Learning Research (05/2024)
variables p(2) that is mapped to the data space via G(2), where G is a differentiable function (implemented as multilayer perceptron) known as the generator network. The goal of G is to generate samples close to the input distribution p(x). Another function Dxx) (implemented as multilayer perceptron), known as the discriminator network, is defined to represent the probability that z indeed came from the data distribution Pdata rather than the generalor Py. The discriminator of assigning the correct label both training examples from input distribution and samples generated from G. Simultaneously, G trained to minimize log(1 D(G(=))). Formally, G and D play the following minimax zero-sum game:
trained to maximize the probability
min max (u D())] + pe=p(=) [log(1 - DDGG()))
(11)
To optimize with respect to both networks, gradient descent is performed to update the generator and gradient ascent is performed to update the discriminator. GAN models are known for potentially unstable training and relatively less diversity in generation due to the adversarial training process.
2.5 Diffusion Models
Inspired by non-equilibrium statistical physics (Jarzynski, 1997)), diffusion models (Sohl-Dickstein et al., 2015; Ho et al., 2020), shown in Figure 4d, were proposed as a likelihood- approach to achieve both flexible model structure and tractable estimation of probability distributions. Unlike VAEs, diffussio models are learned with a fixed procodure, and the latent variable has the same dimension as the original data. Diffusion models have shown impressive results in image generation tasks, even oulperforming GANs many cases (Dhariwal & Nichol, 2021). The idea of the diffusion model is to define a Markov chain of diffusion steps to slowly destroy the structure in data distribution through forward diffusion process by adding noise (Ho al., 2020) and learn reverse diffusion process that restores the structure of the data. Some proposed methods, such DDIM (Song et al., 2021a), break the Markov assumption to speed up the sampling in the diffusion process.
Forward Diffusion. Given some input data sampled from a distribution 2g ~ q(x), the forward diffusion process is defined by adding small amounts of Gaussian noise to the sample in T steps, thereby producing noisy samples T7. The distribution of noisy sample I, is defined a conditional distribution follows:
q( N(xx /1 - 1,01-1,
(12)
where Br E (0, 1) is a variance parameter that controls the step size of noise. As + co, the input sample To loses its distinguishable features. In the end, when = T, Fr is equivalent to an isotropic Gaussian. From Eq. (12), we can then define a closed-form tractable posterior over all time steps factorized as follows:
T
((tr)())) = II q(
(13)
Now, 4e can be sampled at any arbitrary time step t using the well-known reparameterization trick (Kingma & Welling, 2013; Rezende al., 2014). Let a = 3e and ã₂ -[ a;:
q(ze(1o) N(xx Vanza, (1 (1-ā+)])
(14)
Reverse Diffusion. In the reverse process sample from ((=1-1/4)), the goal is to recreate the true sample Zo from Gaussian noise input IT N/(0, T). Unlike the forward diffusion, q(x)-1/2)) is not tractable and thus requires learning model pa to approximate the conditional distributions as follows:
T
Po( = p(aT) II Poo
(15)
p0( Ma (z t), Eo (z(,
In the context of diffusion models, the Markov assumption refers distribution over 2t depending only on 2t-1
9

---

Published in Transactions on Machine Learning Research (05/2024)
where Me is learned via neural networks, typically a UNet (Ronneberger et al., 2015). Further, it turns out that conditioning on the input zo yields a tractable reverse conditional probability
g (2 zo),
(16)
where ji and 3 are the analytical mean and variance parameters derived as a result of conditioning on 20- We can learn the reverse denoising process by optimizing the following variational lower bound:
Bllog Po (zo)] E log {(11)
LVLB
(17)
pa pa(o))))
For detailed derivation, see Sohl-Dickstein al. (2015). Diffusion models often require longer training and sampling times, making them computationally expensive.
2.6 Disentangled Representation Learning
For the longest time; the efficacy of machine learning models relied human ingenuity and prior knowledge to perform feature engineering on data. The realization that such engineering is inefficient and demands a more automated approach gave rise to the field of represenlation learning (Bengio el. al., 2013). Representa- tion learning aims to extract meaningful information (or features) from data that can be used downstream to build robust classificrs or predictors. Thus, it is desirable to build representation-llariin algorithms that incorporate cerlain priors about the world to automatically learn useful representations. In the probabilistic sense, which what we focus on in this survey, a good representation is one that should ideally learn the posterior distribution of the underlying factors of variation for the observed input distribution. Further, representation that is disentangled such that each factor is independently manipulable enables robust representation useful for downstream tasks such as learning fair and robust predictors (Tocatello et al., 2019a).
The problem representation learning can be linked to nonlinear ICA (Hyvärinen Pajunen, 1999). Suppose that our low-level observations x E Rdd are explained by small number of latent variables (21...., 2n), where < d and x is generated by applying an injective (or sometimes diffcomorphic) map g : R Rdd also called the miring function, to such that
I -= (2))
(18)
In nonlinear ICA disentangled representation learning, assumed to be mutually independent such that
II ((2)
(19)
Although there is one commonly accepted definition of disentanglement, Locatello et al. (2020) propose to formalize the notion in the following general definition:
Definition 3 (Locatello et al., 2020)) The goal of disentangled representalion learning is to learn mapping g (x) such that the effect of the different factors of variation is aris-aligned with different coor- dinates. That is, each factor of variation is associated with exactly one group of coordinates of (x) and vice-versa, where the groups are non-overlapping. Thus, varying one factor of variation while keeping the others fixed results a variation of exactly one group of coordinates.
Several proposed VAE variants, such as 3- 3-VAE (Higgins et al., 2017) and FactorVAB (Kim & Mnih, 2018), have proposed objectives disentangle the learned latent codes via independence constraints. However, such models fail to disentangle factors and do not address one major issue in representation learning: unidentifi- ability of representations.
2.6.1 The Identifiability Problem
A significant barrier to representation learning the identifiability problem. Probabilistic generative models suffer from indeterminacy, which refers to the situation where the underlying factors of variation cannot. be
10

---

Published in Transactions on Machine Learning Research (05/2024)
umiquely inferred from empirical data. Characterizing and reducing these indeterminacies is the endeavor of identifiabilly.. Identifiability is an important property to achieve in representation learning since it guarantees the stability and robustness of model. When different representations can explain the same observational data equally well and cannot identify the true factors of variation, we say that the model is unidentifiable. In general, it is infeasible to always uniquely identily the underlying factors and remove all indeterminacies since it is often task-dependent. Thus, identifiable representation learning typically aims to recover the true factors (and mixing function) up to tolerable ambiguities (i.e., transformations, reordering, etc.). It is important to note that model identifiability is an asymptotic property that can be achieved in its strictest form only in the limit of an infinite amount of data (Xi & Bloem-Reddy, 2023). However, weaker identifiability results often suffice to recover ground-truth factors. The theory of identifiability stems from the literature on nonlinear independent component analysis (Hyvärinen & Pajunen, 1999). It has been shown that the identifiability of nonlinear ICA is, in fact, not possible unless certain restrictions are made on the mixing function or other auxiliary information is provided (Hyvarinen ct al., 2018). Locatello et al. (2019b) extend this result to representation learning and show that learning disentangled representations in an unsupervised manner without any inductive biases is impossible. It turns out that the notion of disentanglement and identifiability are actually quite related. Formally, following Khemakhem et al. (2020) and Gresele et al. (2021), identifiability is defined as follows:
Definition 4 (Identifiability) Let G be the set of all smooth, mixing functions g : R" + RR, and P be the set of all smooth, factorized densities p(z). Let Q G x P be the domain of parameters, where the elements of θ paramelerize the miring functions and densities, and let ~ be an equivalence relation on O Then, the generative process in (18) is said to be ~-identifiable on θ if
v0, θ E
pa(x) pz(Ø) 0~~
(20)
If the true model belongs to O then identifiability implies that any model in O learned from infinite amounts of data will be ~-equivalent to the true model. In the context of the VAE? if two different choices of model parameters 0 and 0 Icad to the same marginal density Po(x), then they are equal and the joint distributions should match (i.e., po(2; =) = p)(x, If the joint distribution matches, then we have found the correct priors p=()) = Pz(x) and correct posteriors pe (=|2)) = pz<]<>)) Practically speaking, identifiability essentially guarantees that regardless of how many times we train the representation learner, we will always obtain the same unique solution (up to tolerable ambiguitics). Thus, if identifiability didn't hold, learning representations would not be as useful for downstream tasks.
Linear ICA is well-known to be identifiable given Gau-Gassian noise (Comon, 1994). That is, there exists a unique unmixing function that maps the given observations to a unique set of sources. However, in the general case, nonlinear ICA is ill-defined and not equivalent to blind-source separation (BSS). That is, in the n = case, different mixtures of 2 and žj can be independent. Some classic counterexamples include the Darmois construction (Darmois, 1953; Hyvärinen & Pajunen, 1999) and rotated Gaussian measure- prreerrring automorphisms (Gresele et al., 2021). The following two examples show why nonlinear ICA is unidentifiable in the general case and the need for additional signals and strategies to resolve identifiability.
Example 1 (Darmois Construction - (Hyvärinen & Pajunen, 1999)) The Darmois construction g.: R + (0, 1)" is defined as the recursive application of the following conditional CDF:
( = p(X, S 22|= p(alaa p(a((=1=-1) dat
(i=
(21)
The Darmois solution maps any observed density to a uniform distribution using the CDF above, as shown in Figure 5a. The estimated sources are thus independent of the conditioning variables T1:-1- Thus, y =g (x) are mutually independent uniform random variables that may not be meaningfully related to the true sources of variation. Let the mixing function be defined as fd () and the uniform density
Recall that the VAE model learus a full generative model (o(x, 2) px(x)))po and an inference model (z|ax) which approximates its posterior po(2]2)). However, we generally have no guarantees about what these learned distributions actually are and only the marginal distribution over is meaningful.
11

---

Published in Transactions on Machine Learning Research (05/2024)
22
72 0.8
rotation
Z1
Z1
P.O2)
y g(x) P(X
(+ =1
72
7/2
(a) Darmois CDF Transform
(b) Rotated Gaussian
Figure 5: An illustration of (a) the Darmois construction and (b) rotated Gaussian MPA that are coun- lerexamples to blind-source separalion of nonlinear ICA. The Darmois CDF transform maps the data to independent intervals of uniform distribution. The Gaussian is invariant to rotations and thus represents the same observational distribution after rotation (the left circle and right circle are equivalent distributions).
the interval (0, 1)a by Pu- Then, the Darmois solution (f..Pu) is counterexample to blind source separation since we can always find another set of independent sources that explain the data equally as well. For instance, consider a twovvariable case where have independent random variables ø1 and z2. Then, we have that Y1 = 91 (Z1, and y/2 92(21,22) are independent from the original observed variables xn and 22. Thus, we can construct infinitely many such independent random variables and can never recover the unique sources.
Example 2 (Rotated Gaussian MPA - (Gresele et al., 2021)) Another class of counterexamples in- volves measure-preserving automorphisms (MPA), which are identity functions on the sources that pre- serve the distribution of the source variables. Let R be an orthogonal matrix, and denote by F2(z) = (Fz, (21)...., Fr, (2n)) and ((2) = (Ø(21)..... ((2))) the elementwise CDF of smooth, factorized density Pz and of Gaussian, respectively. Then, the rotated Gaussian MPA a (x(pz) is
a <<(p.) = Fz o Ro.Ro Ro
(22)
a (p()) maps to a slandard Gaussian (rotationally invariant), then applies a rotation, and finally maps back to the sources. In this case, there exists a different set of independent sources through a (p=))) that explain the data equally as well. Interestingly, principal component analysis (PCA) is not identifiable precisely because it is a rotation of a Gaussian through the orthogonal rotation matrices U and V obtained through singular value decomposition of the data covariance matrix. Figure 5b shows the rotational indeterminacy of Gaussian i.i.d. models, where each point in the circle represents equivalent solution. In other words, the joint distribution will always be equivalent no matter what rolations are performed on the Gaussian, uniquely identifying the sources is hopeless.
Therefore, additional constraints and assumptions are necessary if we hope to uniquely identify the factors of variation. The goal is to get closer to recovering the true underlying factors up to some trivial transformation. Under certain conditions, we can often recover the underlying factors up to some affine transformation of the learned factors as formalized in Definition 5.
Definition 5 (Affine equivalence) We say 0 and 0 are affine equivalent if
(g.Pz) ~ (g.
p2) - LL s.t. (g.pz) = (g o -- -- pp))
(23)
where L a linear invertible transformation.
However, to disentangle the sources, we must go beyond affine equivalent identifiability since it allows for any lincar combination of learned factors recover underlying factors, which leads entangled solutions. For most settings, the best we can do is recover the factors up to simple reordering (permutation) and scaling (elementwise reparameterization)) of the learned factors formulated in Definition 6.
12

---

Published in Transactions on Machine Learning Research (05/2024)
Z1
21
Z1
22
Z2
22
Z2
(a) Affine equivalence
(b) Permutation equivalence
Figure 6: A simple 2-variable example of (a) affine equivalence vs. (b) permutation equivalence, where ž denotes the learned latent factor and 2 denotes the ground truth underlying factor. Note that the arrows in the diagram indicate mappings, not causal relations.
Definition 6 (Permutation equivalence) We say 0 and i are permutation equivalent if
(g. ~p (g.Ps) + 3.th s.t. (g.Pz) h . pp:)
(24)
where P is permutation matrix and h(2) (h1(21), .... hn (za)) invertible elementwise reparameteriza- tions. A learned model 0 is said to be disentangled if 0 ~p 0.
According to the above definition, it turns out that identifying latent factors up to permutation (and possibly elementwise reparameterization) of the ground-trutle is equivalent to disentangling the factors of variation. Componentwise equivalence is a special case of permutation equivalence where P = T (identity permu- tation). Block identifiability is a form of identifiability where a group of latents is uniquely identifiable, but latents within the group may not. be. Since disentanglement is impossible to achieve without some form of inductive bias, we next briefly discuss works that propose additional weak supervision and other strategies learn provably identifiable representations. An illustrative example of affine and permutation equivalence is shown in Figure 6.
2.6.2 Towards Identifiability in Disentangled Representation Learning
The natural question becomes: Can we utilize additional structure in data to attain identifiability? The following approaches have been the most promising strategies for dealing with identifiability challenges in representation learning. Many of the identifiability results in disentangled and causal representation learning utilize the general structure of the following proposals.
Latent Conditioning. Based on Hyvarinen al. (2018), Khemakhem et al. (2020) propose a unified theory connecting VAEs to nonlinear ICA called identifiable VAE (iVAE). The authors propose conditioning the latent variable on some auxiliary information, such as time steps or class labels, with an exponential family prior such that the underlying factors are rendered conditionally independent:
k Σ j=1
(r.x(=lv) II Po (z II (x(2)) exp
exp
(.)(s,))),,)))>> -v((())))))
(25)
where h2(2s) is the base measure, T. 3 and T:= = (T......, Tikk are the sufficient statistics, \v()) = (a),...,,>,..... (v)) are the corresponding natural parameters, the dimension of each sufficient statistic, and the remaining term (y) acts as a normalizing constant. Consider the well-known Gaussian distribution, which is in the exponential family. We have that the sufficient statistic is T+(=) = (z, =²) and the base measure is hx(2) 1. The function outputs the natural parameter vector for the conditional distribution, which V2=
is (A1,22) (#4, 14.-11 --) in the Gaussian case. Auxiliary label information (e.g., class label, domain label, time index, etc.) enables certain independence and distributional assumptions to be made on the underlying latent variables, which can be utilized to identify the latent factors.
Multi-view Data and Contrastive Learning. Multi-view nonlinear ICA (Gresele et al., 2019) posits that multiple views of obscrvational data may have a shared latent representation. This paradigm has been utilized in several different ways for disentanglement and identifiability guarantees. Inspired, by this idea, Locatello et al. (2020) propose considering pairs of contrastive data (21, consisting of before and after
13

---

Published in Transactions on Machine Learning Research (05/2024)
changes (interventions) have occurred to system as an inductive bias for representation learning. The generative model is defined as sampling two images from the causal generative process with an intervention on a random subset of factors of variation. The auxiliary information in this setting is the positive pair. Let k be the number of factors in which the two observations differ, S denote the subset of shared factors of size n - ki and S = [x] S. Then, we have the following generative process:
22)) = p(Z))
p(2) = II p(zi,, i=1
S S~p(S)
(26)
t=1
T1 = (z),
12 = 9" 9" (f(2, 9f(2" S))
(27)
f(,,2, = 28 and
(28)
where 9" is a function that maps the latent. variable to observations and f makes the relation between Fi and I2 explicit. Intuitively, when generating z2, f selects entries from 2 with index in S and replaces the remaining factors with 2. Thus the two observations 21 and zg have the same shared factors indexed by S. Contrastive learning has been recently shown to provably invert the data-generating process (Zimmermann et al., 2021). In the case of contrastive learning, the data from multiple views provides sufficient variability in the data to uniquely identify the factors of variation.
Restricting Mixing Function. Gresele et al. (2021) and Reizinger et al. (2022) propose a new theory called independent mechanism analysis (TMA) to learn identifiable representations based on restricting the function class of the mixing function and placing orthogonality constraints on the columns of the Jacobian of the mixing function. Inspired by the Independence of Causal Mechanisms (ICM) principle from causality (Scholkopf et al., 2021), the authors propose new form of independence between the influences of sources on the observation rather than statistical independence. Specifically, the authors restrict the mixing function af for all i = (1....,,1) n). Formally, we have that the mechanisms by
I to satisfy mutual independence of F2, which each source 2; influences the observed distribution are independent of each other in the sense that for all
T
as
log [)(())) = Σ log
(2)
(29)
az;
This implics that the absolute value of the determinant of the Jacobian should decompose as the product af Reizinger et al. (2022) show that the evidence lower bound in VAEs converges to a of the norms of 8z," regularized version of the log-likelihood. The regularization term gives VAEs the ability to perform inde- pendent mechanism analysis and serves as an inductive bias for decoders with volumn-orthogge The authors show that the regularized objective enables the identifiability of the ground-truth latent factors. Further, they show that VAEs satisfy the relf-oonissency property in near-delerministic regimes (i.e., the VAF encoder inverts the decoder even if the VAE is not exactly deterministic). Assumptions on the mixing function (linearity, polynomial, etc.) can also help to narrow down the space of solutions by assuming a simpler data- generaenng process.
Jacobians.
Sparsity. Recently, sparsity principles have been proposed to promote disentanglement (Zheng et al., 2022; Lachapelle et al., 2022; Zheng & Zhang, 2023; Xu et al., 2023). Zheng et al. (2022) propose Structural Sparsity, a new theory of nonlinear ICA with unconditional priors by leveraging sparsity the relations between sources and observations (i.e., mixing). The general intuition behind structural sparsity is that for each latent factor of variation 2,, there exists set of observed variables in x such that 2, is the only latent source that is used to generate the set of observed variables. In addition to sparsity in the mixing, there are other applications of sparsity such as modeling sparse mechanisms between latent variables (Lachapelle et al., 2022; Lachapelle & Lacoste-Julion, 2022) and the use of a sparse number of latents for downstream prediction tasks to promote disentanglement (Lachapelle et al., 2023).
For a more rigorous treatment of identifiability and indeterminacies in generative models, we refer the reader to Xi & gloem-Reddy (2023). Typically, in identifiability proofs, the goal is first show linear-equivalent identifiabiliy under a certain set of assumptions and then leverage independence or sparsity structure to show permutation-eqqiiablee identifiability (ddiiiiiitttyiiitttageeeeee (disentanglementt.
14

---

Published in Transactions on Machine Learning Research (05/2024)
Approach
Mixing Function
Causal Model
Key Assumplions
Identifiability Result
CansalVAE" (Yang al., 2021)
nouparametric, injective
Enear additive noise model
observed
componentwise
auxillary labels
correspoudence
DEAR" (Shen nl., 2022)
nonpaaametric, injective
nonlineor
auxiliary labels,
observed
componentwvie correspondence
causal prior
SCM-VAE (Komanduri et ul., 2022)
nonprammttric, injective
nonlinear
auxiliary labels,
obsorved
componentivise correspondence
causal prior
ICM-VAE' (Komanduri et al., 2023)
nonparametric, diffeomorphic
observed
causal mechanism equivalence, permutation equivalence
nonparametric
auxiliary lahels, cansal diseutanglement prior
Unpaired Multi-domain CRL (Sturma et 2023)
permutaatio equivalence via lincar ICA each domain permutation oquivalence
linear, injectivee
data from d' umpaired domsins
lincar
G-CaRL"
M disjuint groups data
(Morioka & Hyvärinen, 2023)
diffoomorphic diffeomophhie, diew-speclie
nonparametric
Multi-view CRL (Yao et al., 20/33)
multi-view data,
block identifiabiity shared latents
nonparametric
subset latents
for each modality
LEAP (Yan et al., 2022h)
nonparametric, injective
nonparametric & parametric cases
temtorral observational
permutation equivalence of temporal latents
data, nonstationarity
auxiliary labels
TDRL* (Yao et al., 2022a)
nonparametric, injective
temporal observational
permutation equivalent of temporal latents
nonparametric
data, stationarity
observed nonstationarity
NCTRL* (Song al., 2023)
nonparametric, injective
temtoraa obvervational data, unobserved noustationarity
permutation equivalence of temporal latents
nonparametric
Interventional CRL (Alujj et al., 2023)
polynomial, injective
nonparametric
perfect
permutation oquivalence
intervention per node
Linear CRL
perfect intervention
(linear mixing)
linear, injective
linear
per node (umpaired),
permutation equivalence
(Squires ol., 2023)
unknown interv. targels
Linear CRL
perfect intervention
(general mixing)
nonparametric, diffccmorphic
linear
per node (unpaired),
permutation equivalence
(Buchholz et al., 2023)
unknown interv. targets
Discrepancy-VALE VAEE (Zhang al., 2023b)
polynomial, injective
soft intervention
permutation equivalence of ancestral relations
nonparametric
per node (unpaired),
unknown interv. targets
Nomparametric CRL (von Kiigelgen et al,, 2023)
nomarametric,, diffcomorphic
nonparametric
2 perfect interventions
per node (paired).
permutation equivalence
unknown
targets
Parametric CRL
hard Intervention
(Score-basee based) [Variet al., 2023a)
linear, injective
nouparametric
per node (unpaired)/
permutation equivalence/ perfect DAG recovery
1 soft intervention per node
Nomparametric CRI.
2 hard interventions
(Score-baste (Varicı al,, 2023b)
nonparametric, diffoomorphic
nouparametric
per node (umpaired),
permutation equivaleuce
unknown interv. targets
CITRIS (Lippe ol., 2022b)
nouparametric, bijective
nomparametric
temporal sequences,
recover minimal
known interv. targets
causal variables
iCITRIS' (Lippe ot al,, 2023a)
nomparametric, bijective
nonparametric temporal sequencrss, known interv. targets, instautaneous effects
minimal
cansal variables and parents
BISCUIT (Lippe et al., 2023b)
nouparametric, diffoomorphic
nonparametric tempural sequences with action variable, unknown
permutation equivalence
given binary
targets
interaction pattern
Mechanism Spansity Regularization (Lachapelle al., 2022)
permutation equivalenoe
nonparametrie, diffeomorphie
nonparametric tempural sequences with action variable
given sparse tempora action cansal graph
SSL Data Augmenttttion CRL
nouparametric,
augmented counterfactual data
block ideutifiability of content latents
nonparametric
(von Kilgelgen et al., 2021)
paired counterfactual data, perfect interventions, unknown interv. targets
LCM
nonparametric, diffeomorphic
nonparametric
permutation equivalence
(Brehmer et al., 2022)
of the main contributions is a learning framework.
Table 2: Summary of Causal Representation Learning Identifiability Results
15

---

Published in Transactions on Machine Learning Research (05/2024)
3 Causal Representation Learning
Until recently, most representation learning methods assumed the underlying factors of variation to be in- dependent (Khemakhem et al., 2020) and that the data is independently and identically distributed, which is often impractical in real-world scenarios. The world is full of dynamically changing systems consisting of complex interactions that induce correlations or causal dependencics between factors. Träuble et al. (2021) show that existing methods are not sufficient to disentangle the factors when the data is non-iid and factors are correlated. Thus, a more realistic endeavor would be to design models that assume the factors underlying a generative process are causally related. Combining causal modeling and representation learning allows one develop machine learning models for high-dimensional inputs whose underlying factors are governed by an SCM. The goal of causal representation learning is to learn high-level causally related factors that describe meaningful semantics of the data and their causal structure (Scholkopf et al., 2021). Semantically meaningful can refer to several properties of a representation, such robustness, fairness, transferability, interpretabil ity, or explainability. Causal representation learning (CRL) is an amalgamation of multiple lines of work, including identifiable representation learning, causal structure learning, and latent causal DAG learning. this survey, we do not cover causal discovery methods and refer the reader to other surveys that comprehen- sively cover this landscape (Vowels et al., 2021; Squires & Uhler, 2022). However, we note that since causal structure learning is component of CRL, many methods incorporate existing heuristic structure learning methods from observational data (Zheng et al., 2018) or combination of obscrvational and interventional data (Lippe et al., 2022a; Ke et. al., 2023). The key feature of causal variables is that they are variables on which interventions are defined and whose relations are of interest. It is worth noting that independent factors of variation are special case of causal models where the causal graph is trivial. In the following sections, we build up causal representation learning from nonlinear ICA and discuss methods according to their assumptions on the data-generating process: observational, interventional, or counterfactual. Table 2 summarizes the identifiability results, key assumptions, and main contributions of the causal representation learning methods discussed. The key assumptions are generally listed the lype of data assumed (obser- vational, interventional, counterfactual), the parametric form of the latent SCM, and the type of mixing function, in addition to any method-specific assumptions.
3.1
Formulating Causal Representation Learning
3.1.1 Linking Causal Representations to Nonlinear Independent Component Analysis
Recall that in traditional ICA, the ground-trute factors are assumed to be statistically independent. For the sctting of cansal representation learning, we diverge from the independent factors of variation assumption and suppose that 2, are causal variables that may be dependent. The joint distribution for 2 can thus be expressed as the following Markov factorization
p(21....,22) = p p2.l(ppa))
(30)
induced by an underlying acyelic SCM C E, F,P)) with jointly independent noise e; and a set of causal mechanisms
F F= (z= (((=aa, {:)
31))
The goal of causal representation learning is to (1) learn the causal represenlation = = (x), (2) the corresponding causal DAG, and (3) the causal mechanisms p(zlkpp,) (Parascandolo et al., 2018; Bengio et al., 2020). Assuming reduced form3 SCM where we recursively substitute the structural assignments in the topological order of the causal graph, we can write the latent variables 2 directly as a function of the noise
= == =RF
(32)
A reduced form SCM one where the mechanisms are abstracted away obtain causal variables function of noise terms (i.e., = f(r)).
16

---

Published in Transactions on Machine Learning Research (05/2024)
Cause
Effect
XA
XT
Figure 7: Relationship between altitude and temperature (Peters et al., 2017)
The mapping far: : R + R" has a lower triangular Jacobian. So, we can write the mapping from noise to data space as the following composition
= g fRF(C)
(33)
Learning (32) can be seen as a structured form of nonlinear ICA with an intermediate representation learned through far (Schölkopf & von Kügelgen, 2022). There have also been recent efforts in generalizing ICA to a theory of causal component analysis (Liang al., 2023) as a special case of causal representation learning where the causal graph known.
3.1.2 Principles Causal Representations
The following two principles describe the key properties of causal representations.
Principle (Independent Causal Mechanisms (ICM) (Scholkopf al., 2021)) The causal gen- erative process of system's 's variables is composed of autonomous modules that do not inform or influence each other. In the probubilistic case, this means that the conditional distribulion of each variable given ils causes (t.e,, its mechanism) does not inform or influence the other mechanisms.
The independence of causal mechanisms is statement about the algorithmic independence between mech- anisms that generate causal variables. That is, the ICM principle implics the locality of interventions where changing a variable only affects the causal mechanism for that variable. In other words, the components, or mechanisms, are modular, autonomous, and reusable. Statistical dependencies are a result of introduc- ing unexplained random variables into the system. However, a deterministic system consists of physical mechanisms. As a result, an algorithmic model of causal structure can be devised in terms of Kolmogorov complexity (Janzing & Schölkopf, 2010), which is the length of a bit string's shortest compression a Turing machine and mcasure of its information content. However, the Kolmogorov complexity is known not to be computable. Thus, must resort lo stalistical and other information- notions of quantifying independence.
A two-variable example of ICM Principle (Peters et al., 2017). In the two-variable case, whore one is referred to as the cause and the other as the effect, Principle boils down to independence between the cause distribution and the mechanism producing the effect from the cause. The factors are independent in two senses: (i) intervening on one mechanism p(=;|=pa;) docs not change the other mechanisms pl p(=)l=pa,) for # j, and (ii) knowing information about some mechanism p(2)l=pa,) does not give any information about mechanism p(z)lāpa;) for i # j. Suppose we are interested in analyzing average annual temperature Xy vs. altitude XA from different locations. Altitude and temperature are correlated variables. Intuitively, we can infer that this correlation must arise from the fact that changes in altitude cause changes in temperature. For example, at higher altitudes, the temperatures tend to be lower.
Figure 7 illustrates the relationship between the altitude and temperature variable at given location. Suppose collect two datasets from location R and location S. Then, can say that pn(XA. # Ps(XA, Xr) since the altitude marginal distributions likely differ, i.e., PR(XA) * PS())). However, the conditionals (or mechanisms) that characterize the physical mechanism of generating temperature from altitude, p(X)(XA), will likely be almost invariant to shifts in the marginal, ie.,, (s(X7]XA) Re (X))x(X)). Thus, if we assume that XA + Xr, in the joint factorization p(XA,Xr:) = p(XA)p(XX/a)) that the mechanisms are independent and do not influence cach other. However, if we assume an alternate factorization, where Xy + XA, namely ((A,. = (xX we do not enjoy the same invariance property. That is, it would not make sense for the temperature cause a physical change in the altitude and
we can say
17

---

Published in Transactions on Machine Learning Research (05/2024)
would not observe much change in altitude when intervening on temperature. Thus, the correct causal lactorization will always yield independent causal mechanisms.
Principle 2 (Sparse Mechanism Shift (SMS) - (Scholkopf et al., 2021)) Small distribution changes tend to manifest themselves in a sparse way in the Markov/disentangle factorization in Eq. (30), i.e., they should usually not affect all factors simultaneously.
Given that the ICM principle holds, the SMS hypothesis is a statement about distribution shifts. That is, distribution shifts (e.g., the training and testing set may come from different distributions) can often be attributable to sparse changes in causal conditionals of a causal disentangled factorization (or interventions on causal variables). To learn models that are generalizable to distribution shifts, it is desirable to learn causal representations due to their invariance to distribution shifts. There have been several works that use causal models to study distribution shift robustness (Lu et al., 2022) and how to mitigate spurious correlations (Wang & Jordan, 2021).
Guided by these principles, causal representation should consist of independent mechanisms and must be robust to downstream tasks under distribution shifts.
3.1.3 Intuition behind Causal Disentanglement
The notion of causal disentanglement can seem counterintuitive or even contradictory. As mentioned before, disentanglement is not completely well-defined concept. One approach could be to think about it in terms of simply enforcing statistical independence among all the learned factors in an unsupervised fashion, as done in V-E (Higgins et al., 2017). However, such an approach does not isolate the learned factors and semantically meaningful manipulations are difficult to perform; hence, the impossibility implication for arbitrary unsupervised disentanglement (Locatello et al., 2019b). Thus, Definition 3 aims to formulate a consistent definition through a non-overlapping condition on the information that each group of latent codes encodes. Causal representations adhere the principle of independent causal mechanisms (Principle 1). Thus, causal disentanglement should be viewed through the lens of the ICM principle. In setting where the factors of variation are causally related, the mechanisms that produce cach causal variable as a function of its parents should be independent. Thus, we must ensure not only that each group of latent codes captures distinct information but also the recoverability of each independent causal mechanism (Komanduri et al., 2023) and the causal structure among factors (if learned). In a setting where the causal graph is learned jointly with the representation, the recoverability of the causal relations can be formulated as a graph isomorphism between the true causal model and the learned causal model (Brehmer et al., 2022).
3.1.4 Identifiability Problem in Causal Representation Learning
In causal representation learning, on top of identifying the latent representation, the causal graph encoding their relations must also be identifiable. The task of causal structure identification is already difficult in the purely observational setting since we can only recover DAG up to Markov equivalence (Spirtes et al., 2000). However, it gets significantly more challenging when jointly learning latent causal variables. The increase in complexity from traditional disentangled representation learning to causal representation learning is primarily due to this joint identifiability of latent causal factors and the latent causal DAG. In traditional disentangled representation learning, one may be able to utilize strategies such as contrastive learning to disentangle factors (Zimmermann et al., 2021). However, in causal representation learning, we may need data generated according to some causal model (e.g., interventional or counterfactual data) to recover the causal mechanisms among causal factors. Additionally, discovering the causal structure and extracting causal variables becomes a chicken-and-egg problem, which makes CRL severely ill-posed. That is, how do we Iearn the causal structure without the causal variables, and how do we Iearn the causal variables without knowing the structural relationships among causal variables? This dilemma is what makes causal representation learning a challenging endeavor. In the following sections, we summarize work done towards achieving identifiable and disentangled causal representations from various weak-supervision signals and lata-genera assumptions. We primarily focus our attention on VaE-based models used to learn disentangled causal representations, but some methods restrict the mixing function to be linear and thus
18

---

Published in Transactions on Machine Learning Research (05/2024)
only use matrix decompositions. For motivation of the challenges we face in causal representation learning, consider the following simple example that illustrates unidentifiability.
Example 3 (Unidentifiability in CRL Simple Linear Example) Suppose we have observations E I and lalent variables IR² related by Gz, where G is linear mixing function. For simplicity, let the causal variables be related by the following linear additive noise structural causal model
2 A + (I-
6 (~,10
(34)
where A is the causal adjacency matriz. Suppose we have the following solution to the system
0 0
c= - 3 -4
29
-27
A =
E
(35)
-27
26
where E is the covariance of the observations derived consider the solulion
E = E[rx²]
=G(I - -I- Now,
A= =
0 0 0 0
G
5 2 5 -1 _ Σ
29
-27
(36)
27
26
We have that the covariance of the observation is identical for both solutions (i.e., Σ == Σ). solution, assume the structure 22 - 21; in the second solution, assume independent factors of variation. However, we still oblain the same observational distribulion. This implies that the independent factors of variation represent observations equally as well as the causally related factors, and we cannot identify a unique solution.
In the first
The following sections outline causal representation learning methods that rely on certain data-generating assumptions (i.e., observational, interventional, counterfactual) and weak supervision signals to remedy the above unidentifiability issuc and achieve disentanglement causal factors. The majority of causal representation learning methods utilize VAE-based frameworks and maximize a suitable ELBO loss.
3.2 Learning from Static Observational Data
One approach to learning causal representations relies on purely observational data from L of Pearl's Causal Hierarchy. This could be in the form of supervised labels or potentially from different sources. The following methods utilize such auxiliary information to provably disentangle the causal factors this paradigm.
3.2.1 CausalVAE
Learning Framework. Yang et al. (2021) propose CausalVAF, a framework to learn causal representations and causal structure simultaneously given auxiliary information in the form of labels y corresponding to causal factors interest z. CausalVAE assumes a linear SCM with additive noise describing the structure between causal variables, parameterized by causal adjacency matrix A, as follows:
=1==
(37)
where e ~ NoO, T). CansalVAR proposes to encode high-dimensional data into low-dimensional noise vari- ables e and utilize an analytical mapping, (4-A"), to project the noise variables to causal variables z. The weighted adjacency matrix A learned via an acyclicity constraint and the augmented Lagrangian
method (Yu et al., 2019). For identifiability, the labels y are incorporated into a conditional prior, similar Khemakhem et al. (2020), defined as follows:
(=ly)) = II p(z))))) 1=1
p(Z():)) = (()
(38)
where the latent variables are assumed be mutually independent when conditioned on their corresponding labels. The conditional prior is used to regularize the posterior the latent variables using the auxiliary
19

---

Published in Transactions on Machine Learning Research (05/2024)
Encoder
Decoder
SCM Laver
(1 (I-A))-
Z
x
A
y/
21
Independent
22
x
V2
Conditional Prior
24
V4
Figure 8: CausalVAE Framework (Yang et al., 2021)
labels a supervision signal. During inference, counterfactual samples can be generated by encoding image to an exogenous noise term, transforming it to causal variables, intervening on a dimension of the causal representation, propagating effects, and pushing it through the trained decoder to generate a coun- terfactual instance. Although CausalVAB capable of learning causal representations and causal structure simultaneously, the linear SCM assumption often unrealistic in practice. Further, continuous optimization through the acyclicity constraint (Zheng et al., 2018; Yu et al., 2019; Ng et al., 2022) does not necessarily guarantec an accurate learned DAG (recoverable only up to supergraph). The quality of the representation learned is evaluated using the mutual information coelficient (MIC), mutual information measure, which does not capture the disentanglement of causal factors. The overall design of the CausalVAE framework is shown in Figure 8.
Identifiability Result. The authors show that the labels enable the identifiability of causal factors up to component-wis reparameterization. They extend the identifiability result from Khemakhem et al. (2020) and show the identifiability of their framework when given the auxiliary label information.
3.2.2 DEAR
Learning Framework. Shen et al. (2022) propose disentangled generative causal representation learning (DEAR), more general bidirectional generative model (BGM) to learn causal representations by adding a regularization term in the form of a GAN loss (Figure 9). Similar to other approaches, the DEAR framework requires access to all causal variables. They propose SCM prior that derive causal variables based known causal orderings and a supergraph of the true causal graph. The authors propose the following cross-entropy based supervised regularizer to learn to predict each generative factor, thereby taking the place of Gaussian prior, and disentangle the causal factors
Caup
Ex.z [CrossEntropyLLossss =;]]
(39)
Similar to Yang et al. (2021) and Komanduri et al. (2023), cach latent 2; is assumed to be supervised by its corresponding annotated label yi of each ground-truth factor, which they use to show the identifiability of latent factors up to component-wise correspondence. To learn causal representations, the authors propose to learn a nonlinear SCM prior, where causal orderings are given apriori (similar to Komanduri et al. (2023)). The SCM is assumed to follow a post-nonlinear additive noise model, with learnable parameters (f. h, A), follows:
Fg(e) f(11-
(40)
where I and h are elementwise nonlinear Iransformations and f is assumed to be invertible. The causal ordering of the supergraph used as an initialization for causal structure learning using the acyclicity con- straint. Unlike CausalVAE, DEAR does not learn intermediate noise encodings. Rather, noise arbitrarily sampled from a standard Gaussian and mapped to generate causal variables through the learned structural assignments. The authors propose generative model po( 2) po po(x]=))p (=)))))(2)) and stochastic encoder
20

---

Published in Transactions on Machine Learning Research (05/2024)
SCM Prior
h
E1
21
Z2
62
E(x)
Z
G(z)
8
E
73
24
64
Suup ( 2)
Figure 9: DEAR Framework (Shen al., 2022)
(0(2/z) to learn the posterior the latent causal variables. The objective of the generative model is to
minimize the following KT-ddivegeence
Lgen Dx
(41)
with the following variational lower bound
E B (lggsr)) + {
(42)
where p=(=) is the learned SCM prior causal variables DEAR uses GAN-based method to adversar- ially estimate the gradients w.r.t the encoder, decoder (generator), and SCM prior with respective learnable
parameters o, 0, and 8, through specified gradient formulas. The overall loss objective of the generative model and supervised regularizer can be formulated as follows:
L = Lyen + XLeup
(43)
In order to tractably estimate the gradients of Lgen: the authors adopt an adversarial approach using GANs. They train a discriminator D using logistic regression and optimize the following objective
min D'
Σ log(1 +e (0(())))))) + Σ log(1 + e (()nti))
(44)
N₂
Identifiability Result. The authors provide theoretical analysis suggesting that learning from indepen- dent prior will always load to an encoder yielding entangled solution. They also show the identifiability of DEAR up to component-wise correspondences since they assume access to labels as auxiliary informa- tion. Although representations learned using the DEAR method are shown to induce high distributional robustness, there is no quantitative evaluation of the disentanglement of causal factors.
3.2.3 SCM-VAE
Learning Framework. Komanduri et al. (2022) propose SCM-VAE, framework for learning causal representations assuming access to the causal structure. SCM-VAE attempts to remedy the issues from CausalVAE by proposing to learn a post-nonlinear additive noise SCM describing the structure between causal variables
2 = g= + + Vi [I.
(45)
where g is nonlincar function. Further, A is assumed to be binary unweighted adjacency matrix and A; is the ith column of A. SCM-VAE incorporates structural causal prior that induces causal-like factorization of labels
po(2|y) = II po(2i Upa,), (2(22 Upa,) = (N)((((A Inxa); 0)) >3((A + Inxa)s O v)
(46)
21

---

Published in Transactions on Machine Learning Research (05/2024)
Causal Disentanglement Prior
y H
Z1
Z2
4 2/2
Encoder
Decoder
x
- E
Nonlinear Causal Flow
Z
X
V3 H
Z3
4 34
Z4
2 pay Vi)
Figure 10: ICM-VAE Framework (Komanduri et al., 2023)
By leveraging a causally factorized structural causal prior based on the known topological orderings of the causal graph, SCM-VAE addresses a concern of CausalVAE, where the conditional factorized prior simply assumes mutual independence among factors. One issue with SCM-VAE that the Icarned mechanisms are not guaranteed be bijective, which leads issues in optimization and formulating the variational lower bound. Further, the structural causal prior only factorizes the labels, which may be redundant and potentially induce entangled representations.
Identifiability Result. The authors show, similar to CausalVAE, that the supervision signal yields a one-to-one correspondence between learned factors and ground-truth factors. Thus, the causal factors are identifiable up to component-wist reparameterizatiin..
3.2.4 ICM-VAE
Learning Framework. Komanduri et al. (2023) extend the work from Khemakhem et al. (2020) and Yang et al. (2021) and propose ICM-VAE, framework of causal reprosentation lcarning under supervision from labels. The ICM-VAE is based on learning a nonlinear diffeomorphic mapping SRF : E + Z with causal mechanisms of the form
= filess =pa.) = a(zpa + 3(zpa)
(47)
where a and 8 are the neural network parameterized scale and shift parameters of a general nonlinear affine-form autoregressive flow, respectively. The diffeomorphic mechanisms are learned via conditional autoregressive flow mapping a base noise distribution to CAREFL (Khemakhem et al., 2021). This choice is motivated by the efficient and exact likelihood estima- tion and the expressiveness of flow-based models in low-dimensional settings. The theory from Khemakhem et al. (2020) is extended to include causal mechanism equivalence towards a principled definition of causal disentanglement. That is, a causal model, the standard notion of identifiability that yields marginal distribution equivalence is not sufficient to capture the equivalence of each individual mechanism between two models. To this end, the authors propose causal disentanglement prior with similar structure to the temporal prior from Lippe al. (2023a) to causally faclorize the latent space and learn causally disentangled mechanisms. The prior is formulated follows:
complex distríbution over causal variables, similar
044 Zpa.)
po(2)y) = II
po 23 2p,, Ui) = II p(v:)
(48)
Oy:
1=1
1=1
(2 (2.5) Epm, v1) = (2,)) exp(Ti (A, @ 2, v/x(
(49)
where À is defined as an autoregressive causal flow that derives the natural parameters of z; as a function of the label and its parents 2pa,- Note that in this causally factorized prior, each factor z; can be identified due to the access to auxiliary information ye- The JCM-VAL VAE framework shown in Figure 10.
Identifiability Result. The authors define a new notion of causal mechanism disentanglement and show that the causal factors are identifiable up permutation and elementwise reparameterization The intuition
22

---

Published in Transactions on Machine Learning Research (05/2024)
Z1 -1>8 22
/
Z3
Z4
Z5
(1)
x 33)
26 27
x x(2
Figure 11: Unpaired Multi-domain Causal Representation Learning (Sturma el al., 2023)
is that identifiable models only guarantec the equivalence of the marginal distribution of the data and latent variables up to some tolerable ambiguily. However, the theory from Khemakhem et. al. (2020) ignores the case where latent variables could be causally related and induces a Markov factorization. The authors this work propose causal mechanism identifiability
reformulation of permutationn-oquivllent identifiability to take into account causal conditionals in the Markov factorization. That is, causal mechanism identifiability requires the equivalence of causal mechanisms up to some tolerable ambiguity and is a sufficient condition for disentangling the causal factors. The mechanism equivalence definition proposed by Komanduri et al. (2023) is fundamentally different than the IMA theory introduced by Grescle et al. (2021) since IMA falls into the mixing function restriction class of methods rather than latent conditioning (Khemakhem et al., 2020).
3.2.5 Learning from Multi-Domain Observational Data
Problem Setting. Sturma al. (2023) study the setting where one has access to observational data from multiple domains that may share underlying causal reprosentation (i.e., causal information should be invariant regardless of the view it is measured in). The observations from different domains are assumed to be unpaired (Le., only the marginal distribution of each domain is observed but not the joint distribution).
The setup is that the latent variables are sampled from some distribution p(z), where p(2) is determined by an unknown SCM among latent variables. In each domain e E {l.....d'', some domain-specific causal latents zs, are mapped to the observed vector (e)) via a domain-specific (injective) mixing function Je such that
(e) = 9c(2s)
(50)
where S. S H. is subset of indices. The general idea is that shared latent variables capture the key causal relations, and different domains give combined information about the relations. That is, the multi-domain setup "completes the picture" of the causal model in some sense since latent variables captured in one domain may not be able to be captured in another (c.g., causal variables inferred from an imaging modality may not be inferred in sequence modality) and the relations can arise from different domains, as shown Figure 11. Thus, when we put the domains together, we can identify the joint distribution of the latent variables.
Identifiability Result. The authors provide sufficient conditions for uniquely identifying the joint dis- tribution of the causal factors and the causal graph, where the causal variables are described by a linear SCM. Assuming that (1) the marginal distributions are non-degencrate, non-symmetric, with unit variance, a genericity assumption that the errors must be pairwise different, and (2) the (linear) mixing function is full-rank, the authors show identifiability up to permutation. The first assumption restricts symmetric dis- tributions and ensures that the distribution of crrors is non-Gaussian. This allows for the application of the linear ICA identifiability theory from Comon (1994) in each domain individually. The second assumption requires that for each shared latent node there at least. a node in every domain such that the latent node
23

---

Published in Transactions on Machine Learning Research (05/2024)
is a parent of the domain-specific node, as illustrated in Figure 11. However, this work does not address the more realistic situation where the mixing function is nonlinear. One direction to potentially enable identifi- ability in this setting is assuming access to some domain label for each domain and utilizing this augmented information to apply nonlincar ICA results (Hyvarinen et al., 2018) in each domain to identify latent factors.
Ahuja et al. (2024) study CRL in multi-domain settings from weak distributional invariances. This work is inspired by the invariance principle in Arjovsky et al. (2020), which requires that fixed subset of latents is unintervened across domains while its distributional properties remain invariant. Such property often holds when each domain comes from multi-node imperfect intervention. Typically, the distributional properties of representation can be partitioned into a set of stable (invariant) latents and unstable latents. To learn this invariance, the authors propose to train autoencoders with a class of invariance constraints to disentangle the stable latents from the unstable latents. Further, the authors in this work show block-affine identifiability of the underlying causal factors from a sufficiently large number of domains and specific invariance constraints without any interventional data.
3.2.6 Grouped Causal Representation Learning
Problem Setting. Morioka & Hyvärinen (2023) propose Grouped Causal Representation Learning (G- CaRL), a causal representation learning framework to disentangle causal variables from grouped high- dimensional observations without the need for additional supervision or interventions. The authors consider data-generating process with data coming from M groups (e.g., M distinct sensor measurements), where the observational (diffeomorphic) mixing can be separated into M > 1 non-overlapping groups with separate (diffeomorphie) mixing functions and the number of variables can vary arbitrarily across groups. Thus, the observational model is formulated as follows
x=
[x
x xM = g (...("......""" (z' (..))),." (c
(51)
where nThh is group-specific mixing function. In this setting, the latent variables 2"h in each group m are multidimensional and can be causally related in two ways: (1) 2 and nd are causally related for # m' or (2) variables within n are causally related. This strategy is employed so that the causal relations are complex enough to rule out the independence of latent variables. Notably, this formulation is not restricted to directed acyclic graphs and allows cycles to exist. However, the relations must be asymmetric. The graphical model for the G-CaRL setting is shown in Figure 12a.
Identifiability Result. The authors show that if (1) each variable in group m has least one neighbor in some other group and there is sufficient dependency of the derivatives of the causal mechanisms their inputs, then the latents in group (i.e., =n)) can be recovered up to permutation and elementwise invertible transformation. Given the identifiability of the latent variables, the authors also show that the learned causal graph, extracted from the latent variables via a post-processing causal discovery algorithm, is identifiable if (1) the intergroup causal relations are directed, and (2) each variable has a co-parent and co-child in the same group.
Learning Framework. The authors propose a self-supervised learning algorithm, G-CaRL, to learn the causal factors. The general idea is to use contrastive learning to discriminate between the original dataset (((1)) and shuffed version x(!,), where n is the sample index and n, is a shuffled index. The sample (a) consists of M groups known in advance. The elements of the sample ((n.) are assigned to each of the M groups by randomly selecting a sample index for each group m. The authors propose using a logistic regression classifier to distinguish between the two classes. The algorithm supports either joint learning of the causal graph or extracting as post-processing step.
3.2.7 Multi-view Causal Representation Learning with Partial Observability
Problem Setting. Extending work in multi-view nonlinear ICA (Gresele el al., 2019) and Enngleennlleent, Yao et al. (2023) study the identifiability of representations learned from multiple views/ news/modalities We have obsorvations from K different views (c.g., measurements from different sensors) with N underlying cansal variables of interest. The authors consider a partially observed setting where each view constitutes a different nonlinear mixture of subset. of underlying latent variables (diffeomorphic mixing), as shown in Figure 12b.
24

---

Published in Transactions on Machine Learning Research (05/2024)
g
g
x
S
Z1
72
Z
Z3
91
92
Z4
Sz
X1
3
x"
X2
(a) G-CaRL setting with M =
(b) Multiview CRL: 2 views and 4 causal variables
Figure 12: Data- geteraeneg process of (a) G-CaRL (Morioka & Hyvärinen, 2023) and (b) Multiview CRL (Yao et al., 2023)
The key idea here that a subset of latent variables corresponds to each modality. For example, some latent factors could capture information about text modalities while others capture information about image modalitics.
Identifiability Result. The main identifiability result in this paper is that the factors shared across all subsets of any number of views are block-identifiable up to a smooth bijection by using a contrastive learning objective. For example, in Figure 12b, the intersection of views S₁ and S2 enables block identifiability of latent variables 21 and 22- Further, as we add more views, the intersection of latents becomes smaller and thus we can identify the latent factors more precisely. In the naive case, we can train a pair of encoders and maximize content alignment between any pair of views via contrastive learning. This would allow one to achieve identifiability for the intersection of those views. Similarly, by induction, intersection of pair of blocks that have already been identified. However, the described approach requires one to train many encoders and thus is impractical. Ideally, we desire one encoder for each view. To achieve this, we can enforce content alignment only on the subset of latents shared between a pair of views. This can be done by learning an indicator function that effectively selects the latent dimension one cares about when performing contrastive learning between a pair of views. In multimodal sctup, this work generalizes work done by von Kügelgen et al. (2021) any arbitrary intersection of subset of views. Xu et al. (2023) consider similar partially observed setting where the obscrvations are unpaired, but there exists instance-dependent partial observability pattern, and show cennttatiooneqivvllern identifiability of the partially observed latent variables under linear mixing function regime.
identify the
the case where we can identify factors between
3.3 Learning from Temporal Observational Data
Many real-world systems typically evolve through time, where certain events in the past may trigger events in the future following the arrow of time. This has inspired rescarch in lcarning causal representations when we have access lo time series data. The following sections explore observational identifiable causal representation learning where latent factors corresponding to high-dimensional temporal data can be temporally causally related. The general setting for observational temporal CRL methods is shown Figure 13.
3.3.1 Latent Temporal Causal Process
Problem Setting. Yao al. (2022b) propose Latent tEmporal cAusal Processes (LEAP), framework for disentangled representation learning in the temporal setting where latent variables are temporally causally- related. The authors consider a data--gnerating process where we have access to temporally measured observations 21....,277 ØT. We assume that there exists a time-invariant mixing function g that maps some latent variable 2, to the observations via x, = g(*),, for all 1,..., Furthermore, we assume that the latent variables are causally related by some time-lagged causal mechanism 2a fi(Epa, Ga), where pa, denotes the causal parents of latent. variable zμ from some previous timestep and <a is the noise term
25

---

Published in Transactions on Machine Learning Research (05/2024)
Time-delaved Information
Causal Mechanisms fi fipppae Eu yy) ((2,)(z(-)te
Zit--L
Zit
Zi,t
:
2
ZJ,t
Xt-L
Xt-1
Xt
Figure 13: Observational Temporal CRL Setting
corresponding to latent variable 2;. In this work, the authors consider both parametric and nonparametric causal mechanisms in the nonstationary regime, where some side information (e.g., domain index) is observed for each time step.
The identifiability analysis extends work done in nonlinear ICA and the theoretical results from iVAE (Khemakhem et al., 2020) to the setting for temporally causally-related latent variables. The framework iVAE shows that generative models can be rendered identifiable by introducing auxiliary information, such as class labels, time index, or domain index. However, the assumption in iVAE is that the latent, factors of variation are independent. This work serves an extension of iVAE in the temporal setting for causally related latent variables. The theoretical results presented utilize two central assumptions to show latent identifiability in the setting where latent variables are temporally causally-related: nonsta- tionarity and functional constraints on temporal relations. The following are identifiability results under both nonparametric and parametric assumptions on the causal model.
Identifiability Result.
Nonparametric. In the setting where the causal mechanisms are nonparametric, the nonstationarity assump- tion is necessary to enable identifiablity. Specifically, the authors show that (1) if the noise distribution changes based the observed auxiliary variable y (i.e., Pe.lu); (2) the noise 6, mutually independent in each regime y, and (3) there is sufficient variability of the latent transition function for different y, then the temporal latent causal factors are identifiable up to permutation and elementwise reparamelerization.
Parametric, If the causal mechanisms are linear additive noise models (additive transitions), εtr is from a generalized Laplacian distribution, and the state transitions are nonsingular (i.c., the state transition matrix is invertible for at least one time lag 7), then the temporal latent causal factors are identifiable up to permutation and elementwise reparameterization.
Learning Framework. Based the assumptions necessary for identifiability outlined above, the authors propose LEAP, a VAE-based framework to disentangle temporal causal factors. For Iı.T, LEAP learns an encoder that maps the high-dimensional observations to a low-dimensional latent variable ø1.r via bidirec- tional gated recurrent unit. The transition dynamics are modeled by a learned causal prior distribution. To enforce the independence of noise constraint in the framework, the authors propose learning the transition prior, (
terms noise instead of latent causal variables by learning a flow-based mapping from causal variables to noise terms. That is, the function maps the latent causal variable at the current time step and its time-lagged causal parents to the corresponding noise term via éμ2 r, 2 (parameterized by separate neural networks r:). To estimate the noise distribution p(elly), the authors propose transforming standard Gaussian base distributions to noise terms via neural spline flows. To ensure that the estimated noises are mutually independent, the authors follow the strategy from FactorVAE (Kim & Mnih, 2018) and jointly train a discriminator to distinguish between estimated noise terms and randomly perturbed noise terms. The overall objective is to jointly train a VAE with (1) a suitable ELBO term, (2) regularize the latent posterior to follow the learned causal prior network, and (3) enforce independence among noise terms through time.
26

---

Published in Transactions on Machine Learning Research (05/2024)
3.3.2 Temporally Disentangled Representation Learning
Problem Setting. Yao et al. (2022a) consider the same setting as LEAP (Yao et al., 2022b) but only locus on nouparametric transitions, stationary environments, and utilizing distribution shifts on stationary conditions extend results to general nonstationary regimes. Under the stationary assumption, the mixing function 9, transition functions fi (fixed causal dynamic), and noise distributions Pe, are considered to be invariant. Under a nonstationary regime, the authors consider each regime of data yr as result of a distribution shift. The authors consider the following two violations of stationarity: (1) changing causal dynamics, and (2) changing global properties of the time series across domains (e.g., style of video). Finally, the authors consider a general nonstationary case by combining the fixed stationary regime with the two nonstationary regimes such that 2 = (afix, ,4 _ where z!"x is the fixed dynamics, zhe is changing dynamics, and zpbe is the global observation changes. Each of these latent partitions is captured by respective transition functions [f fer fol.
Identifiability Result. In the stationary regime, where the cansal dynamics are fixed, the authors show identifiability up to permutation and elementwise reparameterization if linear independence is satisfied among vectors of gradients of logp(zx,e((-1)) with respect to each latent 2i,-11 In the nonstationary regime, where causal dynamics p(z), are changing across m values of the context variable y and 2k,t are mutually identifiability by showing lin-
independent conditioned on 2t-1; the authors show permutation-qquivaleee ear independence of vector functions defined with respect to the gradients of the conditional distribution across the contexts. In the nonstationary regime where global observation changes occur (i.e., p(zx,ely) changes across m contexts), with the additional assumption that b are mutually independent conditioned on y, the authors show permutation-oquivalleee lar distribution ion shifts with stationary and the two nonstationary conditions, the partitioned latent factors (ofix, ,2 1ef", chg 2gth) are jointly identifiable up to permutation and elementwise reparameterization.
identifiability. Finally, the authors show that under modu-
Learning Framework. Based on the identifiability results, the authors propose TDRL, a VAE-basod framework to recover the temporally causally-related factors of variation in the specified setting. The frame- work extends the sequential variational autoencoder with modules to model different distribution shifts. First, the authors embed the domain index yr into low-dimensional changing factors (dynamics and global properties) and use them as imput to the inverse dynamics function fz and observation function fo re- spectively. Similar to LEAP, the inverse of the three transition functions (modeled as flows) map to random noise terms such that we can apply a change of variables to derive three transition priors p(.s, &umme-lagged); Ur), where (s,c, o} represents fixed, changing, and global obser- vation dynamics, respectively. Similar to other VAE-based methods, the TDRL objective is optimize a suitable ELBO using the joint transition prior (tmmeeeii- yr) to regularize the latent postcrior.
Br), and
3.3.3 Temporally Disentangled Representation Learning with Unknown Nonstationarity
Song et al. (2023) consider the same setting as LEAP, but assume that the latent transition function is also a function of unobserved side-information (nonstationarity) e such that zn C.C)). Further, cr....,rr are assumed to be sampled from Markov chain based on the transition matrix.
Problem Setting.
Identifiability Result. First, the authors show that the nonstationarity e is recoverable up to permutation and elementwise reparamcterization assuming that the number of latent states for nonstationarity is unknown and the transition matrix full rank. That is, the parameters of the Markov process are identifiable. Based on this condition, if we additionally assume that (1) the mixing function is bijective, (2) the latent components at the current timestep z, are conditionally independent given the latent variables at the previous time step Rt-1 and the latent nonstationary ce, (3) and the latent transition function is sufficiently different for different a, then the authors show that the latent causal factors are identifiable up to permutation and elementwise reparameterization.
Learning Framework. The authors propose NCTRL, a VAE-based framework to learn disentangled latent temporally causally related factors. The framework consists of three modules. (1) The Prior Network Module is similar to the causal prior network from LEAP and estimates the latent transition prior distribution via inverse flow- mapping to noise variables. (2) An autoregressive Hidden Markov Model (HMM) learns
27

---

Published in Transactions on Machine Learning Research (05/2024)
to estimate the transition functions, the transition matrix, and utilizes the Viterbi algorithm (Forney, 1973) to obtain the optimal nonstationary states (f1,.... .r). (3) An encoder-decoder architecture is used to learn the latent causal factors.
3.4 Learning from Static Interventional Data
Instead of assuming access to observed labels or only obsorvational data, several approaches leverage in- terventional data from L2 of Pearl's hierarchy to learn identifiable causal representations. The following methods assume access to interventional data under perfect or imperfect interventions. This can often be a realistic assumption in practice since several domains, such as robotics, medicine, and genomics, have an abundance of interventional data available. The general intuition around using interventional data is as follows. Latent causal variables are generated by causal mechanisms defined in structural causal model. According to the ICM principle, the causal mechanisms must be independent and modular components. This implics that upon an intervention, only the manipulated causal mechanisms relating to the intervention actually change. Such sporse changes suggest that interventions are indeed critical to preserving the modu- larity property of causal mechanisms. Interventional data can thus be used to achieve sufficient variability in the observations for identifiability guarantees. The data generating process for the interventional paradigm is similar Figure 15, where there may be observations sampled from several different environments (or contexts) that are induced by perfect or imperfect interventions underlying causal variables.
Before we describe interventional CRL methods, define the important notion of paired vs. unpaired interventional datasets. Paired interventional datasets are those where we couple environments (or datasets) that intervene on the same node. Unpaired interventional datasets are those where we do not know whether the same node was intervened in a pair of environments. In other words, in the umpaired sctting, we have multiple interventional datasets, but we do not know which enviromments may have intervened on the same node. An example of unpaired interventional data is in biology, where for each cell, we can only obtain the mcasurement under single intervention (Zhang et al., 2023b).
3.4.1 Interventional Causal Representation Learning
Problem Setting. Ahuja et al. (2023) explore to what extent access to interventional data can facilitate causal representation learning. The main contribution of this work is identifiability results in the specified observational and interventional data setting.
Obseruational Selling. The observational data-generating process they consider is as follows:
2 p<2()
x = g2)
(52)
where is observational data point rendered from the underlying latent via injective decoder 9. The authors show that in the purely observational setting, we can identify the latent factors up to affine transformation (affine equivalence) if the mixing function is a finite-degree polynomial. Further, if we assume that the latents have independent support (Wang & Jordan, 2021), we can identify the factors up to permutation, shift, and scaling.
Interventional Setting. The interventional data-generating process is given by
2 II p(zilppa, )ōz,=z, x == x=)(²>
(53)
where is sampled from the distribution under intervention on causal factor zj- Access to arbitrary interven- tional data is more realistic scenario since interventions often act sparsely in the real world, and changes in causal conditionals lead to distribution shifts. Modeling distribution shifts through access to data from interventional distributions should intuitively be a sufficient weak supervision signal to learn accurate causal representations.
Identifiability Result. If we also assume access to interventional data from perfect do-interuentions, the authors show identifiailiiy of the intervened factors up to permutation, shift, and scaling and potentially up to componentwise correspondence if the mixing function is assumed to be diffeomorphie. The result
28

---

Published in Transactions on Machine Learning Research (05/2024)
suggests that intervened-upon latents are identifiable up to permutation and the remaining latents up to affine transformation. This can be formulated as optimizing the following constrained autoencoder objective that performs reconstruction under the constraint that an arbitrary latent has been fixed to certain unknown value
min E f.
l f(x) - x1| s.t. fi(x) =
l1
(54)
where hf, is an encoder-decoder pair and f1(x) is the ith latent factor with intervened value ž. However, if we only assume soft interventions (i.e., imperfect interventions) the authors claim identfiability of the factors up to block affine transformation.
3.4.2 Learning Linear Causal Representations with Linear Mixing Function
Problem Setting. Squires et al. (2023) study identifiability and causal disentanglement when one has access to unpaired interventional data with single node interventions. The authors consider a setting with a linear injective mixing function (i.e., x = G2) with a pseudoinverse H and a linear Gaussian additive noise structural causal model. The data-generating setup in this work considers n latent variables that are generated according to linear SCM. Additionally, following the common assumption in identifiability that variables are observed across multiple contexts, the authors define several contexts (i.c., environments) that are a result of an intervention on causal variable. These contexts are indexed by a variable k E (O....,)) K), where k = 0 is the observational context and & > 0 refers to an interventional context. In this setting, the mixing function is assumed to be invariant across contexts. The use-case considered in this work is modeling the internal state of a cell. There are complex interactions between the concentration of proteins, location of organelles, etc, and each context is an exposure to a different small molecule. Each molecule has a highly influential effect, changing only one cellular mechanism.
Identifiability Result. The main identifiability result in this work shows that having access to inter- vened context for each of the causal variables is sufficient condition for identifiability. Furthermore, in the worst case, they show that at least one intervention per latent causal variable is, in fact, a necessary con- dition for identifiability. The intuition behind CRL unidentifiability is shown in Example 3. If we extend this example to multiple contexts by defining a context-specifee noise scaling such that the SCM is = = AB 2 2gge, where A is the weighted adjacency matrix in context l and Q4 diagonal with positive entries, it is clear that without interventional contexts for each latent variable, we collapse to a spurious ICA solution. That is, two different solutions ({Ax) k=0 ) G, (S& and (fAg), G, { (n+1) can produce the same observational distribution (covariance of the observed data X) in all contexts. Thus, the authors show that any setting with access to less than n +1 1 observed contexts renders the latent factors unidentifiable In Example 3, this means that we must observe at least 3 contexts: one for the observational, one for an intervention on 21, and one for an intervention on 22 to identify the latent factors. In the general case, where the covariance of the data x is rank-dcficient (i.c., n < d), the authors propose to decompose the pscudoinverse of the covariance matrix of in each context via RQ decomposition to recover the mixing function. With the aforementioned assumptions, the authors show that the latent factors and invariant mixing function be recovered up to permutation and scaling.
3.4.3 Learning Linear Causal Representations with General Mixing Function
Problem Setting. Buchholz al. (2023) study causal representation learning where the mixing function is assumed to be general (o.g., deep neural networks) instcad of restricting 2023) or polynomial (Ahuja et al., 2023). Consider a data-generating process with n causal variables, where one has access to an observational distribution and n unpaired interventional distributions (i.e., for each sample, we only obtain data under single intervention), each result of an intervention (unknown target) on exactly one causal variable. The latent variables are assumed to be described by a linear Gaussian SCM and the mixing function is injective. The setting considered in this work is quite similar to Squires et al. (2023) and generalizes the results to arbitrary mixing functions.
be lincar (Squires et al.,
Identifiability Result. The main identifiabiliy result in this work shows that the latent factors are identifiable up to permutation and scaling. Three main assumptions are required to keep the soundness of
29

---

Published in Transactions on Machine Learning Research (05/2024)
the proposed identifiability theory. First, the number of interventions must be at least the dimension of the number of causal factors. A central assumption is that there must be al least one intervention on each causal factor. This assumption has been made in several other works such as Ahuja et al. (2023); Brehmer et al. (2022); Squires et al. (2023). The authors show that this is a necessary condition since a violation of this property renders even the weakest form of identifiability to break down. Second, the interventions must not be pure shift interventions. The authors show that if interventions are pure shift (relation to parents stays the same and only mean is changed), any causal graph compatible with the observations and will induce an indistinguishable model (i.., spurious solution). Lastly, similar to Yang et al. (2021), the latent variables are assumed to follow a linear SCM with Gaussian noise. Although post-nonlinear additive noise models have good approximability, an interesting direction is to explore identifiability when assuming nonlinear and non-additive noise models (Brehmer et al., 2022; Komanduri et. al., 2023). However, it is not clear that the same theory would hold in the nonlinear setting.
Learning Framework. To implement interventional causal representation learning, the authors propose a novel contrastive learning approach for interventional learning. The goal is to train deep neural work to distinguish observational samples from interventional samples. Although the identifiability results suggest that model learning from the aforementioned data-generating process can properly disentangle the factors of variation, practical models are still lacking. The authors design variational autoencoder- based approach that still relies on paired counterfactual data to tractably estimate the log-likelihood of the data.
3.4.4 Causal Disentanglement under Soft Interventions
Problem Sctting. Zhang ct al. (2023b) consider a sctting where unpaired observational and interventional data is available with soft interventions on the mechanism of latent variables. In the limit of infinite data, the authors show affine and permutation equivalent identifiability under certain set of conditions and a generalized notion of faithfulness. This work relaxes the assumptions of Squires et al. (2023) and considers polynomial mixing functions with a nonparametric latent causal model. The authors assume that the dataset includes samples from both observational and interventional distributions, where at least one intervention is required per latent node for identifiability. In other words, there must be a distribution (or dataset) induced as a result of an intervention of a latent node, for all latent nodes. Furthermore, the interventions are assumed to be unpaired, which means for each observation, one only obtains a measurement under a single intervention. Thus, this work serves The authors run experiments on a biological genomics dataset, Perturb-seq (Norman et al., 2019), and utilize the framework to predict the effect of unseen combinations of genetic perturbations.
extension of Squires et al. (2023) to arbitrary soft interventions.
Identifiability Result. Zhang et al. (2023b) define an equivalence class, CD-equivalence, as the recovery of the latent variables, causal graph, and intervention targets up to some permutation. In addition to identi- fying the causal factors, we must also identify ancestral relations. In general, the interventional faithfulness assumption, which suggests that interventions on nodes will always change the marginals of all of its descen- dants, can be used to identify the descendants of an intervention target simply by testing if the marginal has changed. However, if we assume a linear mixing, the authors show that the interventional faithfulness assumption is not sufficient to detect the marginal distribution change in descendants when intervening on causal variables. Thus, they use a more general notion of faithfulness to avoid downstream effects being nul- lified by linear combinations of factors. The authors show that in the best case, only the ancestral relations and intervention targets are identifiable up to CD-equivalence.
Learning Framework. The authors propose Discrepancy- VAE, an auloencoding variational Bayes frame- work to disentangle causal factors using interventional data. The proposed framework utilizes two parallel encoders and a joint training objective.
Observational encoding. The observational encoder maps the high-dinensional observations x to latent ex- ogenous noise variables e. Then, the noise encodings are mapped to latent causal variables 2 via some deep structural causal model s that parameterizes the cansal mechanisms between causal variables in the topo- logical order of the causal graph. Finally, the mixing function, which is assumed to be a polynomial class, maps the causal latent variables back to observational space.
30

---

Published in Transactions on Machine Learning Research (05/2024)
Causal Model
Observational Encoder
X
(Ag.
2
Decoder
x
el
Interventional Encoder
1
(gg.Sa,)
Decoder
Counterfactual Causal Model
MMD Loss
x p(x!)
Figure 14: Discrepancy- VAE Framework (Zhang et al., 2023b)
Interventional encoding. The interventional encoder learns to map the intervention label I to an intervention larget i via softmax normalizatiin to approximale one-hot encoding of the intervention targel (i.e., Gumbel- Softmax). Given the intervention target, an interventional deep SCM 81 learned to map the intervened noise encoding cf the intervened causal variable 2/. Since the interventions act on the noise terms here, they are considered be soft interventions (e.g., additive shifts). Accordingly, we get "virtual" counterfactual wf once the (shared) mixing function decodes the intervened causal variables. To supervise the generated virtual counterfactual samples, the authors enforce Maximum Mean Discrepancy (MMD) (Gretton et al., 2012) loss between the generated counterfactual and a sample from the interventional distribution p(x'). In this framework, the causal DAG is assumed to be unknown, so the authors place a sparsity constraint on the causal graph adjacency matrix to ensure that the causal relations are sparse. This is based on the intuition that the latent causal graph can be identified by using the sparsest possible model that fits the data.
3.4.5 Learning from Heterogeneous
Data with Unknown Interventions
Causal Component Analysis. Liang et al. (2023) study an intermediate problem between independent compo- nent analysis (ICA) and causal representation learning (CRL), known as causal component analysis (CauCA), CauCA is generalization of ICA to the setting where the latent variables may have causal dependencies. A rigorous identifiability theory is formulated in multi-environnent setting, where data is collected from sev- eral (perfect or imperfect) interventional environments. In CauCA, both the causal graph and intervention targets are assumed to be known. Thus, any results from CauCA apply as a special case to ICA. However, any negative results from CauCA apply to CRL. That is, if certain tasks cannot be accomplished in CauCA when the causal graph known, they certainly cannot be done CRL when the graph unknown. A key observation of this work that if any pair of observational and interventional environments overlap, we can apply measure-preservee
transformation (e.g., Gaussian rotation) to the overlapping region induce different solution that is observationally equivalent, which implies unidentifiability of the latent factors. To rule out spurious solutions, the following interventional discrepancy requirement is proposed.
Definition 7 (Interventional Discrepancy (Liang al., 2023)) Any pair of observational and inter- ventional densities (ps: Pc) must be different almost everywhere:
Ø (zi (pp,) p. (2g (2)(2pa))
#
(55)
Ōz
Nonparametric CRL. von Kügelgen al. (2023) extend the CauCA framework study CRL in a completely nonparametric setting, where the causal graph and intervention targets both unknown.
31

---

Published in Transactions on Machine Learning Research (05/2024)
Unknown Ground Truth
Learned Model
Co ~ pv
co X
eo 2 ~ qz
eo:
Px
e1
e1
Z qp C1
e1:
pv
px
62 pv
62 Px
2 9z e2
e2:
Figure 15: Multi-environment setup with single node perfect interventions and shared mixing function, where & are the learned factors and u are the underlying ground-truth (von Küigelgn et al., 2023). The rod color indicates the intervened causal variable. We can see that the learned model (right) identifies the factors up permutation of the ground-truth factors (left).
Problem Setting. von Küigelgen et al. (2023) consider learning representations from heterogeneous data from multiple related distributions that arise from interventions in shared underlying causal model. Since only dataset of Li.d observations cannot yield identifiability, the authors propose learning from multiple en- vironments given access to heterogeneous data from multiple distinct distributions. This multi-environment data satisfies the assumption that certain parts of the causal generative process are shared across envi- ronments so that the environments do not vary arbitrarily. All environments share the same incoriant diffeomorphic mixing function and underlying SCM, and any distribution shift occurs as a result of inter- ventions on some causal mechanisms. Such an assumption also consistent with the SMS hypothesis from Principle 2. The shifts across environments from this principle can be formulated as follows:
Definition 8 (von Kügelgen et al. (2023)) Each environment e shares the same mizing function g and each Pe results from the same SCM through an intervention on subset of mechanisms Ie C [m]:
(²))) = (.,.....,
II p² (2 ZEZ*
II p(2ylepm,) JØZ
(56)
where intervention targets Ie are unknown.
Identifiability Result. The authors highlight that imperfect interventions, which change the mechanism but preserve parental dependence, are insufficient for full identifiability, as shown by Brehmer et al. (2022). This is because arbitrary imperfect interventions that preserve parents and only change the noise term produce a spurious ICA solution, which would be indistinguishable from the ground truth. Thus, they consider only perfect interventions (i.e., Ve and E I .I' (z/l2pa,) = pf (2:)). In the two-variable case, they show that environments from 1 perfect intervention per node are sufficient to identify the causal factors up permutation and elementwisc reparameterization Figure 15 shows simple 2-variable causal graph ei 82 where U1 and 12 the ground-trute truth causal variables. In this case, if one has environment intervention on and an environment as result of an intervention on U2, we can learn latent factors #1 and 22 that are identifiable to permutation of the ground truth factors U) and U2 and their causal structure. For a general number latent cansal variables, access to pairs* of environments corresponding two distinct perfect interventions on each node is enough guarantee identifiability up to permutation and clementwise reparamotorization.
result of
Example 4 (2-variable environment setup) In the = 2 case, we have 3 possible enwironments 1 where eg is the observational environment and for + O, e; is the encironment induced by
Note that paired Interventional datasets fundamentally different than paired counterfactuals studied (Brehmer al., 2022)
32

---

Published in Transactions on Machine Learning Research (05/2024)
perfect intervention on one of the two variables. That is, it could be on or 02 since the intervention terget is unknown.
Example 5 (n-variable environment setup) In the general n 2 variable case, we need at least one pair of distinct single-node perfect interventions per node to fully identify the causal factors. If n = 4, we have 2n enwironment pairs from two perfect interventions. That is, we would need the following 8 pairs: e' (i)) and e" = [(e, (ax,,e)) e())=,, where e, is the result of an intervention on one of the variables and ee is the resull of another distinct intervention on the same variable.
Learning Framework. The authors propose potential approaches for learning causal representations from finite interventional datasets sampled from {eE) where E is the collection of all environments. For instance, a VAE could be used to learn an encoder, causal graph, and intervention targets, similar to Brehmer et al. (2022), with 2; 1. E Epa; for unintervened i. Another approach is to use normalizing flows to learn the mixing function and causal mechanisms.
3.4.6 Score-based Causal Representation Learning with Interventions
Problem Setting. Varıeı et al. (2023a) propose a score-based approach to causal representation learning, The general intuition is that parameterizing causal mechanisms with sufficiently complex neural networks is enough to capture the effect of interventions from changes in the score function, which is simply defined as the gradient of the log of the probability density. Thus, a valid transformation (or unmixing) renders the score function essentially invariant across different interventional environments. Such a property can be used to perfectly recover the latent DAG structure with soft interventions. The main motivation behind using score functions is that they often have sparse changes in coordinates upon single-node intervention, which is consistent with the ICM principle.
Nonparametric Identifiability. Variıı et al. (2023b) consider learning causal represontations in a com- pletely nouparametric setup where there are no parametric assumptions on the latent causal model nor the mixing function. The general idea to obtain sufficient interventional diversity by having access to 2 interventional distributions for each latent causal variable. That is, for latent causal variables, we have 2n observational distributions for sufficient coverage.
Minimizing the score difference in the latent space gives us the correct encoder and latent causal variables. However, in practice, we only have access to high-dimensional observations. The authors derive the following relation of score difference between observation and latent space to show that minimizing score differences in the observational space will solve the same objective as that in the latent space up to the linear transformation defined by the Jacobian of the docoder.
s2(3) 37 = [Jaec (sx(2)) xx()))
(57)
where Jaec is the Jacobian of the decoder and (x) V log pog (x) is the score of the mth interventional distribution. The authors then utilize score functions to solve the identifiability problem by minimizing
52t (x)) + Roconstruction Loss (a) ([] (x) (ami))=((())))) =x))) [z) (s2
(58)
we minimize the score difference for each pair of distributions, which is implemented via sliced score matching (Song et al., 2019b), then we can recover the true latent causal variables and the latent DAG up to permutation and clementwise reparamctcrization. In contrast to von Küggelgnn et al. (2023), the authors assume access to 2n environments but do not assume knowledge of the correct coupling of observations corresponding to intervention on target latent variable (i.e,, unpaired interventional data).
Parametric Identifiability. As special case of this result, Vanıı et al. (2023a) show that under (1) only single-node stochastic hard interventions, (2) linear mixing function, and (3) sufficiently nonlinear latent causal model, the latent variables and DAG are identifiable up to permutation and elementwise roparancterization. Furthermore, if we have access to one soft intervention instead of hard intervention per node, the latent DAG can be perfectly recovered and the recovered latent causal variables satisfy the Markov property. This result is similar to the result from Squires et al. (2023), which considers identifiability
33

---

Published in Transactions on Machine Learning Research (05/2024)
E
E
z
+
zipn
x
2
Zip
x
c++1
c
2++
1/1
c+1
4/22
I+1
x
x+1
z11
c+1
122
I+1
x
[++1
1++1
|+1
12 15
|c+1
(a) CITRIS (Lippe et al., 2022b)
(b) iCTTRIS (Lippe et al., 2023a)
Figure 16: Data-generating process of Causal Identifiability from Temporal Intervened Sequences
when both the mixing function and the latent causal model are linear and extends the result of recovering only ancestral relations as best case (Zhang et al., 2023b) to perfect DAG recovery by injecting sufficient nonlincarity in the latent causal model.
3.4.7 Extensions for Interventional Causal Representation Learning
Although most work in interventional CRL focuses leveraging data from single-node interventions for identifiability guarantees, there has been some recent work that has generalized this idea to allow for multi- node interventions undor a cortain set of sparsity or invariance assumptions (Bing et al., 2024; Ahuja et al., 2024). Some work has also studied the extrapolation of results from identifiable CRL to downstteamm tasks predict how interventions affect outcome variables, especially under novel interventions that are outside the support of the training data (Saengkyongam al., 2023).
3.5 Learning from Temporally Intervened Data
Sometimes, causality can have a temporal interpretation. That is, causal factors in dynamic system may evolve through time and be influenced by their values al previous time steps. Unlike methods in Section 3.3, which utilize only unintervened observational temporal data (L1 data), the following methods consider learning causal representations in the temporal setting by leveraging interventional data in the form of intervention targets from L2 of Pearl's hierarchy (i.e., an indicator of the factor being intervened on).
3.5.1 Causal Identifiability for Temporally Intervened Sequences
Problem Setting. Lippe et al. (2022b) present causal iden- tifiability for temporally intervened sequences (CITRIS). a temporal setting for causal representation learning. The au- thors consider a data-generating process with access to a sequence images {=t)== that represent temporal changes (interventions) to a system. The key assumption is that the causal variables at time step z++1 depend some subset of causal variables the previous time step z' along with intervention ion larget l (specifying which latent is inter- vened on). Assuming n multiiimensiional causal variables (2),,.... 24 |T following a dynamic Bayesian network with structural assignments of the form 2/_ = f(z(,p=, <), each la-
Emcoder
Transition Prior
Encoder
Decoder
x1+1_
gl+1
Figure 17: CITRIS/ICITRIS VAE Framework
tent factor zf is assigned lo one of the n causal variable sets via an assignment function &b learned via Stumbel-Softunn per latent variable (sampling latent-to-causal variable assignment).. A set 2[, is also reserved for unintervened factors that do not affect anything (noisc).
The causal factors are assumed to satisfy causal sufficiency and the induced distribution satisfies faithfulness. The data-generating process of CITRIS is shown in Figure 16a.
34

---

Published in Transactions on Machine Learning Research (05/2024)
Learning Framework. CITRIS is a VAE-based method that utilizes prior over latent variables condi- tioned on the corresponding intervention target to learn to disentangle the causal factors. This ensures that intervened variables are no longer dependent on parents while unintervened variables preserve their parental dependencics. The observed data is of the form fa", x , I.1]]. Notably, we have that the intervention targets are assumed to be known.
The temporal causal structure is time-invariant, so the parents are the same between all time steps. The authors also propose the notion of "minimal causal variables", which are latents in each causal set that split into invariable (independent of interventions) and variable (sensitive to interventions) parts. In other words, we are trying to find the minimum factors that are sensitive to interventions. Further, the following factorized prior distribution, called the transition prior, consisting of the current latent conditioned on the latent at the previous time step and intervention target, is proposed to disentangle the causal factors
po(2+112"' 1 = po po |2", 1,t1)
(59)
To include instantaneous effects, the authors extend CITRIS and propose iCITRIS (Lippe et al., 2023a), which can handle instantaneous causal relations within timestep when given perfect interventions with known intervention targets (e.g., flicking light switch and the bulb turning on can happen instantaneously). In this setting, both temporal and instantancous causal relations exist, as shown in Figure 16b. To disentangle and identify the causal factors, the authors propose the following causally factorized transition prior
Po.a (2, 1 = po(2tt1119 (ot) II Po (2 12", att
p I2+1
(60)
where A denotes the instantaneous causal graph. The VAE framework for CITRIS and iCITRIS is shown in Figure 17.
Identifiability Result. The identifiability analysis shows that although all causal factors are not identifiable in this setting, the minimal causal variables (intervention-sensill we condition each latent variable on exactly one intervention target (in addition to instantancous parents for iCITRIS). In this setting, the authors show that for every pair of causal variables, exactly one must be intervened on to achieve identifiability. If both causal variables are simultancously intervened on or neither are intervened on, we lose the ability to identify the factors.
components) are uniquely identifiable if
3.5.2 Binary Interactions for Causal Identifiability (BISCUIT)
Problem Setting. Lippe al. (2023b) propose Binary Interactions for Causal Identifiability (BISCUIT), method for causal representation lcarning from videos of dynamic interactive systems. The setup for BISCUIT is quite similar to CITRIS, where we have n causal variables (24) E VT in a dynamic Bayesian network, with an assignment function te mapping latent factors to causal groups, as shown in Figure 18a, The authors assume that an agent's 's interaction with a causal variable can be described by an unknown binary interaction variable similar to the action variable in Lachapelle et al. (2022). Under this assumption, the causal variables of an environment are shown to be identifiable. BISCUIT is an extension of the CITRIS (Lippe et al., 2022b) framework to the setting where intervention targets are unknown.
Learning Framework. Given a temporal setting, BISCUIT is a VAE that encodes x++1 and x' to the latent space where each causal variable is encoded in a separate set of dimensions. The prior over the latent variables is conditioned on the previous frame and a known rogime variable R' consisting of action information. Given the regime variable al each time step, a binary interaction largel for each latent causal variable predicted via a multilayer perceptron (MLP) that takes as input the action information and representation from the previous frame. This can be viewed as each causal variable having an observational (I, = 0) and interventional (I, 1) mechanism. The transition prior for the model is defined as follows:
Pa (2 R(+1)= = II Po ( 1+ |2 MLP! (R+112²))
(61)
35

---

Published in Transactions on Machine Learning Research (05/2024)
E
zijn
x
x
c++1
c+1
++
2/p2
++1
x
Rt+1
MLP
|++1
Transition Prior
1
1++1
1
1t+1
Rc+1
x
Encoder
Decoder
gt+1
(a) BISCUIT data-generating process
(b) BISCUIT VAE Framework
Figure 18: Binary Interactions for Causal Identifiability (Lippe et al., 2023b)
where m the dimension of the latent space such that m > n and w denotes the parameters of the MLP. The "regime" variable and representation of the previous time step can be considered the auxiliary information that we condition on to achieve identifiability, consistent with Khemakhem et al. (2020). Thus, BISCUIT disentangles the causal factors. Their identifiability result mainly relies on the binary interaction variable I; which gives a distinct interaction pattern for causal variable and a minimum of [log2 nJ + actions. This mcans that an agent intoracts with each cansal variable in a distinct pattern and docs not always interact with any two causal variables at the same time. If 7 = O, they claim the unidentifiability result of nonlinear ICA (Hyvärinen & Pajunen, 1999). The overall VAE framework for BISCUIT is shown in Figure 18b. Similar to other causal representation learning models, during inference, BISCUIT supports the manipulation of the latent codes to generate counterfactual instances.
Identifiability Result. In short, the authors show that if two models have the same likelihood, each causal variable has a distinct interaction pattern, and sufficient dynamics and temporal variability conditions are met (similar to Lachapelle et al. (2022); Komanduri et al. (2023)), then a learned model identifies the true model up to permutation and componentwise invertible transformation.
3.5.3 Mechanism-Sparsity Regularization
Problem Setting. Lachapelle et al. (2022) propose a new theory of identifiability from the perspective of temporal dynamics. They explore similar setting to Lippe et al. (2022b), where they consider sequence of observations {2')²-, that can be explained by semantically meaningful latent variables {2') and a sequence of corresponding auxiliary variables in the form of actions {a'] taken by agent (i.e., an action that changes the state of the latents). Following the form of the conditionally factorial prior from Eq. (25) (Khemakhem et al., 2020), the authors propose a conditional prior where latent factors at a given time step are conditionally independent given their state at the previous time step and the action variable as follows:
02 = II p( a55))
(62)
p( aa+) = hx(2 exp (T)() (A 02 -
(63)
where A denotes the action graph specifying the action variables that the causal parents of 2, A specifies the causal parents of 2t from the latents 2²-1 at the previous time step, vi is the normalizing constant, and h; À are defined as in Eq. (25). The authors propose VAE-based method with the defined conditional prior to regularize latent mechanisms to be sparse via A and A which act as masks to select the direct parents of z². The key assumption behind their approach that the relations between causal variables across time and the effect of action variables on the latent variables are both sparse. One limitation of their exploration is ignoring the case where there could be instantancous causal effects within time step, as considered by Lippe et al. (2023a).
36

---

Published in Transactions on Machine Learning Research (05/2024)
Soft intervention
style change
S
x
x
Figure 19: Problem formulation of self-supervised learning for causal representation learning
Identifiability Result. Under sufficiently varying natural parameter functions with respect to both tem- poral latent factors and actions, the authors show permutation-cquivalee of latent factors when only the mean is learned, the decoder diffeomorphic onto its image, and the learned connectivity graphs are sufficiently sparse, which can be enforced by sparsity constraint in causal discov- ery. Follow-up works also consider partial disentanglement by generalizing the specific graph criteria on the ground-truth cansal graph (Lachapelle & Lacoste-Julien, 2022).
identifiability (disentanglement)
3.6 Learning from Paired Counterfactual Data
Another class of methods assumes access counterfactual data from Ly of Pearl's hierarchy to aid in learning identifiable causal representations. The following methods assume counterfactual data augmentations contrastively paired counterfactual data weak supervision to learn causal representations. Works such Gresele et al. (2019) and Locatello et al. (2020) have taken multi-view approach to nonlinear ICA and have indicated that. learning a representation that represents multiple different views the same underlying factors is a good inductive bias to learn disentangled representations.
3.6.1 Self-supervised Learning with Data Augmentations
Problem Setting. von Kügelgen et al. (2021) study the role of data. augmentations in self-supervised learn- ing (SSL), specifically in the context of identifiable representation learning. SSL with data augmentations is closcly related to identifiability in the multi-view sctting (Gresele et al., 2019; Locatello et al., 2020), where we are given a second view 전 of some observation æ, resulting from a modified version ž of the underlying factors z. In this work, instead of applying transformations at the observation level (i.e., on x), as typically done in self-supervised learning, the authors propose a process of applying transformations in the represen- tation space on 2, where g (x), to obtain a modified which is decoded to obtain the augmented image 5.
The authors propose VAE-based objective. First, we train an encoder g using neural networks to obtain the representations (2, 2) for the pair (z, 2) and use a negative L2 loss as a similarity measure. The authors propose splitting the latent variable into a content part c, which would be invariant to augmentations and thus presorve semantic information, and a style part 8, which captures the stochastic variation from the augmentations. Diverging from the typical assumption of independent factors of variation, the authors assume that there can exist any nontrivial statistical or causal dependencies in the latent space. Thus, a causal view of the data augmentation process is counterfactuals generated under soft style interventions, shown in Figure 19. Based this assumption, we can define underlying SCM on the content and style latents, with causal graph c -> 8, as follows:
fcle)
s s= fs,, (Wes Uv) Puo Pu,
(64)
where de, u, are independent exogenous noise variables, and fer fo are deterministic functions. Then, the causal latents be decoded into observations g(c, Given the factual observation oF g(
37

---

Published in Transactions on Machine Learning Research (05/2024)
N
Eneoder
Decoder
Projection
X
x
S(E)
Z
p())
Intervention according to il
Interveation Encoder
2
p(ElE,1)
Encoder
Decoder
X
Projection
Figure 20: Implicit Latent Causal Model (ILCM) (Brehmer et al., 2022)
which resulted from (w ug ) we may ask the counterfactual question: "What would have happened the style variables had been randomly perturbed without changing anything else?" Thus, we can consider a soft (imperfect) intervention on the style variable s, which changes the mechanism fs as follows:
do(s = fale, uay uA))
(65)
where UA an additional source to account for the randomness the augmentation. After decoding this intervened latent variable, we obtain a distribution over counterfactual observations .C = g(c sCF) that can be computed from the modified SCM by fixing the exogenous noise variables to their factual values and performing the soft intervention follows:
cp=
CF == un uA),
UA Pus
(66)
Identifiability Result. In this setting, the authors prove that identify the invariant content partition under a weaker set of assumptions compared to existing ideutifiability results from nonlinear ICA. In this work, rather than disentangling each factor of variation, the focus is to disentangle the content from style (i.e., block identifiability. This theoretical guarantee has seen empirical successes downstream tasks such as domain generalization and out-of-distribution applications (Chen et al., 2021; Mahajan et al., 2021).
3.6.2 Latent Causal Models
Problem Setting. Brehmer et al. (2022) consider contrastively paired counterfactual data and extend the work by Locatello et al. (2020) to the causal setting where the data pairs (I, .) represent a system before and after a random unknown, perfect, and sparse intervention on a causal variable, respectively. They propose Latent Causal Models (T.CMs) and consider the following weakly supervised data-generating process.
Definition 9 (Weakly-superviseed counterfactually paired generative process) The weakly super- vised generalive process of counterfaclual paired dala (I, is defined as follows:
< Pe: 1 3
z == ==)(e);
x x- g(2) - g(
(67)
I~PE VIE PE. 1.
= s(t),
where is the pre-intervention noise, E the post-internention noise, z the pre-intervention causal vari- ables, & is the post-intervention causal variables, Pi is the distribution over intervention targets, 9 is diffeomorphic mizing function, and is diffeomorphic solution function that maps the noise variables to causal variables.
Learning Framework. Implicit LCM (ILCM). For the implicit, noiso-based formulation, the pair (ar, is mapped a pair of noise encodings (c, using a projective noise encoder learn a posterior pe(c, 2|x, $). In this formulation, ē is equivalent to the SCM noise term that would have generated the post-intervention
38

---

Published in Transactions on Machine Learning Research (05/2024)
causal variable å under the unintervened causal mechanisms. They propose learning an intervention encoder (e)) to predict the (single) intervention target based on the dimension-pair with the minimum KT- divergence, The pair of noise encodings are then restricted to differ only in the dimensions of the intervention target. The remaining dimensions are forced to be the same using different averaging schemes (Locatello et al., 2020; Hosoya, 2019; Bouchacourt et al., 2018). Then, the pre-intervent ion noise is mapped to the causal variable via a conditional flow 8. The causal structure is inferred based on conditional independence testing to simulate causal parents. The causal structure is extracted from the learned causal variables after training the LCM. The noise-centric formulation aims to infer the noise distribution of the post-intervention data by performing a random intervention on the causal variable (based on the predicted intervention target) to yield & and learning an inverse flow prior to then infer the noise variable é for intervened dimensions. The post-intervention noise prior is defined as follows:
Oz. de:
p(Ele, II Ó(E, Ei) II ((z.)
(68)
sE7
where E; = a(E;; (pn;) with base density p(z) (N, 1) and 5 is the Dirac delta function with delta peak where the pre and post-intervention noise variables are equivalent. The overall architecture of the ILCM is shown in Figure 20.
Explicit LCM (ELCM). The authors also propose an explicit latent causal model that does not encode an intermediate noise variable and parameterizes all components of the SCM. In this formulation, the authors consider only the causal variables as the latents. The idea is map the pair (2, directly to causal latents (z, 2) via stochastic encoder q((/z) and decoder p(z|2). Further, the authors propose prior p(z, 2) that consists of parameterized causal graph and causal mechanisms. The model is called as an explicit I.CM
as it directly parameterizes all components of an LCM and contains an explicit representation of the causal graph and causal mechanisms.
For both formulations, the authors utilize interventional causal discovery algorithms, such as ENCO (Lippe et al., 2022a), as post-processing step extract the latent causal structure from the learned causal variables. The authors note although optimally trained ELCMSs correctly identify the causal structure and disentangle causal variables simple dataset, jointly learning explicit graph and representations presents challenging optimization problem.
Identifiability Result. The main identifiability result shows that LCMs are identifiable up to relabeling and elementwise reparameterizations of the causal variables under weak supervision from contrastively paired data samples. The authors prove this by leveraging graph isomorphisms between the true and Icarned causal models and showing that the entailed weakly supervised distributions must be equivalent. The main assumption behind the identifiability result is that paired data counterfactual data is available and interventions are perfect.
3.7 Other Works
Related to the methods discussed in this section, there have been alternate formulations of causal disen- tanglement in representation learning. Suter et al. (2019) consider a causal perspective of disentangled representation learning and define the notion of a disentangled causal process as shown in Figure 21. In this setting, the generative factors of variation Æ1,-.., Fn are independent when conditioned on a set of confounders C.
Reddy al. (2022b) extend the disentangled causal process from Suter al. (2019) to encoders (to study interventions) and generators (to study counterfactuals) of latent variable models. Further, the authors propose concrete metrics to account for confounding in the generative process. Induced spurious correlations among generative factors in observational data are often due to the existence of confounders. The authors propose two metrics to evaluate causal disentanglement in their setting: uncomfoundedness and counterfactual generafiveness. The unconfoundedness metric evaluates if the model is able map the generative factor a unique set of latent codes. The counterfactual generativeness metric is formulated as an average cansal effect between counterfactuals generated by intervened factors and those generated by unintervened factors.
39

---

Published in Transactions on Machine Learning Research (05/2024)
Z1
Zn- 1
Zn
Figure 21: Disentangled Causal Process (Suter et al., 2019)
That is, this metric ensures that only interventions the desired group of latent variables influence the generated image.
Based on the disentangled causal process definition, Wang & Jordan (2021) lay out desiderata for un- supervised disentanglement representation learning. They propose an independence of support score (IOSS) as metric to disentangle representations learned from observational data. The 1OSS is defined the Hausdorff distance, which measures the distance between two subsets of metric space, between the joint support of the representation {z1...., 2n} and the product of each individual latent's support (i.e., du (supp(2)..... En), supp((z) x supp(>2) x .... supp(z,))) This condition essentially enforces that each factor must take on a distinct set of values (i.e., the set of values that a factor can take does not depend on the values that a different factor can take). Enforcing IOSS can be seen as ensuring orthogonality between groups of latent factors and thus satisfying the disentanglement definition in Definition 3. The authors utilize the IOSS to prove the permutation-oqqiialeen identifiability of representations under certain conditions.
3.8 Discussion
Advantages. Although representation learning has been used in some of the most impactful applications, its causal blindness prevents generalizability and interpretability. The primary advantage of learning a causal representation is its robust application to downstream tasks. A causal perspective of representation learning enables generalizability to distribution shifts. For instance, take the canonical example of an image of a cow grass (Beery et al., 2018). A prediction lask that utilizes representations learned from a dataset of images can easily fall into the trap of associating grass with cows (spurious correlation) and fail to perform well when it sees a cow on different terrain. However, causal representations are invariant to distribution shifts (Arjovsky et al., 2020). The intuition comes from the Sparse Mechanism Shift hypothesis (Principle 2), where distribution shifts can be attributed to sparse changes in causal conditionals in the Markov factorization. For instance, the causal representation could capture information about latent factors corresponding to the shape of the cow, Such information invariant to what enviromment the image comes from. One could also learn more robust downstream models by utilizing causal representations to generate counterfactuals for data augmentation (von Kügelgen et al., 2021). Causal representations are also important in fairness applications (Locatello et al., 2019a; Zuo et al., 2023). Disentangled causal factors enable one to model dependencies between factors and perform interventions on sensitive factors of variation to observe their effect on other relevant factors. Thus, causal representations would have high utility in causal fairness settings when dealing with high-dimensional data such as images.
Limitations. As emphasized in previous sections, the identifiability of causal representations remains to be great challenge. Most methods rely upon strong assumptions such as linear and additive noise latent SCMs, no unobserved confounding, and no misspecification of SCMs. Such assumptions make it quite difficult to use causal representation learning (CRL) methods in practice. Further, many CRL moth require access to interventional data, which may be infeasible to obtain in many cases. We detail more limitations of CRL in Section Depending on the situation, methods from the observational, interventional, and counterfactual
5 Note that support refers the possible values representation can take.
40

---

Published in Transactions on Machine Learning Research (05/2024)
Causal Mechanism
Counterfactual Inference"
Key
Approach
Architecture
Features
CansalGAN (Kocsoglu et al. 2018)
causal generators
interventional distribution
GAN
heuristic
sampling
CGN
analytic
models invariant
GAN
heuristic
(Sauer & Geiger, 2021)
composition
(stable) features
CONIC
data augmentattin
models confounding
GAN
heuristic
(Reddy et al., 2022a)
DSCM (Pawlowski et al., 2020)
conditional
mechanism
Flow
analytical
flows
learning
CausalHVAE (De Sousa Ribeiro et al., 2023)
models
VAE
hierarchical VAE
analytical
mediators
Diff-SCM
stochastic
anti-causal predictor guidance
Diffusion
analytical
(Sanchez Tsaftris, 2022)
differential equation
Counterfactuals computed heuristically analytically using Pearl's abduction, action, prediction steps (Pearl, 2009).
Table 3: Summary of Controllable Counterfactual Generation Methods
layers can be employed. For example, if one has access to data from multiple domains and domain labels, obsorvational methods can be naturally applicd. However, learning the causal graph jointly with the cansal representation in this paradigm can prove to be quite challenging as we need to make restricting parametric assumptions the causal model assume knowledge of the causal ordering. In domains such as gemomics robotics, where perturbed data is feasible to obtain, interventional methods can be useful. Interventional CRL methods can often have nonparametric assumptions on the causal model (Zhang et al., 2023b; Varıa et al., 2023b; von Kügelgen et al., 2023) and interventional data can be used to identify both the latent representation and causal relationships. However, for most domains, collecting interventional data can be infeasible due to ethical concerns or lack of interventional facilities. Thus, developing more accurate CRL models in the observational paradigm would be highly beneficial. Although some methods utilize counterfactual data to learn causal representations, for most real-world applications, obtaining counterfactual data is simply impossible. However, getting data that may be close to counterfactual can potentially be utilized to learn both lalent factors and their causal dependencies. Overall, recent research efforts seem to suggest that methods utilizing combination of observational and interventional data are quite effective.
4 Controllable Counterfactual Generation
The objective of controllable generation is to produce novel samples based on specified attributes rather than simply mimicking the characteristics of the training data in non-controllable generation. However, their ability to extrapolate beyond the training distribution is constrained because of their reliance on sampling from observational distribution. Controllable counterfactual generation incorporates encoded causal knowl- edge that can further generate samples in previously unobserved contexts and enable users to edit specific samples by intervening on targeted attributes, with downstream causal effects automatically incorporated. Although causal representation learning methods are also capable of controllable generation through latent space manipulations and decoding, the focus is primarily tations for downstream tasks. Counterfactual generation methods usually assumne access to ground-truth causal information and thus have the advantage of not being plagued by unidentifiability issues. In contrast to representation learning, controllable counterfactual generation models often utilize GANs or diffusion models to generate realistic counterfactual images. Thus, such models are often more suited for real-world applications, where domain knowledge can be exploited to describe the causal dependencics of a system. We cover the main controllable counterfactual generation approaches proposed recent years. A summary of the main methods is shown in Table 3.
learning robust and interpretable represen-
41

---

Published in Transactions on Machine Learning Research (05/2024)
y,
J1
J/2
Vn-1
Yn
Nyg
Nyz
1/2
(a) CansalGAN generators for causal graph
(b) CausalGAN data-generating process
Figure 22: CausalGAN (Kocaoglu et al., 2018)
4.1 GAN-based
GANs have shown good performance in tasks involving sampling from data distributions. Although the standard GAN does not encode any causal information, it turns out we use generators to learn mecha- nisms between causal variables and formulate mechanism learning as an adversarial problem. The following methods focus on utilizing GAN components to learn causal mechanisms from label space to image space and enable counterfactual generation.
4.1.1 CausalGAN
Problem Sctting. As a modification of the ConditionalGAN (Mirza Osindero, 2014) support sampling from interventional distributions, Kocaoglu al. (2018) propose CausalGAN, Traditional GANs have shown impressive performance in generating samples from a given data distribution. GANs are example of implicit generative models, which sample from data distributions but cannot estimate likelihoods for data points. However, GANs cannot sample from interventional distributions. Assuming that we have a dataset with labels, mut ually independent labels can be thought of as a trivial causal graph. In this case, the ability to intervene on a variable and get novel samples is desirable. We assume the labels cause the image, as shown in Figure 22b. For instance, if color and species are independent labels, then color=blue should give us blue eagles and blue jays. However, if labels are causally related, such as gender - mustache, conditioning on the mustache will only give images of males. However, intervening mustache will give males and females with mustaches. Interventions preserve ancestral distributions, whereas conditioning doesn't.
The main idea of CausalGAN is to model each label using generator neural network that uses the causal parents of the label as its imputs. In this sctting, the authors assume access to the complete causal graph among the labels. Since jointly training the labels and image generators infeasible, the authors propose two-stage framework: (1) train generative model over only labels y and (2) use the node labels in the causal graph to train conditional generative model for images.
Learning Framework.
Stage 1. First, the authors propose to learn the Causal Controller, a causal implicit generative model between labels using a feedforward neural network for variables such that each variable is conditioned on the outputs of its parents' neural network satisfying structural equations the form v = f(pa. e). The arrangement of the causal variable neural networks for the graph 1/1 + 1/3 + 1/2 is shown in Figure 22a. This process can be split into two subtasks. First, we learn a causal implicit generative model over a small of variables and the first set using conditional generative network.
then learn the remaining set of variables conditioned Each node in the first set should come before any node in the second set. The Causal Controller module decides what distribution the image will be sampled from (interventional or obscrvational) and generates labels according lo the causal graph. For example, if we want to sample from the observational distribution, we can specify the values of each label. If we want to sample from the interventional distribution, we can fix the value of a specific label and propagate causal effects via the feodforward neural networks. This module is pre-trained trained using a Wasserstein GAN (Arjovsky al., 2017) to produce binary class labels with a specific probability.
42

---

Published in Transactions on Machine Learning Research (05/2024)
Stage 2. Let Paata be the real data distribution between the image and the binary label / E {0, 1). Similarly, let P/ denote the joint distribution between the labels given to the generator and the generated images. We also assume perfect Causal Controller such that Paata = p. =, where is the probability of class label Conditioncd on the label distribution from the Causal Controller, the CausalGAN learns an implicit class conditional generative model for images. There are three main components in the CausalGAN.
1. Labeler. The Labeler discriminator DER is trained to estimate the labels of real images in the training dataset. The Labeler discriminator module solves the following optimization objective
p pBa (w)
[log [ogg(Dn(())) + (1 p)lE (z) [log(1 - DIn(2))]
(69)
DLR
Anti-Labeler. The Anti-Labeler discriminator DLG is trained to estimate the labels of the images sampled from the generator (labels produced by Causal Controller). The Anti-Labeler is especially used to prevent label-conditioned mode collapse. The Anti-Labeler module solves the following optimization objective
max p
+ (1- ()) DLG(()))
(70)
DiG
Generator. The generator produces realistic images conditioned on labels from the Cansal Con- troller by minimizing the Labeler loss and avoids unrealistic image distributions that are easily labeled by maximizing the Anti-Labeler loss, thereby distinguishing between classes and images conditioned those classes.
For fixed generator, the discriminator D solves the following:
max Ex (oggg + Exzn, (a) log 1-D(x))
D(x))
(71)
D
D
For a fixed discriminator, Labeler, and Anti-Labeler, the generator solves the following:
min E (er +Ea (x) log 1 1- (L-D(x)) }
G
D(x)
(72)
pE (o (Drn - Be [lo(1 DIæ(X))]
+ +p (logg
- ()- [log(1 Dra(X))]
During inference time, one can sample from the interventional label distribution produced by the Causal Controller and generate novel counterfactual images unseen during training.
4.1.2 Counterfactual Generative Networks (CGN)
Problem Setting. Sauer & Geiger (2021) propose counterfactual generative networks (CGN), a generative model aimed at modeling images realization of shape, texture, and background variables, where spurious signals strong predictors of the class label. In contrast to CausalGAN, CGN aims to obtain classifiers that are interpretable and robust to spurious correlations that arise result of some confounding variable C. The authors design model that learns independent causal mechanisms m Sshape(-), f = Stexture()) ) b = fbookground()) and an analytical composition mechanism g that combines the three mechanisms as
follows:
f (m, f.b) m e f (1)m)mmm b
(73)
Given some Gaussian noise and labels sampled uniformly weak supervision (labels for high-level concepts like shape and background are hard to obtain), the architecture consists of BigGAN (Brock et al., 2019) backbone to model each mechanism as generator. The overall data-generating process CGN is illustrated in Figure 23a.
Learning Framework. A loss is specified for each of the mechanisms, which are then combined together by composite function (analytical, not learned) to generate the image combination of shape, texture,
43

---

Published in Transactions on Machine Learning Research (05/2024)
Cm
color
7
f
b
1
Zr
digit
(a) CGN (Sauer Geiger, 2021)
(b) CONIC (Reddy et al., 2022a)
(c) ColoredMNIST example
Figure 23: Data-generating process of (a) counterfactual generative networks (CGN) and (b) counterfactual generation under confounding (CONIC), and (c) an example of spurious correlation between digit and color from CNloredMNIST dataset
and background. To incorporate some form of a supervisory signal, the architecture proposes simultaneously learning a conditionalGAN to generate ground-truth images. A correspouding reconstruction loss is also defined between generated images from mechanisms and those from the conditionalGANN After training the CGN, the authors propose learning an invariant classifier that is invariant of any spurious features such texture and background and only relies shape changes for prediction tasks by using counterfactually generated data to predict mechanismn-peecffe labels. Take the canonical example of images of cows standing on a field. Through counterfactually generated data from the trained CGN model, the invariant classifier is effectively trained to use only shape attributes to predict labels. This is consistent with preventing the occurrence of spurious associations in distribution shifts, such as predicting 'cow' based on background spurious features. The authors empirically evaluate their framework on variants of MNIST and Tmage. which are benchmarks, and their results seem to show that they are able to randomly sample noise and labels to controllably generate realistic counterfactuals. Since CGN utilizes a BigGAN (Brock et al., 2019) architecture to train each of the generalive mechanisms,, it can be computationally demanding, inefficient, and not generalizable. Further, it does not take into account confounding effects.
4.1.3 Counterfactual Generation under Confounding (CONIC)
Problem Setting. Reddy et al. (2022a) study the ad adversarial effect of confounding in observational data classifier's performance and how data augmentation using counterfactual data can be used to mitigate these confounding effects. The authors assume causal generative process, based on Suter et al. (2019) and Reddy et al. (2022b), consisting of generative factors 2.,..., , (e.g., background, shape, texture, etc.) and some label y, that determine an observation x through some unknown mechanism f. These variables are considered to be confounded by set of confounders C......C C.. The generalive factors here are arbitrary and can be viewed as generalization of the shape, background, and texture features from Counterfactual Generative Networks (Sauer & Geiger, 2021), and perhaps have more potential to robustly capture important information about the generative process. The authors propose Counterfactual Generation under Confounding (CONIC) to remove the effect of confounding, which can lead to spurious correlations, by performing counterfactual data augmentation.
Learning Framework. To quantify uncertainty, a connection can be made between the amount of con- founding and the correlation between any pair of generative factors in the causal process from Figure 23b. The authors propose KL-divergence metric between the conditional and post-interventional distribution of pair of generative factors (i.e., V(2,, 2y), Drr. ( (l) founding, have that p(a)|d((2z)) = p(z(2z). Thus, the variables are unconfounded if the KL divergence is close to 0. In general, the existence of confounding can be viewed through backdoor paths between factors of the form + C. + Zj, where + j. To break this confounding, the authors lay out a counterfactual augmentation procedure using conditional CycleGANs (Zhu et al., 2017). The CycleGAN is a GAN model that is used predominantly in domain shift scenarios. The idca of CycleGAN is to use unpaired training data samples learn a mapping between two domains A and B, while preserving semantic information and satisfying the cycle consistency property (invertible mapping). Tn CONIC, the authors assume A X and
In an ideal scenario with no con-
44

---

Published in Transactions on Machine Learning Research (05/2024)
B = X', where X' is the counterfactual sample space generated as result of intervening on generative factor and producing an image.
Consider the ColoredMNIST benchmark dataset (Arjovsky et al., 2020), which adds colors to the digits and alters the strength of association between the color and digit label between the training data and test data, thereby inducing spurious correlations. We can enforce an extremely high correlation coefficient between the color and digit factors, thereby inducing confounding. For example, assume that if a digit is red, it is more likely to be the digit 7, as shown in Figure 23c. Suppose 2j represents the color and 2 is the digit class. The authors consider two subsets, Ti and T2, to remove this confounding effect. Ti consists of samples where exactly one factor is different from the original image. For example, if the original image was red 7, then Ti would consist of the digit 7 of any other color except red. T2 consists of the original samples. The mapping M is learned between T1 and T2. For any given sample that differs in a factor, M will map X to a counterfactual instance X', where 2, is changed to the original value. These counterfactual instances can be augmented to the original data to break any correlation between 2j (color) and 2 (digit). We can repeat, the process for each confounding edge from Figure 23b to generate set of counterfactual samples that remove the spurious correlations. These augmented data points differ from the original in only one feature. In order to satisfy this condition, the authors introduce a contrastive learning objective with a pre-trained discriminator that takes X and X' and ensures that the representation 2j is different between the two samples. Similarly, a discriminator is introduced to ensure that the representation 21 is similar between the samples. That is, color can change, but the digit class is invariant. Finally, to evaluate the counterfactual generation under confounding, the authors consider downstream classification task where they use the counterfactual augmented data to train a classifier to predict class labels. The empirical results show that CONIC outperforms empirical risk minimization, invariant risk minimization, and other basclines test set accuracy.
4.2 Flow& & Variational Inference-based
Although GANs can enable mechanism learning, they are notorious for their unstable training. Furthermore, we are often interested in estimating exact densities of dist.riuttions in a tractable way, which GANs are incapable of doing. Flow and VAs-based models are a flexible alternative for learning causal mechanisms. Further, they have several rigorous theoretical properties that make them stable and versatile. The following approaches leverage these models to enable tractable counterfactual inference.
4.2.1 Deep Structural Causal Models (DSCM) for Counterfactual Inference
Problem Setting. Pawlowski et al. (2020) propose framework for building SCMs using normalizing flows and amortized variational inference to tractably estimate exogenous noise of the SCM for counterfactual inference. This work studies the selting where we have access to observed data (images) with known labels that may be causally related. The general setup a structural causal model defined by the interactions between known labels and the obscrved data. x. The obscrved data could be image generated from high-level labels containing semantic information relevant to the image. For example, Figure 24c shows an MRI scan of a brain from the UK Biobank Imaging Study (Sudlow et al., 2015) and the causal structure of the associated data-generating process. In the causal graph, the brain volume (b) and ventricle volume (v) directly influence the generation of the image. However, other factors such as age (a) and sex (s) indirectly influence the image but directly affect the volume variables. Unlike CausalGAN (Kocaoglu et al., 2018), only a subset of labels directly influence the image.
Learning Framework. The authors propose using deep learning components to parameterize each mapping between variables in the SCM. The authors propose deep structural causal models (DSCM), an approach that is capable of performing counterfactual inference with fully specified causal models with no unobsorved confounding. A central assumption in this work is that the mapping ween observed data and its noise variable is bijective with a differentiable inverse. This is to ensure when given a set of observations, we can estimate the noise terms for counterfactual inference. Several works have emphasized the importance of reversibility for counterfactual inference models (Monteiro et al., 2023; Nasr- Basr-EEfahany et al., 2023). The authors leverage normalizing flows, an expressive class of generative models that offer flexibility in modeling
45

---

Published in Transactions on Machine Learning Research (05/2024)
рak
pak
ek
Zk
gk
Ek
Xk
ukk
fk
h
(a) Invertible explicit likelihood
(b) Amortized explicit likelihood
(c) Brain MRI example
Figure 24: DSCM modeling (Pawlowski et al., 2020)
complex probability distributions. The general idea of DSCM is learn a mapping between a causal variable and its exogenous noise. This can be done by assuming base distribution (typically Gaussian) and using flow model the distribution over the causal variable. To model causal relations, the authors conditional flows to additionally use parent variables to model the causal mechanisms. Thus, an invertible and explicit parameterization of the SCM, as shown in Figure 24a, would yield mechanisms of the form
f(xxipa.)
(74)
where the exogenous noise can be inferred as Eg = f
When introducing complex nonlinear functions in high dimensional settings, inference of counterfactual queries can quickly become intractable. To overcome this, the authors propose using variational inference in addition to normalizing flows to model the noise term of the observed data, as shown in Figure 24b. This approach involves using amortized variational inference to learn a noise representation of the observed data. The amortized, explicit approach VAE-based method that separates structural assignment, into a low-level (invertible) component h and a high-level (non-invertible) component g such that
2g = (ex: ) = hu(ux: 9a (x pa.). pa.)
(75)
where pag denotes the parent variables of the observation 2y (labels), the noise qj is decomposed into (ux, 2x) corresponding to low-level and high-level components of (ha,))) is a neural network that oulputs mean and variance of Ig, and h is a reparameterization.
Counterfactual Inference. During inference, we can compute answers to counterfactual questions. Fol- lowing Pearl's counterfactual analysis, the three steps can be achieved as follows.
Abduction. Each noise variable is assumed to affect only the respective observed variable. Thus, the noise Eg, conditionally independent given the observed data The following product factorization can approximate the noise posterior
p(ck Ik, pa.) TF pay)p(uulzs, pay) q(2x (1133 pa,)) (tr00).0)__) (ux)) (76)
where h inverts the reparameterization to yield the noise term
Action. To perform intervention the SCM G, the structural assignment of the desired inter- vened variable F& is replaced by a constant, 2 = ãg. making it independent of the parents pa, and exogenous noise (y- This induces a counterfactual SCM 9 = ($)x)()())
To sample from the counterfactual SCM G, the inferred noise can be plugged into the forward mochanism. The counterfactual distribution cannot be characterized explicitly in the explicit, amortized scenario. However, for each sample 8, we can approximately sample from via Monte Carlo follows:
Prediction.
a ~ q(2xlex ( _ h - ga (zf pay) pay) 56 hu ON (=f; pay): pay)
(77)
(78)
(79)
46

---

Published in Transactions on Machine Learning Research (05/2024)
Upa
21:1
Uz
pax x
Uz
pa
Z1=L
Z=1
x
Ux
X
U1
24i
Myy
Ux
(a) Causal HVAE Latent Mediator Causal Graph
(b) MorphoMNIST Latent Mediator Graph
Figure 25: Causal Hicrarchical VAE (a) mediator graph and (b) example (De Sousa Ribciro et al., 2023)
Example 6 Given a high level parometeriaation ge (=(= pay) (n(z2: paa), o² (zk, pa,)) and low-level pa- rameterization hu(uki (H, (²) paag) = - a² uk.
uf (Ix p1=;; pa (oaa))/(((( pa.)
ff. ((2t: pa,) + (z pa,) uf
(80)
4.2.2 High Fidelity Counterfactual Inference via Probabilistic Causal Models
Problem Setting. De Sousa Ribeiro et al. (2023) extend DSCM (Pawlowski et al., 2020) and propose causal generative modeling framework for image counterfactual generation with accurate estimation of interventional and counterfactual queries including direct and mediated causal effects. The problem setup consists of a high-dimensional image generated by a set of n factors. The factors are set of labels pa, that may be causally related. This work aims to use VAE-based models to model the mechanisms between the labels and the image æ. Then, one can infer exogenous noise, perform interventions on the set of labels to obtain pa, and generate counterfactual instances using the decoder. The authors introduce a new framework that includes latent mediators between pa, and 2, which allow one to compute direct, indirect, and total effects of interventions from the learned mean and variance of The latent mediator model is shown in Figure 25a. Since directly estimating the exogenous noise can often be impractical and expensive, one can alternatively learn a latent mediator model. Consider the MorphoMNIST dataset (Castro et al., 2019) causal graph in Figure 25b. In this system, thickness, intensily, and the label all causally influence the generated image. Thus, inferring the latent mediator can enable one to compute indirect effects on the generated image.
The authors point out the fundamental notion of self-consistency of deterministic VAEs, which posits that the VAE encoder inverts the decoder. Thus, deterministic VAËs effectively function normalizing flows, Reizinger et al. (2022) prove that VAEs actually satisfy self-consitenccy even in the near-determniiticc regime. Pawlowski et al. (2020) model the SCM noise variable as a latent variable learned through variational inference. Tn this setup, the VAF is near-deterministcc but proposes to abduct the noise of the observations through deterministic mapping, which leads to poor sample quality in DSCMs. To remedy this issue, the authors propose utilizing more complex generative causal model for accurate noise abduction and high- fidelity counterfactual generation.
Learning Framework. The authors leverage hierarchical latent variable models (HLVM), which define priors over L layers of hicrarchical latent variables. Thus, HLVM can be viewed as a generalization of latent variable models where the latent variables have a hierarchical structure. In a hierarchical variational autoencoder (HVAE) (Sonderby et al., 2016), the latent variable 21:1 assumed to be hierarchical. That is, each latent z, is determined by latent 1 in the L-layers. In HVAEs, the prior is learned from data instead of assumed to be standard Gaussian. The authors propose a conditional HVAE model that decouples the prior from conditioning the causal parents pa.- The generative model is defined as follows:
I-
p(x, (11/paa) = p(2/21LL pa.))(()))))) II p(2)2>)))
(81)
47

---

Published in Transactions on Machine Learning Research (05/2024)
The conditional prior forces the prior p(21:2) to be independent of the causal parents pag, while preserving the likelihood's is dependence on paz- In this formulation, z is the exogenous noise of the observed data æt
The authors point out that the preceding formulation suffers from unidentifiability due to the unconditional prior. Based on the identifiability result from Khemakhem et al. (2020), the authors propose conditional prior for the hierarchical latent variable p(z1:z), where the auxiliary conditioning information the set of parents pa.. The conditional prior takes the following form
L-1
p(21.1/pa (p,,) = ((a) II i=1
p(alesss.pp pa.)
(82)
where the prior is from an exponential family (Gaussian). In this formulation, the hicrarchical latent variables depend on the causal parents and can be viewed as the latent mediators to be inferred.
The authors point out that conditioning the generative model on the counterfactual parents pa, can often be ignored with fixed abducted noise. To remedy this issue, the authors propose a constrained counter- factual training procedure optimized via the Lagrangian to maximize the mutual information between counterfactual image I and its counterfactual parents pa..
One of the main contributions of this approach is that the medialor-based model enables computing direct, indirect, and total effects for generated images.
4.3 Diffusion-basee
Diffusiin probabilistic models have become the state-of-the-as generative model for high-quality data gener- ation. In order to enable high-quality counterfactual generation, recent work has focused on modcling causal processes using diffuusio models. The following approach elaborates on this idea and presents a method to perform tractable counterfactual generation using diffusion models and the theory of causality.
4.3.1 Diffusion Causal Models for Counterfactual Estimation
Problem Setting. Sanchez & Tsaftaris (2022) propose a unifying framework between diffusion models and causal models that is capable of tractable counterfactual estimation. The authors take inspiration from the connection between diffusion models and score-based generative models (Song et al., 2021b) and propose a forward diffusion process formulated as the weakening of causal relations among variables of an SCM via a set of stochastic differential equations (SDEs).
(k) = - - (1) dt + Vi,dw, 2
1
Vk E [1,
(83)
where 3, is the variance parameter and w denotes Brownian motion, k is the index of the causal variable, and K is the total number of causal variables. The distribution over causal variables induces the following causal Markov factorization
K
p(zb)
(8) = II ((z pa()))
(84)
j=k
where (b) is observed causal variable k. The generative process is the solution to the following reverso-time SDE
[ - 3r -
(x)
3.V me log)( dt + BBew
(85)
where w denotes Brownian motion (noise) in the reverse SDE. The reverse SDE represents the process of strengthening causal relations between variables.
Learning Framework. The authors propose Diff-SCM, a diffusion-basod causal model enable counterfac- tual estimation. The model is trained, as described above, with the reverse diffusion gradually strengthening the causal relationships between variables. In order to enable interventions and counterfactual estimation,
48

---

Published in Transactions on Machine Learning Research (05/2024)
4a))
t = 0
t=
y
Yo
Yt
u(L)
2222
X
Xt
4(2)
Figure 26: Diffusion Causal Models (Sanchez & Tsaftaris, 2022)
the authors propose an anti-causal predictor that trained for each edge the causal graph. To intervene on node z(1) and see its effects on node 2((),, we need to estimate p(x(k) (do(=()) = x())). However, this dis- tribution be sensitive to changes in the cause distribution p(x(*)). Thus, proposed in Schölkopf et al. (2012), it would be easier to estimate p(z(i)=(()) and use Bayes' rule to estimate the original distribution. This objective aims to help guide the intervention toward the desired counterfactual class, similar to how classifier guidance is used to generate samples from a specified class label (Dhariwal & Nichol, 2021). The sampling process guided by the anti-causal gradient (o (2x(0))2)1)) The anti-causal predictor is trained separately from the diffusion model.
Counterfactual Inference. After training both the anti-causal predictor and diffusion model, the authors outline three-step procedure to perform counterfactual inference using their framework. First, abduction is performed via forward diffusion using the SDE from Fq. (83) to infer the exogenous noise terms for the factual observation. Then, action is performed by intervening on the desired variable and removing edges between the intervened variable and ils parents. Finally, prediction performed via reverse diffusion controlled by the anti-causal gradients to obtain the counterfactual. First, we perform the intervention follows:
6 (z) ,1 sVI V ait log logpo(sderla!""
(86)
where is the UNet of the diffusion model, is hyperparameter that controls the scale of the anti-causal gradient, and a₂ is the variance parameter, V 2(th log Ps (=) (1 ((())) is the anti-causal gradient of the predictor
that estimates the counterfactual class of causal variable j, FO given the causal variable r
(k)
Then, we sample the counterfactual
(k) F
are
It-1 f- va--1
+ Va--ie
(87)
The authors model the causal variables in the diffusion process as a causal relationship between class label (x(")) y) and image sample (x(1) = x), as shown in Figure 26. For example, we can consider the relationship between a digit class and the corresponding image of the digit. from the MNIST dataset. Thus, have the causal graph
Example 7 Given two variables x (image) and y (label), where y - E, we can model the interventional distribution p(a] do(Y = =y)) using anti-causoa gradients. Specificully, we have the following
logogp(Y = y/z)
(88)
One drawback of Difl-SCM is that it is limited to the bivariate setting. It is not clear that the method would able to scale to larger causal graphs. However, this approach could be a feasible solution for the specific use case of labels and images. There have been other lines of work utilizing the diffusion framework to study counterfactual explanations (Goyal al., 2019; Augustin al., 2022; Jeanneret et al., 2022), which is a related objective.
49

---

Published in Transactions on Machine Learning Research (05/2024)
4.4 Other works
In this survey, we only cover counterfactual image generation methods that utilize the SCM formalism. However, there are other lines of work that utilize the potential outcomes framework (Rubin, 2005) or counterfactual inference methods that do not focus on controllable generation.
Potential Outcomes. Bose et al. (2022) propose causal probing of deep generative models (CAGE), a framework for controllable counterfactual generation in latent variable models based on causal reasoning. Given some data x explained by attributes, perhaps in the form of annotated labels, CAGE proposes to estimate the average treatment effect of attributes considered to be effect variables. Unlike existing work such as CausalGAN and CausalVAF, which assume apriori that certain variables are causally related, CAGE aims to infer the causal relationships that are inherently captured by an arbitrary generative model capable of generalizing outside the training dataset. Given a pre-trained model, such as a VAE, CAGE studies the treatment effects of deep generative models. In the context of the potential outcomes (PO) framework (Rubin, 2005), the fundamental problem in causal inference is the inability to mcasure the effect of two different treatments simultaneously. To circumvent this problem, the authors leverage generalive models to explicitly generate counterfactuals via some latent space manipulation strategy. The authors propose the notion of generative average treatment effect (GATE), which is an extension of the average treatment effect (ATE) from the potential outcomes framework to quantify the difference in outcomes for factual and counterfactual generations.
Counterfactual Inference. Rather than counterfactual generation, other related works focus on tractable counterfctual inference from non-image data (i.e., tabular data).
CAREFL. Khemakhem et al. (2021) propose causal autoregressive flows (CAREFL) to model causal mechanisms in bivariate SCMs. In addition to approximating interventional distributions, the authors show that the invertible nature of flows allows for tractable counterfactual inference. The authors show that a generalized form of the additive noise model is also identifiable if the noise from a Gaussian distribution. Based on this result, the authors develop a causal discovery method for bivariate causal models. The approach utilizes the tractable likelihood estimation capability of flows to check which causal direction produces the highest likelihood. However, CAREFL is limited to interventions on root nodes and does not evaluate non-root node interventions.
VACA. Sanchez-Martin et al. (2022) propose variational causal graph encoders to generalize CAREFL to arbitrary causal graphs without parametric assumptions on the causal model. The general idea here is to use a variational graph autoencoder to learn the causal relationships among causal variables given causal graph adjacency matrix A. First, GNN encoder, with no hidden layers, maps the original data x to corresponding latent variable with the graph structure spec- ified by A. Then, a GNN decoder, with the number of hidden layers at least the length of the longest shortest path between any two causal variables, models the structural mechanisms bet ween causal variables via message passing. In this context, if i then 2ạ must be used to reconstruct Zy. Interventions are performed by modifying the desired variable and the causal adjacency matrix. Further, counterfactual inference is performed by replacing the desired variable from the factual GNN encoder output with the variable from the intervened GNN encoder output and decoding.
DCM. Chao et al. (2023) propose a diffusion- based causal model (DCM), a framework to learn causal mechanisms to enable sampling under interventions and counterfactual inference. Given n causal variables (Ii,..., (in the form of observed data) and a causal graph, DCM proposes to learn separate diffusion model for each causal variable. The authors formulate a forward and reverse diffusion process a modification of the DDIM (Song et al., 2021a) objective. The forward diffusion process adds noise to the causal variable until at time step T, the variable a! is the SCM noise term for causal variable 2;, Then, in the reverse diffusiin process, the parameterized noise predictor Eg is additionally conditioned on causal variable z 's denoised parents Ipa, That is, for the 69 paramoterization, we have ca :p,,,t). t). Counterfactual inference facilitated by performing abduction via forward diffusion, action by intervening on a causal variable, and prediction by performing reverse diffusion (in topologiaal order of the known causal graph).
50

---

Published in Transactions on Machine Learning Research (05/2024)
4.5 Discussion
Advantages. Using traditional controllable generation approaches, such as GANs, to conditionally gener- ate image samples has been an important direction of research for diverse data generation. However, such methods are simply conditional and do not encode any causal information. Thus, one cannot generate coun- terfactual images relative to factual observations. Controllable counterfactual generation (CCG) methods primarily work by modeling causal relationships between labels and high-dimensional observations. Thus, one could interpret this class of methods as form of structural mechanism learning. So, we can ask the question, if the image labels were different, what would the image look like? The generative component of causal representation learning methods is also capable of counterfactual generation upon intervent on latent variables. However, in CCG, the main task is to generate high-fidelity image counterfactuals when given access to observed labels. CCG can be utilized to generate high-quality counterfactuals that can be used to augment datasets for more robust training. Further, the ability to generate novel scenarios never before observed through counterfactual inference can be crucial for applications in medicine, where patient data is scarce and often difficull to obtain due to privacy concerns. Finally, although the identifiability of counterfactuals can pose challenge (Nasr-Esfahany & Kieiman, 2023), unlike representation learning, we do not have to worry about representation identifiability.
Limitations. The main disadvantage of causal controllable generation methods is the strong assumptions that are often required to be made, such as no unobserved confounding. Implicit generative models, such as CausalGAN, can often be quite difficult to optimize due to mode collapse. However, GAN-based methods can be useful for data augmentations to account for confounders (c.g.. CONIC). Thus, explicit density estimating frameworks such as VAF-based, flow-based, and diffusion-based approaches are promising in controllable counterfactual generation tasks. A combination of VAE and flow-based methods, such as DSCM, can be used to tractably estimate distributions over causal variables and counterfactuals in principled fashion using Pearl's counterfactual analysis (i.e., abduction, action, prediction). Diffusion models have shown superior generated image sample quality compared to GANs and thus can be used to generate high-quality counterfactuals. Lastly, most of the CCG methods are not scalable to the scenario where there are dozens of causal variables.
5 Evaluation Metrics
In this section, we list several important metrics to evaluate learned causal representations and generated counterfactuals. While some metries are quite heuristic, others have strong ties to the theoretical properties of generative models.
5.1 Metrics for Causal Representation Learning
Disentanglement, Completeness, and Informativenoss (DCI). The DCI metric (Eastwood & Williams, 2018) quantifies the degree to which ground-truth factors and learned latents are in one-to-one cor- respondence. The DCI disentanglement (D), completeness (C), and informativeness (I) scores are computed based on a feature importance matrix quantifying the degree to which each latent code important for pre- dicting each ground truth causal factor. The feature importance matrix is computed using gradient-boosted trees. The informaliveness (T) score is the prediction error in the latent factors predicting the ground-truth generative factors. We train a generative model on a dataset with underlying true generative factors E R and obtain the latent representation E R Given 2, we can learn a mapping y = =f()) (often a regressor). We can train K such predictors, one for each of the K generative factors (i.e., Vi E {1...., K),y = f((z)) Thus, we get a matrix of relative importances R where Riy is the relative importance of 2; in predicting yj-
Disentanglement (D). The degree to which a representation disentangles the underlying factors of variation with each variable capturing at most one generative factor. Let P.₃ R2,/ Σk=0 be the probability of 2; being a strong predictor of yy- Then, the disentanglement score is defined as
/t//A>1 R.s
K-1
D. = (1+
Σ
Pix log Pas)
(89)
k=
51

---

Published in Transactions on Machine Learning Research (05/2024)
If 2 is strong predictor for only single generative factor, D; = 1. If 2; is equally important in predicting all generative factors, Dj = 0. Let Pi =; Ruj/ Enj Ray be the relative latent. code importances. The total disentanglement score is simply a weighted average of the individual D,
D= = D.D.
P.D:
(90)
Completeness (C). The degree which single latent code captures each underlying factor is defined as the completeness score
N Σ
Cj = (1+
Pag logn P.j)
(91)
where Paj obtained in the same manner as P.j. If single latent code contributes to the prediction of By, the completeness score C2 is 1. If all latent codes contribute equally to the prediction of z/ the completeness score is O.
Informativeness (I). The amount of information that a representation captures about the underly- ing factors of variation. The informativencd score is quantified by the prediction error, averaged over the dataset, between the true factor and the predicted factor. Informativeness requires perturbations in a single latent factor be systematic and informative.
Eastwood et al. (2023) extend the DCI framework and incorporate explicitness (E), which captures the ease-of-use of the representation, and size (S), which captures the amount of information that could be represented for evaluating representations. Further, they show that the DCL is a robust metric that has strong connections to lincar and permutation--qqiiallent identifiability
Interventional Robustness Score (IRS). is often important that we achieve the robustness of groups of features in the latent variable with respect interventions on groups of generative factors. To evaluate how changes in the generative factors affect the latent factors, one compute the interventional robustness score (TRS) (Suter et al., 2019), which is similar to an R² value. TRS involves computing a wost-interventional discrepancy that quantifies how much the learned latent factor shifts when intervening on extrinsic generative factors while keeping the generative factors we are interested in capturing at their predefined values. IRS has ties to the DCI metric and subsumes disentanglement as a special case. An IRS score of 1 implies perfect robustness and implics no robustness.
Mean Correlation Coefficient (MCC). The mean correlation coefficient is obtained by first computing the Pearson correlation matrix C' Rax= between the learned representation and ground-truth latent variables. Then, the MCC is computed as MCC = maKrépermutations - [Cr, This metric has been used to assess et al. (2022).
(disentanglement) in Khemakhem et al. (2020) and Lachapelle
Mutual Information Gap (MIG). Chen et al. (2018) propose the Mutual Information Gap, which is a classifier-free metric to measure the disentanglement of latent factors. The empirical mutual information between latent variable 2j and the ground-truth factor ys can be estimated using the joint distribution q(2,, = ((w)))))))ll
We can then define mutual information
1(2z;8k) Ey(es
log Σ ((3)(1p(((gn)) Te
H(g))
(92)
A high mutual information I, implies that consists of a lot of information about Sh and is maximal if there exists a bijection between 2j and y/k- To cover discrete cases, we can additionally normalize I, by the entropy (y)) (i.c., I(22; (yx)) Disentanglement, according
Definition 3, requires each learned factor be axis-aligned with the ground truth. Thus, the following Mutual Information Gap (MIG) metric enforces axis-alignment by measuring the difference between the top two latent variables with the highest
52

---

Published in Transactions on Machine Learning Research (05/2024)
mutual information
1((29)
max L((2z18))) 5410)
(93)
(wss
where j({) = arg max, (x we) and K is the number of known factors. The MIG significantly penalizes unaligned variables, which are a sign of entanglement.
Causal Disentanglement Score (CDS). The DCI metric sometimes cannot distinguish between statistical correlation and causal link. Leeb et al. (2022) propose the causal disentanglement score (CDS), a metric based on decoding and reencoding latent samples through a lalent response function, which helps to extract semantic information from the latent space without knowing how the information is encoded. The problem with evaluating disentanglement from only encoder is that some latent variables may be spuriously correlated with the true factors without. having a causal effect on the factors when generating new samples. The latent response function is defined as h = fo 9, where f is some encoder and g is some decoder. First, we compute the following latent response matrix, which quantifies the degree to which an intervention on latent dimension j causes a response in latent dimension i and can be interpreted as a weighted causal graph
1 - M33
hn()))
(94)
2
where (z))+=;)(2) refers to the interventional sample where the jth latent variable is sampled from the marginal of the aggregate posterior 2g (o(2)).
If we have access to ground-truth variables Y, a variant of the latent response function, called the conditioned response matric, can be used to quantify how much the learned factors correspond to the true ones. Tet Ye denote a causal variable of interest and Y-c denote all other causal variables except Ye. In order to condition the set of interventions on a specific factor Ye, one can choose a subset of observations that are semantically the same except for the factor Y. (i.e., Y-e is the same for the observations). Tn other words, we sample latent variables where all causal labels are identical except Y-- This guarantees that the sampled latent can be used as a valid intervention to invoke a response in the ground truth factor Y.. The conditioned response matrix is defined as follows
1 M - pxp --))o(Y_o) | [ - h=)
(95)
where M2, quantifies how much an intervention on each latent variable zj affects the ground-truth factors of variation ye- The causal disentanglement score (CDS) is an extension of the DCI metric take into account the learned generative process and is computed as follows:
CDS E, max. Mej
(96)
Ees Maj
Similar to the DCI metric, CDS is single-value score that quantifies the degree to which learned repre- sentation can discntangle the factors of variation.
5.2 Metrics for Controllable Counterfactual Generation
Counterfactual Latent Divergence (CLD). Sanchez & Tsaftaris (2022) propose the counterfactual la- tent. divergence (CLD) metric, which is used to measure the proximity and minimality of counterfactual samples. The CLD metric involves utilizing the Kullback-Leibler (KI.) divergence to measure the divergence between latent distributions. The authors propose a relative distance metric to ensure that the minimality property of counterfactuals is satisfied (i.e., proximity to factual). Based on the relative distance metric, the counterfactual latent divergence (CT.D) is the TogSumFxpo of two probability measures, one enforcing minimality and one enforcing larger distances from factual.
xf)), a VAE is used to project the samples to a low-dimensional latent representation Pi, a, F(x Then, we can measure the divergence ((=(),=0)) (1) D (N(H, The
That is, for a pair (x().=0))
53

---

Published in Transactions on Machine Learning Research (05/2024)
(D)
authors define two sets SF and Scr. The set Sp consists of all relative distances comparing the factual T'p' to all possible samples with the same factual class. A low P(Sp 2 ( enforces larger distances from the factual class. The set SCF consists of all relative distances comparing the factual a0 to all possible samples with the same counterfactual class. In this case, low P(ScF S D(= z(2)) indicates that the counterfactuals satisfy minimality. The counterfactual latent divergence is simply defined as
CLD = log(exp P(SoF S D(ECP.=P)) + exp P(SeF 2 D(ECP:"F)) (1))
(97)
Axiomatic Evaluation. Monteiro et al. (2023) propose evaluation metrics based on axiomatic principles of counterfactuals. Specifically, the soundness of counterfactuals can be formulated in terms of effectiveness (interventions properly affect the desired variables), composilion (an identity intervention renders all vari- ables unaffected), and reversibility (mapping between observations and counterfactuals is deterministic and invertible) (Pearl, 2009; Galles & Pearl, 1998). Let x - g(e, pa) be the causal mechanism generating æ from its noise term e and parents pa. Then, for counterfactual analysis, we have x' g(abduction(z, pa), pa"), where represents the counterfactual. We can then generalize this counterfactual function and define func- tion x² (zz pa, pa') with the samne parameters. Suppose we have approximate counterfactusl function f in place of the ideal counterfactual function f. The three axiomatic metrics to evaluate the soundness of counterfactuals are formally defined as follows
Composition. We can measure how much the approximate model deviates from the true model by computing the distance between the original observation and the null transformation (trivial intervention) applied to the observation m times. This is done to capture any sources of corruption from the approximate model. Formally, we have that the following should be low
composition (xx,pa) (x, pa) = (x, f(m) (x.pa, pa. pa))
(98)
where dx is an arbitrary distance metric.
Reversibility. In order for the mechanism to be invertible, the cycle-consistency property must hold. So, we can compute the distance between the original observation and the output after applying the transformation m times. First, we apply î to perform abduction and map to a counterfactual x via counterfactual parents pa'. Then, apply again to map back to by performing abduction with the counterfactual x" via the original parents pa. That is, have the composite function p(x, pa. pa") = f(f(z, pa, pa)),,),, pa)). Formally, reversibility is defined as follows
(eversibility.(0)) (ar, pa, pa") = dx (a, (mm)))))))))) (x. pa, pa"))
(99)
Effectiveness. It is more difficult to measure effectiveness objectively. Thus, effectiveness can be measured individually for each parent of the observation by constructing an oracle function Paa(:), implemented as a regressor or classifier, that returns the value of the parent pag given the observed data. Formally, the effectiveness of each parent is defined as follows
effectiveness: (x. pa, paa) )- = ds (Pak (fe( pat, pay, paj)).
(100)
where de(:.-) is distance metric.
5.3 Discussion
In the domain of causal representation learning, especially causal disentanglement, the two most utilized metrics are DCI and MCC since they are theoretically grounded and have strong connections to identifiability theory (Eastwood et al., 2023). However, we have listed several other metrics (c.g., IRS, CDS, etc.) that are also promising in evaluating disentanglement and causal representations. We note that most controllable counterfactual generation methods rely on qualitative visual analysis to evaluate performance. However, several metrics such as CLD, Compositionality, Reversibility, and Effectiveness were recently proposed as principled metries evaluate counterfactuals We believe these metrics and others that build from the foundational theory of counterfactual analysis should be used for evaluating generated counterfactuals.
54

---

Published in Transactions on Machine Learning Research (05/2024)
Synthetic Image
Real-World Image
Temporal
Blological
Video
& Medical
Pendulum
Water Flow
Causal Cunaalirruiit
CausalaDIdent
CelebA
MPI3D
MNIST
Morpho-MNIST
ImageNet
KITTIMask
Mass-Spring
CMU MoCap
Cartpole
MoSeg
Causal Pinball
Interv. Pong
Ball-in-Boxen
Voronol
CausalWorld
iTHOR
Perturb-Sed
Blobank
MIMIC-CXR
Dataset
Van
157
SCM-VAE
DEAR
ICM-VAE
TDRL
NCTRL
Discr. VAE
CITRIS
iCITRIS
BISCUIT
LCM
CausaGGA
CGN
CONIC
DSCM
CansalHVAE
Diff-SMM
Table 4: Datasets used in Causal Representation Learning and Controllable Counterfactual Generation
6 Datasets
In this section, we enumerate datasets that are often utilized in causal generative modeling tasks. We group them into four main categories: synthetic image datasets, real-world image datasets, temporal video datasets, and biological/modiie /edical datasets. We note that many CRL methods, especially those with more theoretical contributions, are also evaluated on a common non-image synthetic benchmark that involves generating random DAGs and transforming them via normalizing flows high-dimensional observational data. The Fédös-Rényí random graph model (Erdös & Rényi, 1960) is primarily used to generate synthetic data for experiments in Unpaired Multi-domain CRL (Sturma et al., 2023), Score-based CRL (Varicı et al., 2023b), Nonparametric CRI. (von Kügelgen et al., 2023), Interventional CRI. (Ahuja et al., 2023), Linear CRL (linear mixing) (Squires et al., 2023), and Linear CRL (general mixing) (Buchholz et al., 2023). We show examples of some datasets and their causal graphs in Figure 27 to illustrate causal modeling. Table shows CRL and CCG methods and the visual datasets used for experiments.
6.1 Synthetic Image Datasets
Pendulum. The Pendulum dataset (Yang et al., 2021), consisting of roughly 7,000 images with 96 x 96 resolution, describes the interaction between four causal variables: pendulum angle, light position, shadow length, and shadow position. The pendulum system shows light source on the pendulum object that casts a shadow with a specific position and length. An example image and the causal structure of this system are shown in Figure 27a.
Water Flow. The Water Flow dataset (Yang et al., 2021), consisting of roughly 000 images with 96 96 resolution, describes the interaction between four causal variables: ball size, water height, hole position, and water flow. The flow system shows ball submerged in glass of water, which has hole in the side.
CausalCircuit.
The CausalCireuit, datasel (Brehmer et al., 2022) consists of 512 x 512 x 3 resolution images generated by 4 ground-truth latent causal variables: robot arm position, red light intensity, green light intensity, and blue light intensity. The images show a robot arm interacting with a system of buttons and lights. The data is rendered using an open-source physics engine. An example image and the causal structure of this system shown in Figure 27b.
55

---

Published in Transactions on Machine Learning Research (05/2024)
Angle
Light Position
Robot Anm
Gender
Smile
Blue Light
Green Light
Shadow Longth
Shadow Position
Narrow Eyes
Mouth Open
Red Light
(a) Pendulum
(b) CausalCircuit
(c) CelebA-Smile
Figure 27: Causal Graphs of Selected Example Datascts used in Causal Representation Learning
Causal3DIdent. The 3DIdent dataset (Zimmermann et al., 2021) is a benchmark created for evaluating the identifiability latent variable models. The dataset consists of 224 x 224 resolution images containing hallmarks of natural environments such as shadows, lighting conditions, and 3D objects. The Causal3DIdent dataset (von Kügelgen al., 2021) consists of seven classes: Teapot, Hare, Dragon, Cow, Armadillo, Horse, and Head with causal graph over the latent variables. The latent factors capture the spotlight position, object class, background color, spotlight color, object position, object rotation, and object color.
6.2 Real-World Image Datasets
CelebA. The CelebA dataset (Liu et al., 2015) is large-scale face attributes dataset with more than 200K images of celebrity faces, each with 40 attribute annotations. We can imposc causal structure among any subset. of the 40 attributes, such as CelebA-Smile, as shown in Figure 27c.
MPI3D. The MPI3D dataset (Gondal et al., 2019) consists of over one million images of physical 3D objects with seven factors of variation, such as object color, shape, size, and position.
MNIST. The MNIST dataset (Deng, 2012) consists of 70K, grayscale images of handwritten digits (zero to nine) with 28 x 28 resolution. Several methods utilize the MNIST dataset to generate counterfactual samples of different digits by modeling the causal mechanism between class labels and images.
MorPhoMNIIST. The MorphoMNIST dataset (Castro et al., 2019) is produced by applying morphological transformations on the original MNIST handwritten digit dataset. The digits can be described by measurable shape attributes such as stroke thickness, stroke length, width, height, and slant of digit. Pawlowski et al. (2020) impose 3-variable SCM to generate the morphological transformations. The authors define stroke thickness cause of the brightness of each digit. That is, thicker digits are often brighter, whercas thinner digits are dimmer.
ImageNet. The ImageNet dataset (Deng et al., 2009) consists of over 14 million annotated images with 1000 object classes. Image Net has been utilized primarily in counterfactual generation methods such as those proposed by Sauer Geiger (2021) and Sanchez & Tsaftaris (2022).
6.3 Temporal Video Datasets
The KiTTiMask vidco dataset (Klindt et al., 2021) includes pedestrian sogmentation masks sampled from KITTIMOS, an autonomous driving benchmark. In each time frame, the position and scale of pedestrian masks measured.
KiTTiMask.
56

---

Published in Transactions on Machine Learning Research (05/2024)
Mass-Spring system. The Mass-Spring video dataset (Li et al., 2020) consists of ball movement rendered in color and invisible springs. There are ten causal relations, where six are set connected and four are disconnected.
CMU MoCap. The MoCap dataset (Yao et al., 2022b) consists of 3D point clouds of skeleton-basee signals. Tt a human motion capture dataset with various motion capture recordings (e.g., walking, jumping, etc.) performed by over 140 subjects. Each activity consists of several trials of recordings. The skeleton-basee signal measurements have 62 observed variables that correspond to the locations of joints in the human body at each time step.
Cartpole. The Cartpole video dataset (Huang et al., 2022) consists of a cart and a vertical pendulum attached to the cart using pivol joint. The cart is allowed to move laterally left or right. The ultimate goal is to prevent the pendulum from falling off the cart by applying force on the cart to move it, where the action space is defined by moving left or right. The original dataset also consists of two change factors: varying gravity/mass of the cart and the noise level of the image. These factors correspond to different domains. The two causal variables in this system are the cart position and the pole angle.
MoSeq. The MoSeq dataset (Wiltschko et al., 2015) is a video datasct of mice exploring an open field. The temporal observations are taken to be the first 10 principal components of the video data.
Causal Pinball. The Causal Pinball dataset (Lippe et al., 2023a) implements the idea of the popular game Pinball. In Pinball, the user controls two fixed paddles on the bottom of a field and tries to hit the ball such that it collides with various objects in the playground to score points. In Causal Pinball, there are 5 main components: the ball, left paddle position, right paddle position, bumpers, and the score.
Interventional Pong. The Interventional Pong dataset (Lippe et al., 2022b) models the dynamics of the game Pong. The underlying causal factors in this system are ball x-position, ball y-position, ball velocity, left paddle position, right paddle position, and the score.
The Ball-in-Boxes dataset (Lippe el al., 2022b) is a synthetic datasel consisting of 32 x 32 images of a ball in 2D space that can be in one of two boxes. The ball can move within the box it is in but requires an intervention to move boxcs. The dynamics of the system are governed by two causal variables: ball x-position and ball y-position, which both follow truncated Gaussian distribution. The dataset is generated by using sequence of 100K frames and sampling intervention targets from Bernoulli distribution.
Ball-in-Boxes.
Voronoi Benchmark. The Voronoi Benchmark dataset (Lippe et al., 2022b) is a synthetic one created by generating one sequence with 150K timesteps, during which single-target interventions may have been performed. The interventions are sampled with 1/(n + 2) for each variable and with 2/(n + +2 for the observational regime, where is the number of patches that 32 32 image is partitioned into (also representing the causal variables). The n patches are transformed via normalizing flow to obtain hues that are mapped to RGB space to render an image.
Causal CausalWorld. World. The Causal (Ahmed et al., 2021) is a benchmark for causal structure and representa- tion learning in a robotic manipulation environment that a simulation of an open-source robotic platform. The dataset consists of 128 128 resolution images from many tasks such as the robot assembling 3D complex shapes given a set of blocks, similar to how children reason in the world. The dataset includes underlying causal factors such as the robot and object masses, sizes, and colors.
iTHOR. The iTHOR dataset is one of the datasets from The House Of inteRactions (THOR) framework (Kolve et al., 2022). It consists of photo-realistic 3D door scenes, where AI agents can navigate the scenes and interact with objects to perform different tasks. iTHOR is the original set of scenes that includes 120 room-sized scenes, covering bedrooms, bathrooms, kitchens, and living rooms. The causal variables in this system are represented by objects in the scene and actions taken upon them.
6.4 Siological/Medicaa Datasets
Perturb-Seq. The Perturb-Seq (Norman et al., 2019) is a biological dataset created to study genetic interactions and phenotypic outcomes. Perturb-Seg is high-dimensional dataset that measures the tran-
57

---

Published in Transactions on Machine Learning Research (05/2024)
scriptional response of cells under different genetic perturbations. This dataset has been used to study interventional causal representation learning (Zhang et al., 2023b). Their approach utilizes both perturbed and unperturbed data samples to predict the effect of unseen combinations of interventions.
UK Biobank. The UK Biobank dataset is from the UK Biobank Imaging Study (Sudlow et al., 2015) that uses the FSL neuroimaging toolkit to preprocess 3D T1-weighed brain MRI scans. The dataset also provides age, biological sex, and precomputed brain and ventricle volumes for each brain MRI sample.
MIMIC-CXR. The MIMIC-CRR dataset (Johnson et al., 2019) consists of chest X-ray images with reso- lution 192 x 192 and several corresponding attributes such as sex, race, age, disease, etc.
7 Applications
In this section, we first focus on three important aspects of Al trustworthiness: fairness, privacy preservation, and robustness against distribution shift, in the context of causal generative modeling. The incorporation of these AI trustworthiness properties is commonly required in various applications. We then delve into the application of causal generative modeling in precision medicine due to the necessity of explainable generative models in the medical domain. Finally, we discuss an application of causal representation learning in biological sciences, where one often has access to an abundance of multi-domain data.
Achieving Fairness in Causal Generative Models. Achieving fairness is an imperative task in gener- ative model learning. Zemel et al. (2013) first study fair representation learning and formulate fairness as an optimization problem of finding a good representation of the data that encodes the data as much as pos- sible while simultaneously obfuscating the membership information. Louizos et al. (2015) extend the VAE objective with a Maximum Mcan Discrepancy to ensure independence between the latent represcntation and the sensitive factors. Song et al. (2019a) further propose an objective for learning maximally expressive representations subject to fairness constraints and allow the user to control the fairness of the representations by specifying limits on unfairness. Locatello et al. (2019a) study the downstream usefulness of disentangled representations through the lens of fairness.
For data generation via GAN models, Xu et al. (2019) develop causal fairness-aware generative adversarial networks (CFGAN), which ensures various causal fairness criteria based on a given causal graph. CFGAN extends the FairGAN model (Xu et al., 2018) that achieves fair data generation in terms of disparate treatment and disparate impact. CFGAN adopts two generators whose structures are purposefully designed to reflect the structures of causal graphs and interventional graphs. The two generators can respectively simulate the underlying causal model that generates the real data and the causal model after the intervention. On the other hand, two discriminators are used for producing a close-to-real distribution and achieving various fairness criteria based on causal quantitics simulated by generators.
Preserving Privacy in Causal Generative Models. Representation learning and generative modeling enable domain users to use representations or generated data for downstream tasks, which, to some extent, improves the privacy protection of the original sensitive data. However, relcasing models generated data could still be detrimental since it may umintentionally leak sensitive information to adversaries. Differential privacy (DP) (Dwork et al., 2006) is a popular mechanism for training machine learning models with bounded leakage about the presence of specific individuals in the training data. There have been a few works (Xie et al., 2018; Zhang et al., 2018; Chen et al., 2020; Torkzadehmahani et al., 2019) on training GANs in a differentially private way, and some recent works (Weggenmann et al., 2022; Yang et al., 2022) on training VAEs in a differentially private way. However, there has been no study on achieving differential privacy in causal generative models.
Most works model based on the differentially private stochastic gradient descent (DP-SGD) algorithm (Abadi et al., 2016). The DP-SGD algorithun involves clipping the gradients to bound sensitivity, adding calibrated noise to the sum of clipped gradients, and using the moment accountant technique to keep track of the privacy budget during backpropagation. The dp-GAN algorithm (Zhang et al., 2018) achieves DP by injecting random noise in the SGD training of the discriminator. Similarly, the DPGAN algorithm (Xie al., 2018) adds noise to the gradient of the Wasserstein distance with respect to the training data. The GS-WGAN
GANs adopt the idea of tampering with the training process of the target
58

---

Published in Transactions on Machine Learning Research (05/2024)
algorithm (Chen et al., 2020) selectively applies sanitization to a necessary and sufficient subset of gradients, thus enabling the exploitation of the theoretical property of Wasserstein GANs. Torkzadehmahani et al. (2019) develop a differentially private conditional GAN (DP-CGAN) training framework that can generate not only synthotic data but also corresponding labels. It further clips the gradients of discriminator loss on real and fake data separately, allowing the user to better control the sensitivity of the model to real (sensitive) data.
Weggenmann et al. (2022) propose end-to-end differentially private obfuscation mechanism called DP- VAE for generating private synthetic text data. The idea is to constrain the probabilistic encoder to facilitate differentially private latent sampling. Yang et al. (2022) propose a method that first maps the source data to the latent space through the VAE model to get the latent code, then performs a noise process satisfying metric privacy on the latent code, and finally uses the VAE model to reconstruct the syuthetic data.
Causal Representation Learning for Distribution Shift Modcling. Distribution shift refers to changes in data distribution between the training and testing phases, leading to a decline in model per- formance. In real-world scenarios, the data encountered during the testing or deployment phase often comes from a slightly different distribution due to various factors, such as temporal changes, geographical changes, or different data collection processes. This discrepancy can lead to a model that performs well on the training data but poorly on new, unseen data. Causal representation learning addresses distribution shifts by focusing on the stable and consistent causal structures underlying the data invariant across distribution shifts. The ability of a model to generalize well across distribution shifts or domains is termed out-o--distribution (OOD) generalization. Sun et al. (2021) propose a variational Baraaal--yyiia----- bassed framework that learns the underlying causal structure of the data and the source of distribution shifts from multiple training domains based on conditionally factorized priors about the latent factors. It specifies latent causal invariance models (LaCIM) that consider latent factors separable into latent semantic factors (e.g., the shape of an object) and variation factors (e.g., background, object detection), and a domain variable, as a selection mechanism to generate spurious correlations between the semantic and variation factors across distribution shifts. The latent seman- tie factors are direct causes of predictive variables (e.g., object class) and invariant to distribution shifts or domains. Lu et al. (2022) consider a causal generative model similar to LaCIM but assumes a more flexible conditionally non-factorized prior based on general exponential family distributions. The authors propose invariant Causal Representation Learning (iCaRL), which enables out-of-distribution (OOD) generalization based on learning from multiple training domains. Liu et al. (2021) propose a causal semantic generative model (CSG) for OOD generalization from a single training domain. CSG is similar to LaCIM but does not contain a domain variable to generate spurious correlations between the semantic and variation factors. The authors develop methods with a novel design in variational Bayes for efficient. learning and OOD prediction.
Causal Generative Modeling in Precision Medicine. Clinical decision support systems are powered by machine learning models but tend to learn only associations between variables in the data (Sanchez et al., 2022b). Thus, healthcare systems are prone to spurious correlations. In a field as high stakes medicine, it is paramount to use explainable and interpretable models to achieve robust knowledge about a system. Causal machine learning offers several potential directions to learn more robust models, including causal representation, discovery, and reasoning. When treating patients, healthcare professionals are often concerned with actionable insights from data, In the context of causal inference, this means treatment (interventional) decisions basod on a comparison between treated and a control group. Before reasoning about the effects of causal variables, we must. be able represent them. In the domain of medieine, the data can be highly complex, high-dimensional, and multimodal. Thus, it is critical to learn meaningful representations of the data before using it for decision-making.
Consider a heart application in Alzheimer's disease (AD) (Sanchez et al., 2022b). To understand the root causes of AD, many have recently proposed understanding it through a causal lens that takes into account several biomarkers and uses domain knowledge for deriving ground-truth causal relationships. For instance, we can consider causal variables such as chronological age, brain MRI image, and AD diagnosis. If we impose an SCM over these variables, guided by domain knowledge, we can properly model distribution shift scenarios. Further, we can learn semantically meaningful variables from MR images that may be causally related. Accurately modeling cansal dependencies leads to the ability to directly generate counterfactual scenarios by sampling from a counterfactual SCM. One can also model distribiiistiittiinn shifts in the
59

---

Published in Transactions on Machine Learning Research (05/2024)
data, such as population shifts. For example, if we train classifier on data from younger individuals with a predisposition for AD, our model would likely perform poorly when we try to extrapolate to older populations. However, one can intervene on the age variable in causal models to generate new instances of older individuals. Due to this invariance property, causal models prove to be robust to distribution shifts. Recently, there has been work using causal modeling in the medical imaging domain to synthetically generate counterfactuals using diffusion probabílistic models to analyze the health state of brain MRIs of patients and predict lesions from healthy and unhealthy patient data (Sanchez et al., 2022a).
Causal Representation Learning from Biological and Genomics Data. In biological sciences, one is often interested in discovering the underlying mechanisms governing biological systems. For example, if we want to study a specific disease and develop a suite of drugs to combat the disease, we would often want to know how proteins interact and the downstream effects of perturbations. Such a problem necessitates the use of causal modeling to explain complex biological processes. Unlike many other fields, biological sciences often utilize several tools that allow perturbational screens and multi-modal views (c.g., protcomics, transcriptomics, melabolomics, etc.) to oblain an abundance of interventional data or different. views on the same system. In the language of causality, such perturbation is equivalent to interventions. Some examples of interventions can be setting the expression of certain gene to zero (i.c., gene knockouts), chemical perturbations (adding drugs), and mechanical perturbations. Given access to such data, learning what causal variables underlie a system along with their cansal relationships can have significant implications for scientific discovery. There have been several recent works that explore identifiable causal representation learning (Squires et al., 2023; Zhang et al., 2023b) and intervention design in causal models (Zhang et al., 2023a) applied to sequencing data.
8 Challenges and Open Problems
Causal Representations under Nonlinear General Form SCMs. Although there have been efforts in causal discovery for general nonlinear SCMs, it has been limited to bivariate settings. It is still desirable to develop causal discovery algorithms and identifiability theory in this setting, especially in the purely observational setting. In general, causal structure identifiability has been explored for linear (and post- nonlinear) additive Gaussian noise SCMs. Despite the expressivity of this class of models, it can be quite a restricting assumption for universal approximability. In the context of causal representation learning, we seek to identify the causal factors and their rich causal structure. However, if we do not have access to interventional data, learning the causal structure with general-form nonlinear models is challenging. Even linear causal discovery methods (Zheng et al., 2018; Yu al., 2019) have their limitations and can only recover a supergraph of the true causal graph. As we try to learn causal representations with purely observational data, it becomess crucial to expand the scope of latent SCMs that we can model. An interesting direction that ties the causal discovery task to representation learning is using the Jacobian of invertible mechanisms to derive causal structure (Reizinger et al., 2023). Further, it may sometimes be more feasible to assume that there exists an inherent causal structure that latent variable models can extract from the training data (Leeb et al., 2022).
Causal Representation Learning from Weaker Forms of Supervision. A natural question in causal representation learning is how we learn ddentifiable representations as we decrease direct supervision. von Kügelgen et al. (2023) show that causal representation learning is possible without any parametric assump- tions. However, there are still open questions about how interventions on multi-node causal latent variables can generalize to arbitrary causal graphs. We note that there still lack of practical causal representation learning algorithms. Another challenge in learning causal representations is the so-called chicken-and-egg problem. That is, without causal variables, it is difficult to learn the causal structure and vice-versa. As highlighted in this survey, the data-generating paradigms fall into the observational, interventional, and counterfactual rungs of Pearl's Causal Hicrarchy. Thus, it is important to understand the limitations of each paradigm. For instance, the observational data paradigm may be restricted to simpler causal models and often requires stronger supervision signals such as labels. Interventional data can sometimes be hard to obtain in many domains and identifiability can be casily violated unless we have interventions for every causal variable in the system. Ground-truth counterfactual data pairs are impossible obtain in the real world since they represent. hypothetical samples, but they may be used an inductive bias through con-
60

---

Published in Transactions on Machine Learning Research (05/2024)
trastive learning. Due to these constraints, it is important to assess what type of models most feasible for real-world use cases to extract meaningful insights. A practical outlook on CRL from weak supervision is using multi-domain observational data, which can often be obtained due to access to large quantities of multi-modal data in the real world. Thus, results from multi-domain CRL methods (Sturma et al., 2023; Morioka & Hyvärinen, 2023; Ahuja et al., 2024) are especially of interest to bridge the gap between theory and practice in CRL. For example, there is plentiful multi-modal and multi-view data collected the bio- logical sciences from a variety of different tools. Some recent work formulates a general framework for causal representation learning from multiple distributions under sparsity constraint on the recovered causal graph in a completely nonparametric setup and without the assumption of interventional data (Zhang et al., 2024). This is a promising direction indicating the applicability of CRL methods in real-world systems.
Diffusion-based Causal Representation Learning. With the emergence of diffusion probabilistic mod- els as an effective generative modeling framework, even outperforming GANs in image sample quality, a natural research direction is to explore if we can Iearn causal representations using diffusion and score-basod generalive models. Preechakul al. (2022) and Mittal et. al. (2023) explore learning semantically meaningful representations in the diffusion framework. Mittal et al. (2023) propose learning a latent representation for cach timestep in the diffusion process to capture richer representations. This area is quite underexplored and an interesting direction could study the disentanglement of causal representations under different data- generating assumptions and compare the robustness of representations with those learned from VA E-bsedd models. Studying counterfactual image generation using diffusion- based causal representations is another interesting research direction.
Practical Assumptions for Causal Generative Models in Real-world Applications. Although there has been rapid progress in theoretical results for range of weakly supervised CRL settings, many assumptions are violated in real-world applications. Relaxing such assumptions to make way for models that can show tangible results on real-world datasets such as medical images is still necessary. Further, causal representation learning has not been explored in the natural language domain, which may require different assumptions and algorithuns altogether. Further, it is worth considering how to relax identifiability assumptions for downstream tasks such as domain generalization and transfer learning. Since CRL methods often depend on strong assumptions, such causal sufficiency, they are prone to the effeets of confounding in complex datasets. Thus, practical models should also focus on how to model confounders in a latent SCM to mitigate the effects of induced spurious correlations.
Metrics to Evaluate Causal Representations w.r.t Counterfactual Quality. Although correlated metrics such as DCI and Mean Correlation Coefficient (MCC) have been used to evaluate disentanglement of causal factors, there is still a lack of standardized metrics evaluate the disentanglement and counterfac- tual generation quality of representations and models. Future work should systematically study evaluation procedures that rigorously test the causal model's ability to learn disentangled representations, generate quality counterfactual samples, and achieve robust downstream OOD performance. Metrics such as the Frechet Inception Distance (FID) (Heusel et al., 2017) could be modified to consider counterfactual quality. Since knowing the underlying data-generating process is not always feasible, exploring how to evaluate coun- terfactual quality without access to the underlying lata-generating process is also an important direction. For instance, Monteiro et al. (2023) propose a general framework for evaluating image counterfactuals and derived distance metrics to compare counterfactual inference models.
Lack of Benchmark Datasets. The majority of the work in causal generative modeling, especially rep- resentation learning, considers synthetic data for which we know the underlying data-generating process. However, no standardized benchmark real-world dataset exists to study causal generative models. A push toward evaluating causal models on a set of benchmarks would go a long way to enable consistent. and trackable progress in the field. However, it is important to note that obtaining image data from causal interventions can be difficult in practice. Interventions from randomized control trials may not always be feasible. Further, obtaining counterfactual data for evaluating generated counterfactuals is impossible. Thus, the community must work towards unified framework for evaluating causal generative models. Further- more, it could be worthwhile to explore more robust datasets that are prone to confounding and learning causal models that violate causal sufficiency. For example, Reddy et al. (2022b) construct CANDLE, a dataset for causal disentanglement that includes confounding between generative factors. Moreover, con-
61

---

Published in Transactions on Machine Learning Research (05/2024)
structing datasets useful for a variety of downstream tasks is also important direction. For instance, Liu et al. (2023) construct CausalTrrileet, real-world intervention-centric causal representation learning bench- mark dataset. Each sample in the dataset consists of a pair of images captured before and after a high-level action. The dataset features high visual complexity, actionable counterfactual observations, and downstream interventional reasoning, which are lacking in existing datasets for causal representation learning.
Causality in Large-scale Generative Models. Recently, large-scale generative models, such as large language models (LLMs) (Zhao et al, 2023) and diffusion models (Yang et al., 2023), have shown impressive performative capability in language and vision tasks, respectively. Concepts are defined at higher level of abstraction in largo-scale generative models since they are trained on large and diverse data. Thus, perfect recoverability of concepts is impossible to achieve. Theoretical properties of the representation space of large- scale generative models would be an interesting direction to explore since even weak identifiability results could suggest the robustness and stability of such models to be used reliably for downstream tasks. The latent space of LLMs is certainly rich in structure that has not been explored well yet, espocially through causal lens. There has been some preliminary work using counterfactual language to understand model control in LLMs and pretrained text-to-image diffusion models by leveraging the linear representation space (Park et al., 2023; Wang et al., 2023). Another thrust of work studies mechanistic interpretability in large- scale models through causal abstractions (Wu et al., 2023). Looking forward, in addition to developing CRL algorithms from a set of assumptions, proving and empirically studying properties of the representation space of pretrained large-scale generative models through a causal interpretation can lead to a better understanding of model control and increase the applicability of these models in real-world systems.
Open Source Software. To help the transition from theory to the application of causal generative models in real-world systems and to share framework for the research community, it is important to implement an open-source library for causal representation learning and causal generative models. The disentanglement __ibbb package (Locatello et al., 2019b) is an open-source library developed for disentan- gled representation lcarning. The library consists of model training pipelines, pro-trained disentanglement models, data processing modules, evaluation metrics, and visualizations. The package contains standard un- supervised generative models, weakly supervised models, and applications fairness and abstract reasoning tasks. A similar library developed for causal representation learning and counterfactual generation methods would greatly benefit the community and perhaps be a step towards ciniicatioo-orienttdd causal generative modeling research. The library should provide comprehensive metrics and algorithms that domain users and developers can incorporate into their real systems and/or design custom causal representation learning models via the provided APIs.
9 Conclusion
In this paper, we presented a technical survey on causal generative modeling focusing on identifiable causal representation learning and controllable counterfactual generation. We taxonomized causal representation learning methods based on the assuned data-generating process from Pearl's causal hierarchy and coun- terfactual generation methods based on the type of generative model. We targeted our survey for both beginners to the field of causal generative modeling and experts looking for a review and provided an in- tuitive introduction, with respect to theory and methodology, to fundamental approaches in the field. In addition to hodh We covered common datasets and metrics used in causal representation learning and controllable counterfactual generation. We outlined promising applications of causality-basee generative models in trustworthy AI, precision medicine, and biology. Finally, we discussed several current challenges and potential directions for future work.
Acknowledgements
This work is supported in part by National Science Foundation under awards 1910284, 1946391 and 2147375, the National Institute of General Medical Sciences of National Institutes of Health under award P20GM139768, and the Arkansas Integrative Motabolic Rescarch Center at University of Arkansas.
onttps://githinn http:://..tttttccoooeeeeeeeeaaaa
62

---

Published in Transactions on Machine Learning Research (05/2024)
References
Martin Abadi, Andy Chu, Ian Goodfellow, H. Brendan McMahan, 1yy Mironov, Kunal Talwar, and Li Zhang, Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC conference on computer and communications securily, pp. 308 -38,, 2016.
Ossama Ahmed, Frederik Träuble, Anirudh Goyal, Alexander Neitz, Manuel Wuthrich, Yoshua Bengio, Bernhard Schölkopf, and Stefan Bauer. Causalworld: A robotic manipulation benchmark for causal structure and transfer learning. In International Conference on Learning Representations, 2021.
Kartik Ahuja, Divyat Mahajan, Yixin Wang, and Yoshua Bengio. Interventional causal representation learning. In Proceedings of the 40th International Conference on Machine Learning, pp. 372 407. PMLR, 2023.
Kartik Ahuja, Amin Mansouri, and Yixin Wang. Multi-domain causal representation learning via weak distributional invariances. In Proceedings of The 27th International Conference on Artificial Intelligence and Statistics, 2024.
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In Proceedings of the 34th International Conference on Machine Learning, Procoodings Machine Learning Research, pp.
PMLR, 2017.
Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprint Xr::::1907.0220 02899, 2020.
Maximilian Augustin, Valentyn Boreiko, Francesco Croce, and Matthias Hein. Diffusion visual counterfactual explanations. In Aduances in Neural Information Processing Systems, pp. 364-377, 377, 2022.
Elias Bareinboim, Juan D. Correa, Duligur Tbeling, and Thomas Ieard. On Pearl's Hierarchy and the Foundations of Causal Inference, pp. 507 556. Association for Computing Machinery, 1 edition, 2022.
Sara Beery, Grant Van Horn, and Pietro Perona. Recognition in terra incognita. In Proceedings of the European Conference on Computer Vision (ECCV), 2018.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspec- lives. IEEE Transactions on Pullern Analysis and Machine Intelligence, 35(8):1798 1828, 2013.
Yoshua Bengio, Tristan Deleu, Nasim Rahaman, Nan Rosemary Ke, Sebastien Lachapelle, Olexa Bilaniuk, Anirudh Goyal, and Christopher Pal. A meta-transfer objective for learning to disentangle causal mecha- nisms. In International Conference on Learning Representations, 2020.
Simon Bing, Urmi Ninad, Jonas Wahl, and Jakob Runge. Tdentifying linearly-mixed causal representations from multi-node interventions. In Conference on Causal Learning and Reasoning, 2024.
Joey Bose, Ricardo Pio Monti, and Aditya Grover, Controllable generative modeling via causal reasoning. Transactions on Machine Learning Research, 2022.
Diane Bouchacourt, Ryota Tomioka, and Sebastian Nowozin. Multi-level variational autoencoder: Learning disentangled representations from grouped observations. Proceedings of the AAAT Conference on Artificial Intelligence, 32(1), 2018.
Johann Brehmer, Pim De Haan, Phillip Lippe, and Taco Cohen. Weakly supervised causal representation learning. In Advances in Neural Information Processing Systems, 2022.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. In International Conference on Learning Representations, 2019.
Simon Buchholz, Goutham Rajendran, Elan Rosenfeld, Bryon Aragam, Bernhard Schölkopf, and Pradeep Kumar Ravikumar. Learning linear causal representations from interventions under general nonlinear mixing. Tn Advances in Neural Information Processing Systems, 2023.
63

---

Published in Transactions on Machine Learning Research (05/2024)
Daniel C. Castro, Jeremy Tan, Bernhard Kainz, Ender Konukoglu, and Ben Glocker. Morpho-MNIIT: Quan- titative assessment. and diagnostics for representation learning. Journal of Machine Learning Research, 20 (178), 2019.
Patrick Chao, Patrick Blöbaum, and Shiva Prasad Kasiviswanathan. Interventional and counterfactual inference with diffusion models. arXiv preprinl aix:22200 00860, 2023.
Dingfan Chen, Tribhuvanesh Orekondy, and Mario Fritz. Gs-wgan: A pradient-sanitiinn approach for learn- ing diffcrentially private generators. Advances in Neural Information Processing Systems, 533:12673-16644 2020.
Ricky T. Q. Chen, Xuechen Li, Roger Grosse, and David K Duvenaud. Isolating sources of disentanglement variational autoencoders. In Advances in Neural Information Processing Systems, 2018
Yang Chen, Yu Wang, Yingwei Pan, Ting Yao, Xinmci Tian, and Tao Mci. A style and semantic memory mechanism for domain generalization. In 2021 IEEE/CVE International Conference on Computer Vision (ICCV), pp. 9144 9153, 2021.
Pierre Comon. Independent component analysis, a new concept? Signal Processing, 36(3):287-314, 1994. Higher Order Statistics.
G. Darmois. Analyse générale des liaisons stochastiques: etude particulière de l'analyse factorielle linéaire. Revue de l'Institut International de Statistique / Review of the International Stotistical Institute, (1/12)) 2-8, 1953. ISSN 03731138.
Fabio De Sousa Ribeiro, Tian Xia, Miguel Monteiro, Nick Pawlowski, and Ben Glocker. High fidelity image counterfactuals with probabilistic causal models. In Proceedings of the 40th International Conference Machine Learning, pp. 7390-7425. PMLR, 2023.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248-28 258-,55, 2009.
Li Deng. The mnist database of handwritten digit images for machine learning research. IEEE Signal Processing Magazine, 29(6):141-144 142, 2012.
Prafulla Dhariwal and Alexander Quinn Nichol. Diffuusion models beat GANs on image synthesis. In Advances in Neural Information Processing Systems, 2021.
Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adamn Smith. Calibrating noise to sensitivity in pri- vate data analysis. In Theory of Cryptography: Third Theory of Cryptography Conference, pp. 265-284. Springer, 2006.
Cian Eastwood and Christopher K. I. Williams. A framework for the quantitative evaluation disentangled representations. In International Conference on Learning Representations, 2018.
Cian Eastwood, Andrei Liviu Nicolicioiu, Julius Von Kügelgen, Armin Kekić, Frederik Träuble, Andrea Dittadi, and Bernhard Schölkopf. DCI-ES: An extended disentanglement framework with connections identifiability. In International Conference on Learning Represenlations, 2023.
Paul Erdös and Alfred Rényi. On the evolution of random graphs. Publ. Math. Inst. Hungory. Acad. Sci., 1960.
G.D. Forney. The viterbi algorithm. Proceedings of the IEEE, 61(3):268 278, 1973.
David Galles and Judea Pearl. An axiomatic characterization of causal counterfactuals. Foundations of Science, 3:151-182, 1998.
64

---

Published in Transactions on Machine Learning Research (05/2024)
Muhammad Walced Gondal, Manuel Wuthrich, Djordje Miladinovic, Francesco Locatello, Martin Breidt, Valentin Volchkov, Joel Akpo, Olivier Bachem, Bernhard Schölkopf, and Stefan Bauer. On the transfer of inductive bias from simulation to the real world: a new disentanglement dataset. In Advances in Newral Information Processing Systems, 2019.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets, In Advances in Neural Information Processing Systems, 2014.
Yash Goyal, Ziyan Wu, Jan Ernst, Dhruv Batra, Devi Parikh, and Stefan Lee. Counterfactual visual explanations. In Proceedings of the 36th International Conference Machine Learning, pp- 2376 2384, 2019.
Luigi Gresele, Paul K. Rubenstein, Arash Mchrjou, Francesco Locatello, and Bernhard Schölkopf. The incomplete rosetta stone problem: Identifiability results for multi-view nonlinear ica. arXiv preprint arXiv:1905.0664. 2019.
Luigi Gresele, Julius Von Kügelgen, Vincent. Stimper, Bernhard Schölkopf, and Michel Besserve. Independent mechanism analysis, new concept? In Advances in Neural Information Processing Systems, 2021.
Arthur Gretton, Karsten M. Borgwardt, Malte J. Rasch, Bernhard Schölkopf, and Alexander Smola. A kernel two-sample test. Journal of Machine Learning Research, 2012.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In Advances in Neurul Information Processing Systems, 2017.
Trina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mo- hamed, and Alexander Lerchner. beta-VAE: Learning basic visual concepts with constrained variational framework. In International Conference on Learning Representations, 2017.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems, pp. 6840-6851, 2020.
Haruo Hosoya. Group-based learning of disentangled representations with generalizability for novel contents. In Proceedings of the 28th International Joint Conference on Artificial Intelligence, pp. 2506-2513, 2019.
Biwei Huang, Fan Feng, Chaochao Lu, Sara Magliacane, and Kun Zhang. AdaRL: What, where, and how to adapt in transfer reinforcement learning. In International Conference Learning Representations, 2022.
Aapo Hyvarinen, Hiroaki Sasaki, and Richard E. Turner. Nonlincar ica using auxiliary variables and gener- alized contrastive learning. arXiv preprint 1=Xxx:1805.0663 2018.
Aapo Hyvärinen and Petteri Pajunen. Nonlinear independent component analysis: Existence and uniqueness results. Newral Networks, (2(3):429-433 1999.
Dominik Janzing and Bernhard Schölkopf. Causal inference using the algorithmic markov condition. 1EEE Transactions on Information Theory, 56(10):5168-5111 2010.
C. Jarzynski. Equilibrium free-energy differences from nonequilibrium measurements: A master-equation approach. Phys. Rev. E, 56:5018 14500185005,, Nov 1997.
Guillaume Jeanneret, Loic Simon, and Frederic Jurie. Diffusion models for counterfactual explanations. Tn Proceedings of the Asian Conference on Computer Vision (ACCV), pp. 858-877 2022.
Alistair EW Johnson, Tom J Pollard, Nathanicl R Greenbaum, Matthew P Lungren, Chih-ying Deng, Yifan Peng, Zhiyong Lu, Roger G Mark, Seth J Berkowitz, and Steven Horng. Mimic-cxr-jpg, a large publicly available database of labeled chest radiographs. arXio preprint arxio: 2019.
65

---

Published in Transactions on Machine Learning Research (05/2024)
Jean Kaddour, Aengus Lynch, Qi Liu, Matt J. Kusner, and Ricardo Silva. Causal machine learning: A survey and open problems. Xiv preprint or arXiv:200..1 1 5575, 2022.
Nan Rosemary Ke, Olexa Bilaniuk, Anirudh Goyal, Stefan Bauer, Hugo Larochelle, Bernhard Schölkopf, Michael Curtis Mozer, Christopher Pal, and Yoshua Bengio. Neural causal structure discovery from interventions. Transactions on Machine Learning Research, 2023. ISSN 2835-8856.
Nyes Khemakhem, Diederik Kingma, Ricardo Monti, and Aapo Hyvarinen. Variational autoencoders and nonlinear iea: A unifying framework. In Proceedings of the Twenly Third International Conference on Artificial Intelligence and Statistics, 2020.
Tlees Khemakhem, Ricardo Monti, Robert Leech, and Aapo Hyvarinen. Causal autoregressive flows. Tn Proceedings of The 24th International Conference Artificial Intelligence and Statistics, 2021.
Hyunjik Kim and Andriy Mnih. Disentangling by factorising. In Proceedings of the 35th International Conference on Machine Learning, pp. 2649 2658. PMLR, 2018.
Diederik P Kingma and Max Welling. Auto-cncoding variational bayes. arXiv preprint arXtv:1112.611 2013.
David A. Klindt, Lukas Schott, Yash Sharma, Ivan Ustyuzhaninov, Wieland Brendel, Matthias Bethge, and Dylan Paiton. Towards nonlinear disentanglement natural data with temporal sparse coding. In International Conference on Learning Representations, 2021.
Murat Kocaoglu, Christopher Snyder, Alexandros G. Dimakis, and Sriram Vishwanath. CausalGAN: Learn- ing causal implicit generative models with adversarial training. In International Conference on Learning Representations, 2018.
Eric Kolve, Roozbeh Mottaghi, Winson Han, Eli VanderBilt, Luca Weihs, Alvaro Herrasti, Matt Deitke, Kiana Ehsani, Daniel Gordon, Yuke Zhu, Aniruddha Kembhavi, Abhinav Gupta, and Ali Farhadi. Ai2- thor: An interactive 3d euvironment for visual ai. Xiv preprint ar 1r:: 171.00577,, 2022.
Ancesh Komanduri, Yongkai Wu, Wen Huang, Feng Chen, and Xintao Wu. Sem-vac: Learning identifiable causal representations via structural knowledge. In IEEE Internalional Conference on Big Data (Big Data), 2022.
Aneesh Komanduri, Yongkai Wu, Feng Chen, and Xintao Wu. Learning causally disentangled representations via the principle of independent causal mechanisms. arXiv preprint arXiv:2306.012) 2023.
Sebastion Lachapelle and Simon Lacoste-Julien. Partial disentanglement via mechanism sparsity. In UAI 2022 Workshop on Causal Representation Leurning, 2022.
Sebastien Lachapelle, Pau Rodriguez, Yash Sharma, Katie E Everett, Rémi LE PRIOL, Alexandre Lacoste, and Simon Lacoste-Julien. Disentanglement via mechanism sparsity regularization: new principle for nonlinear ICA. In First Conference on Causal Learning and Reasoning, 2022.
Sébastien Lachapelle, Tristan Deleu, Divyat Mahajan, Ioannis Mitliagkas, Yoshua. Bengio, Simon Lacoste- Julien, and Quentin Bertrand. Synergies between disentanglement and sparsity: Generalization and iden- tifiability in multi-task learning. In International Conference on Machine Learning, 2023.
Felix Leeb, Stefan Bauer, Michel Besserve, and Beruhard Schölkopf. Exploring the latent space of autoen- coders with interventional assays. In Advances in Neural Information Processing Systems, 2022.
Yunzhu Li, Antonio Torralba, Anima Anandkumar, Dieter Fox, and Animesh Garg. Causal discovery in physical systems from vidcos. In Advances in Neural Information Processing Systems, 2020.
Wendong Liang, Armin Kekić, Julius von Kügelgen, Simon Buchholz, Michel Besserve, Luigi Gresele, and Bernhard Schölkopf. Causal component analysis. In Advances in Neural Information Processing Systems, 2023.
66

---

Published in Transactions on Machine Learning Research (05/2024)
Phillip Lippe, Taco Cohen, and Efstratios Gavves. Efficient neural causal discovery without acyclicity constraints. In International Conference. on Learning Represenlations, 2022a.
Phillip Lippe, Sara Magliacane, Sindy Löwe, Yuki M Asano, Taco Cohen, and Stratis Gavves. CITRIS: Causal identifiability from temporal intervened sequences. In Proceedings of the 39th International Conference on Machine Learning. 2022b.
Phillip Lippe, Sara Magliacane, Sindy Löwe, Yuki M Asano, Taco Cohen, and Efstratios Gavves. Causal representation learning for instantancous and temporal effects in interactive systems. In The Eleventh International Conference on Learning Representations, 2023a.
Phillip Lippe, Sara Magliacane, Sindy Löwe, Yuki M Asano, Taco Cohen, and Eistratios Gavves. Bis- cuit: Causal representation learning from binary interactions. In The 39th Conference on Uncertainty in Artificial Intelligence, 2023b.
Chang Liu, Xinwei Sun, Jindong Wang, Haoyue Tang, Tao Li, Tao Qin, Wei Chen, and Tie- Yan Liu. Learning causal semantic representation for aut-of-distributie prediction. In Advances in Neural Information Processing Systems, pp. 6155-6170, 2021.
Yuejiang Liu, Alexandre Alahi, Chris Russell, Max Horn, Dominik Zietlow, Bernhard Schölkopf, and Francesco Locatello. Causal triplet: An open challenge for intervention-centric causal representation learning. In Conference on Causal Learning and Reasoning, pp. 553 573. PMLR, 2023.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In 2015 2015.
IEEE International Conference on Computer Vision (ICCV), pp.
Francesco Locatello, Gabriele Abbati, Thomas Rainforth, Stefan Bauer, Bernhard Schölkopf, and Olivier Bachem. On the fairness of disentangled representations. In Advances in Neural Information Processing Systems, 2019a.
Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Ractsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentangled represen- tations. In Proceedings of the 36th International Conference Machine Learning, 2019b.
Francesco Locatello, Ben Poole, Gunnar Rätsch, Bernhard Schölkopf, Olivier Bachem, and Michael Tschan- nen. Weakly-superriseed disentanglement without compromises. In Proceedings of the 37th International Conference on Machine Learning, 2020.
Christos Louizos, Kevin Swersky, Yujia Li, Max Welling, and Richard Zemel. The variational fair autoen- coder. arXiv preprint #xXiv:1511000800 2015.
Chaochao Lu, Yuhuai Wu, José Miguel Hernández-Lobato, and Bernhard Schölkopf. Invariant causal repre- sentation learning for out-of-distribution generalization. In International Conference on Learning Repre- sentations, 2022.
Divyat Mahajan, Shruti Tople, and Amit Sharma. Domain generalization using causal matching. In Pro- ceedings of the 38th International Conference on Machine Learning, volume 139, pp. 7313-7324. PMLR, 2021.
Mchdi Mirza and Simon Osindero. Conditional generative adversarial nets. arXiv preprint arXiv:1411.1784, 2014.
Sarthak Mittal, Korbinian Abstreiter, Stefan Bauer, Bernhard Schölkopf, and Arash Mehrjou. Diffusion based representation learning. In Proceedings of the 40th International Conference on Machine Learning, pp. 24963-24982 2023.
Miguel Monteiro, Fabio De Sousa. Ribeiro, Nick Pawlowski, Daniel C. Castro, and Ben Glocker. Mcasur- ing axiomatic soundness of counterfactual image models. In The Eleventh International Conference on Learning Representations, 2023.
67

---

Published in Transactions on Machine Learning Research (05/2024)
Hiroshi Morioka and Aapo Hyvärinen. Causal representation learning made identifiable by grouping of observational variables. or preprint or arXiu:2310.15 157009 2023.
Arash Wasr-Esfahann and Emre Kiciman. Counterfactual (non-lideniffabblliy of learned structural causal models. arXiv preprint arXim:2301.0000 009031, 2023.
Arash Nasr-Esfahany, Mohammad Alizadeh, and Devavrat Shah. Counterfactual identifiability of bijective causal models. In Proceedings of the 40th International Conference on Machine Learning, pp. 25733 25754.
PMLR, 2023.
Ignavier Ng, Shengyu Zhu, Zhuangyan Fang, Haoyang Li, Zhitang Chen, and Jun Wang. Masked gradient- based causal structure learning. In Proceedings of the 2022 SIAM International Conference on Dala Mining (SDM), 2022.
Thomas M. Norman, Max A. Horlbeck, Joseph M. Replogle, Alex Y. Ge, Albert Xu, Marco Jost, Luke A. Gilbert, and Jonathan S. Weissman. Exploring genetic interaction manifolds constructed from rich single- cell phenotypes. Science, 365(6455):786 793, 2019.
OpenAL. Gpt-4 technical report, 2023.
George Papamakarios, Eric Nalisnick, Danilo Jimenez Rezende, Shakir Mohamed, and Balaji Lakshmi- narayanan. Normalizing flows for probabilistic modeling and inference. Journal of Machine Learning Research, 22(1), 2021.
Giambattista Parascandolo, Niki Kilbertus, Mateo Caruula,, and Bernhard Schölkopf. Learning inde- pendent causal mechanisms. In Proceedings of the 35th International Conference on Machine Learning, 2018.
Kiho Park, Yo Joong Choc, and Victor Veitch. The linear representation hypothesis and the geometry of large language models. In Causal Representation Learning Workshop at NewrIPS 2028, 2023.
Nick Pawlowski, Daniel Coelho de Castro, and Ben Glocker. Deep structural causal models for tractable counterfactual inference. In Advances in Neural Information Processing Systems, volume 33, pp. 857-869, 2020.
Judea Pearl. Causalily. Cambridge University Press, Cambridge, UK, 2 edition, 2009. ISBN 978-0-521- 89560-6. doi: (0.1017/CB09780511188811
Judea Pearl. The seven tools of causal inference, with reflections on machine learning. Communications of the ACM, 6243)654-00, 2019.
Judea Pearl and Dana Mackenzie. The Book of Why. Basic Books, New York, 2018. ISBN 978-0-465-09766-9.
Jonas Peters, Dominik Janzing, and Bernhard Schlkopf. Elements of Causal Inference: Foundations and Learning Algorithms. The MIT Press, 2017. ISBN 0262037319.
Konpat Preechakul, Nattanat Chatthee, Suttisak Wizadwongsa, and Supasorn Suwajanakorn. Diffusion autocncoders: Toward a meaningful and decodable representation. In JEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Proceedings of the 38th International Conference on Machine Learning, pp. 3748-8763. PMLR, 2021.
Abbavaram Gowtham Reddy, Saloni Dash, Amit Sharma, and Vineeth N. Balasubramanian. Counterfactual generation under confounding. In NewrIPS 2022 Workshop on Causality for Real-world Impact, 2022a.
Abbavaram Gowtham Reddy, Benin Godfrey L, and Vinceth N Balasubramanian. On causally disentangled representations. Proceedings of the AAAT Conference on Artificial Intelligence, 36(7):8089 18990099-977,,, 2022b.
68

---

Published in Transactions on Machine Learning Research (05/2024)
Patrik Reizinger, Luigi Gresele, Jack Brady, Julius von Küggelgen, Dominik Zietlow, Bernhard Schölkopf, Georg Martius, Wieland Brendel, and Michel Besserve. Embrace the gap: Vaes perform independent mechanism analysis. In Advances in Neural Information Processing Systems, pp. 12040 12057, 2022.
Patrik Reizinger, Yash Sharma, Matthias Bethge, Bernhard Schölkopf, Ferenc Huszár, and Wieland Brendel. Jacobian-based causal discovery with nonlinear ICA. Transactions on Machine Learving Research, 2023.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagatton and approximate inference in deep generative models. Tn Proceedings of the 31st. International Conference on Machine Learning, volume 32, pp. 1278 1286. PMLR, 2014.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. arXiv preprinl arXiv:1505. 04597, 2015.
Donald B. Rubin. Causal inference using potential outcomes: Design, modeling, decisions. Journal of the American Statistical Association, 100(469):322 331, 2005, ISSN 01621459.
Sorawit Saengkyongam, Elan Rosenfeld, Pradeep Ravikumar, Niklas Pfister, and Jonas Peters. Identifying representations for intervention extrapolation. arXiv preprint arXiv:2310.0422 04295, 2023.
Pedro Sanchez and Sotirios A. Tsaftaris. Diffusion causal models for counterfactual estimation. In First Conference on Causal Learning and Reasoning, 2022.
Pedro Sanchez, Antanas Kascenas, Xiao Liu, Alison Q. O'Neil, and Sotirios A. Tsaftaris. What is healthy? generative counterfactual diffusion for lesion localization. In Deep Generative Models, pp. 34-44. Springer Nature Switzerland, 2022a.
Pedro Sanchez, Jeremy P. Voisey, Tian Xia, Hannah I. Watson, Alison Q ONeil, and Sotirios A. Tsaftaris. Causal machine learning for healthcare and precision medicine. Royal Society Open Science, 2022b.
Axel Sauer and Andreas Geiger. Counterfactual generative networks. In International Conference Learn- ing Representations, 2021.
Bernhard Schölkopf, Dominik Janzing, Jonas Peters, Eleni Sgouritsa, Kun Zhang, and Joris Mooij. On causal and anticausal learning. In Proceedings of the 29th International Conference on Machine Learning, pp. 459 466, 2012.
Bernhard Scholkopf, Francesco Locatello, Stefan Bauer, Nan Rosemary Ke, Nal Kalchbrenner, Anirudh Goyal, and Yoshua Bengio. Toward Causal Representation Learning. Proceedings of the IEEE, 109(5): 612 634, May 2021.
Bernhard Schölkopf and Julius von Kügelgen. From statistical to causal learning. arXvi preprint arXiv:2204.00607, 2022.
Xinwei Shen, Furui Liu, Hanze Dong, Qing Lian, Zhitang Chen, and Tong Zhang. Weakly supervised disentangled generative causal representation learning. Journal of Machine Learning Research, 23(241): 1 55, 2022.
Jascha Sohl-Dickstein. Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibriun thermodynamics In Proceedings of the 32nd International Conference on Machine Learning, volume 37, pp. 2256 2265. PMLR, 2015.
Jiaming Song, Pratyusha Kalluri, Aditya Grover, Shengjia Zhao, and Stefano Ermon. Learning controllable fair representations. In The 22nd International Conference on Artificial Intelligence and Statistics, pp. 2164 2173. PMLR, 2019a.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In International Conference Learning Represendations, 2021a.
69

---

Published in Transactions on Machine Learning Research (05/2024)
Xiangchen Song, Weiran Yao, Yewen Fan, Xinshuai Dong, Guangyi Chen, Juan Carlos Niebles, Eric Xing, and Kun Zhang. Temporally disentangled representation learning under unknown nonstationarity. In Advances in Neural Information P Processing Systems, 2023.
Yang Song, Sahaj Garg, Jiaxin Shi, and Stefano Ermon. Sliced score matching: A scalable approach density and score estimation. arXiv preprint arXiv:1905.0705. 2019b.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole, Score- based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021b.
P. Spirtes, C. Glymour, and R. Scheines. Causation, Prediction, and Search. MTT press, 2nd odition, 2000.
Chandler Squires and Caroline Uhler. Causal structure learning: A combinatorial perspective. Foundations of Computational Mathematics, pp. 1-35, 2022.
Chandler Squires, Anna Seigal, Salil Bhate, and Caroline Uhler. Linear causal disentanglement via inter- ventions. In International Conference on Machine Learving, 2023.
Nils Sturma, Chandler Squires, Mathias Drton, and Caroline Uhler. Unpaired multi-domain causal repre- sentation learning. In Advances in Newral Information Processing Systems, 2023.
Cathie Sudlow, John Gallacher, Naomi Allen, Valerie Beral, Paul Burton, John Danesh, Paul Downey, Paul Elliott, and Martin Landray et al. Jane Green. Uk biobank: an open access resource for identifying the causes of wide range of complex diseases of middle and old age. PLoS Medicine, 12(13), 2015.
Xinwei Sun, Botong Wu, Xiangyu Zheng, Chang Liu, Wei Chen, Tao Qin, and Tie-Yan Liu. Recovering latent causal factor for generalization to distributional shifts. Advances in Neural Information Processing Systems, 34:16846 16859, 2021.
Raphael Suter, Djordje Miladinovic, Bernhard Schölkopf, and Stefan Bauer. Robustly disentangled causal mechanisms: Validating deep representations for interventional robustness. In Proceedings of the 36th International Conference on Machine Learning, 2019.
Pablo Sánchez-Martin, Miriam Rateike, and Isabel Valera. Vaca: Designing variational graph autoencoders for causal queries. Proceedings of the AAAT Conference on Artificial Intelligence, 36, 2022.
Casper Kaac Sonderby, Tapani Raiko, Lars Maaløe, Søren Kaac Sonderby, and Ole Winther. Ladder varia- tional autoeneoders. arXiv preprint or X6v::1600..0000 02282, 2016.
Esteban G. Tabak and Cristina Vilma Turner. A family of nonparametric density estimation algorithms. Communications on Pure and Applied Mathematics, 66, 2013.
Esteban G. Tabak and Eric Vanden-Eijnden. Density estimation by dual ascent of the log-likelihood. Com- munications in Mathematical Sciences, 8:217 2211-233,,1 2010.
Reihaneh Torkzadehmmaaani, Peter Kairouz, and Benedict Paten. Dp-cgan: Differentially private synthetic data and label generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognilion Workshops, pp. 0 0, 2019.
Frederik Träuble, Elliot Creager, Niki Kilbertus, Francesco Locatello, Andrea Dittadi, Anirudh Goyal, Bern- hard Schölkopf, and Stefan Bauer. On disentangled representations learned from correlated data. In Proceedings of the 38th International Conference Machine Learning, 2021.
A. M. Turing. 1.- COMPUTING MACHINERY AND INTELLIGENCE, Mind, LIX(236):433 460, 10 1950. ISSN 0026-4423.
Burak Varie,, Emre Acarturk, Karthikeyan Shanmugam, Abhishek Kumar, and Ali Tajer. Score-based causal representation learning with interventions. Xiv preprint arXiv:2301. 08230, 2023a.
70

---

Published in Transactions on Machine Learning Research (05/2024)
Burak Varia, Emre Acartürk, Karthikeyan Shanmugam, and Ali Tajer. General identifiability and achiev- ability for causal representation learning. arXiv preprint or arXiv:2300. 05150, 2023b.
Julius von Kügelgen, Yash Sharma, Luigi Gresele, Wieland Brendel, Bernhard Schölkopf, Michel Besserve, and Francesco Locatello. Self-supervised learning with data augmentations provably isolates content from style. In Advances in Neural Information Processing Systems, volune 34, pp. 16451 16467, 2021.
Julius von Küggeggen, Michel Besserve, Wendong Liang, Luigi Grescle, Armin Kekić, Elias Barcinboim, David Blei, and Bernhard Schölkopf. Nonparametric identifiability of causal representations from unknown interventions. In Advances in Neural Information Processing Systems, 2023.
Matthew Vowels, Necati Cihan Cangoz, and Richard Bowden. D'ya like dags? a survey on structure learning and causal discovery. arXiv preprint Xiic:2003.0258, 2021.
Yixin Wang and Michael I. Jordan. Desiderata for representation learning: A causal perspective. arXiv preprint arXiv:2109. 08795, 2021.
Zihao Wang, Lin Gui, Jeffrey Negrea, and Victor Veitch. Concept algebra for (score-based) text-controlled generative models. In Advances in Neural Information Processing Systems, 2023.
Benjamin Weggenmann, Valentin Rublack, Michael Andrejczuk, Justus Mattern, and Florian Kerschbaum. Dp-vae: Human-readable text anonymization for online reviews with differentially private variational autoencoders. In Proceedings of the ACM Web Conference 2022, pp. 721-731, 2022.
Alexander B. Wiltschko, Matthew J. Johnson, Giuliano lurilli, Ralph E. Peterson, Jesse M. Katon, Stan L. Pashkovski, Victoria E. Abraira, Ryan P. Adams, and Sandeep Robert Datta. Mapping sub-second struc- ture in mouse behavior. Neuron, 88(6): 2015. ISSN 0896-6273.
Zhengxuan Wu, Atticus Geiger, Thomas Icard, Christopher Potts, and Noah Goodman, Interpretability at scale: Identifying causal mechanisms in alpaca. In Advances in Neural Information Processing Systems, 2023.
Quanhan Xi and Benjamin Bloem-Redde Reddy. Indeterminacy in generative models: Characterization and strong identifiability. In Proceedings of The 26th International Conference on Artificial Intelligence and Statistics, 2023.
Kevin Xia, Kai-Zhan Lee, Yoshua Bengio, and Elias Bareinboim. The causal-neural connection: Expressive- ness, learnability, and inference. In Advonces in Neural Information Processing Systems, 2021.
Liyang Xie, Kaixiang Lin, Shu Wang, Fei Wang, and Jiayu Zhou. Differentially private generative adversarial network. arXiv preprint arXiv: 002,0699, 2018
Danru Xu, Dingling Yao, Sebastien Lachapelle, Perouz Taslakian, Julius von Kügelgen, Francesco Locatello, and Sara Magliacane. A sparsity principle for partially observable causal representation learning. In Causal Representation Learning Workshop at NewrIPS 2023, 2023.
Depeng Xu, Shuhan Yuan, Lu Zhang, and Xintao Wu. Fairgan: Fairness-aware generative adversarial networks. In 2018 IEEE International Conference on Big Data (Big Data), pp. 570-575. IEEE, 2018.
Depeng Xu, Yongkai Wu, Shuhan Yuan, Lu Zhang, and Xintao Wu. Achieving causal fairness through generative adversarial networks. In Proceedings of the Twenty-Eights International Joint Conference Artificial Intelligence, 1JCAI-19, pp. 1452 1458, 7 2019.
Ling Yang, Zhilong Zhang, Yang Song, Shenda Hong, Runsheng Xu, Yue Zhao, Wentao Zhang, Bin Cui, and Ming-Hsuan Yang. Diffusion models: A comprehensive survey of methods and applications. ACM Computing Surveys, 56(4):1-39, 2023.
Mengyue Yang, Furui Liu, Zhitang Chen, Xinwei Shen, Jianye Hao, and Jun Wang. Causalvac: Discntangled representation learning via neural structural causal models. In IEEE Conference on Computer Vision and Pattern Recognition, 2021.
71

---

Published in Transactions on Machine Learning Research (05/2024)
Ruikang Yang, Jianfeng Ma, Yinbin Miao, and Xindi Ma. membership inference attacks. arXiv preprint NXi::2202005466 2022.
generative framework against
Dingling Yao, Danru Xu, Sébastien Lachapelle, Sara Magliacane, Perouz Taslakian, Georg Martius, Julius Kügelgen, and Francesco Locatello. Multi-view causal representation learning with partial observabil- ity. arxiv preprint orXiv:231100005 2023.
Weiran Yao, Guangyi Chen, and Kun Zhang, Temporally disentangled representation learning. In Advances in Neural Information Processing Systems, 2022a.
Weiran Yao, Yuewen Sun, Alex Ho, Changyin Sun, and Kun Zhang. Learning temporally causal latent processes from general temporal data. In International Conference on Learning Regresenlations, 2022b.
Yue Yu, Jie Chen, Tian Gao, and Mo Yu. DAG-GNN: DAG structure learning with graph neural networks. In Proceedings of the 36th International Conference on Machine Learning, Proceedings of Machine Learning Research, 2019.
Rich Zemel, Yu Wu, Kevin Swersky, Toni Pitassi, and Cynthia Dwork. Learning fair representations. In International conference on machine learning, pp. 325-333. PMLR, 2013.
Jiaqi Zhang, Louis Cammarata, Chandler Squires, Themistoklis P. Sapsis, and Caroline Uhler. Active learning for optimal intervention design in causal models. Nature Machine Intelligence, 2023a.
Jiaqi Zhang, Kristjan Greenewald, Chandler Squires, Akash Srivastava, Karthikeyan Shanmugam, and Car- oline Uhler. Identifiability guarantees for causal disentanglerent from soft interventions. In Advances in Neural Information Processing Systems, 2023b.
Kun Zhang, Shaoan Xie, Ignavier Ng, and Yujia Zheng. Causal representation learning from multiple distributions: A general setting. arXiv preprint ar orXiv:2402.0000 2024.
Xinyang Zhang, Shouling Ji, and Ting Wang. Differentially private releasing via deep generative model 2018.
(technical report). arXiv preprint
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Be- ichen Zhang, Junjie Zhang, Zican Dong, et al. survey of large language models. arXiv preprint arXiv:2303. 18223, 2023.
Xun Zheng, Bryon Aragam, Pradeep K Ravikumar, and Eric P Xing. Dags with no tears: Continuous optimization for structure learning. In Advances in Neural Information Processing Systems, 2018.
Yujia Zheng and Kun Zhang. Generalizing nonlinear ICA beyond structural sparsity. In Advances in Neural Information Processing Systems, 2023.
Yujia Zheng, Tgnavier Ng, and Kun Zhang. On the identifiability of nonlinear TCA: Sparsity and beyond. Tn Advances in Neural Information Processing Systems, 2022.
Guanglin Zhou, Shaoan Xie, Guangyuan Hao, Shiming Chen, Biwei Huang, Xiwei Xu, Chen Wang, Liming Zhu, Lina Yao, and Kun Zhang. Emerging synergies in causality and deep generative models: A survey. arXiv preprinl ar Xim: 12351, 2023.
Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexci A Efros. Unpaired taage-to-image translation us- ing cycle-consistent adversarial networks. In 2017 IEFF International Conference on Computer Vision (ICCV), 2017.
Roland S. Zimmermann, Yash Sharma, Steffen Schneider, Matthias Bethge, and Wieland Brendel. Con- trastive learning inverts the data generating process. In Proceedings of the 38th International Conference on Machine Learning, 2021.
Zhiqun Zuo, Mohammad Mahdi Khalili, and Xueru Zhang. Counterfactualyy fair representation. In Advances in Neural Information Processing Systems, 2023.
72


Reviews:
[{"review_id": "9hP1mIHfZV", "reviewer": "Reviewer_A2sk", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. The author effectively categorizes causal generative modeling, offering clarity on identifiable causal representation learning and controllable counterfactual generation.\n2. The paper demonstrates a strong understanding of causal representation learning by organizing methods and providing a comprehensive discussion on evaluation metrics, datasets, and results.\n\nWeaknesses:\n\n1. The authors did not list the existing datasets in full. Many of the referenced papers include some datasets. Including these datasets in the paper would make the survey more comprehensive.\n2. Causal generative modeling is also employed to address issues related to learning causal relationships among observed variables, why do the authors only focus on the causal representation learning and controllable counterfactual generation? Moreover, it would be better to provide the relationship between causal representation learning and controllable counterfactual generation.\n3. There are also some works on using generative models to address causal representation learning that have not been included in the paper, such as LEAP[1], TDRL[2], and so on.\n\n[1] Yao, Weiran, et al. \"Learning Temporally Causal Latent Processes from General Temporal Data.\" International Conference on Learning Representations. 2021.\n[2] Yao, Weiran, Guangyi Chen, and Kun Zhang. \"Temporally disentangled representation learning.\" Advances in Neural Information Processing Systems 35 (2022): 26492-26503.\n\n4. Although a symbol table is provided in the paper, there is some inconsistency in the use of symbols. For instance, in Figure 7, \"A\" is used to represent a variable, while in Equation 34, \"A\" is used to denote the causal adjacency matrix. This could lead to confusion in understanding.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper provides a thorough and insightful technical survey on causal generative modeling, specifically focusing on causal representation learning and controllable counterfactual generation methods. The contributions are as follows:\n1. The author categorizes causal generative modeling into identifiable causal representation learning and controllable counterfactual generation tasks. A detailed classification of causal representation learning methods based on Pearl's Causal Hierarchy is provided, along with a breakdown of controllable generation approaches by the type of generative model used.\n2. This paper discusses evaluation metrics, benchmark datasets, and results for causal generative models.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See the weaknesses above.", "broader_impact_concerns": "Not applicable.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "AlbzFWRcwm", "reviewer": "Reviewer_Q6jz", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- The previous works that the submission surveyed is comprehensive.\n- Compared to previous surveys, the submission also includes a good summary of identifiability theory and challenging open problems.\n\nWeaknesses\n- The summary and discussion of previous methods are relatively shallow. Mostly the problem setting is given and the method is described. However, there is no much in-depth discussion to compare the pros/cons, when certain methods should be preferred than others, etc.\n- The contents for datasets, applications and open problems are relatively small, compared to the other like fundamental theory, methodology and metrics.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The work presents a survey on causal representations and controllable counterfactual generation using generative models.\nIt includes a comprehensive review of fundamental theory, methodology and (evaluation) metrics, and has a brief discussion on datasets, applications and open problems.\nThere is no new knowledge claimed nor presented in this work.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I have the following requested changes:\n\nCritical\n1. The survey is very much based on Pearl’s SCMs and Pearl's causal hierarchy. While they are widely adapted in causality research, they are not the only tools for causal inference and ways to categorize related concepts. I think this should be reflected in the title somehow to set the right expectation for readers. Just to make it clear, I'm questioning this choice and not asking the author(s) to change the way the survey is done, I'm suggesting to reflect this in the title somehow.\n2. Citation format is wrong for the entire paper. Please use \\citet and \\citep properly, e.g. `\\citet{} proposes XXX` and `such as XXX \\citep{}`.\n3. While not being actively maintained, the [`disentanglement_lib`](https://github.com/google-research/disentanglement_lib) should at least be mentioned in the \"Open Source Software\" as it has been used in many follow-up work.\n\nNon-critical but would be good to do to improve the draft\n1. Having an in-depth comparison between methods is highly encouraged. For example, the current table 3 is not that useful, especially the \"Key Features\" simply gives some keywords. Perhaps the authors can find a few important properties to form a set of columns and have each row tick some of the properties the corresponding method satisfies.\n2. The diagrams seems to be distorted in the PDF at certain zoom levels. They can be improved by using TIkZ or anything software produces vector graphics.\n3. Another useful table to have is one where the rows are methods and columns are datasets (or types of datasets). We can use this table to provide a view of which datasets have already be studied on which methods. This might be a good starting point to catch up the missing common benchmark datasets issue.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "qQR3YpmfYe", "reviewer": "Reviewer_pB3s", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n- Overall this is an informative, thorough, and well-written review.\n- The technical preamble is succinct yet informative, and informs the reader regarding the nuances of important concepts such as identifiability and disentanglement.\n- The reviewed methods are presented accessibly and within a logical categorization, the reader is usually directed to a (recreated) graphical model representing the modeling assumptions of the method in question, and the interrelations between methods are highlighted frequently. The tabulation of the presented work in Tables 2 and 3 is additionally helpful.\n- The inclusion of commonly used datasets, important applications, and open questions increase the value of the paper as a reference for the topics in question.\n- The paper is very well and carefully written, and is very easy to follow overall.\n\n### Weaknesses\n- As appropriate as the inclusion of an extensive introductory discussion of the issues of identifiability and disentanglement is, I think some parts of this discussion, as well as how they transfer to the context of CRL are underdeveloped, given the level of detail present throughout the paper. See below for details.\n- Although both CRL and CCG methods are diligently categorized within each other, the paper could include an additional (sub)section that compares these class of methods to their alternatives more broadly, both positively and negatively. See below for details.\n- Why the authors focus on a subset of research is not always sufficiently discussed, and the reader is not always directed to other surveys (if exists), for research that falls out of scope of the paper. This is the case in the authors limiting their survey to CRL and CCG in Section 1 and limiting CRL to VAE-based models in Section 3.1.4. The authors also declare non-medical applications out-of-scope without any references for the interested reader.\n- The metrics section lack a consistent evaluation or comparison of metrics with each other, as well as how frequently they are used in the literature or in the papers reviewed.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper provides a thorough review of recent generative models that either focus on causal representation learning (CRL) or controllable counterfactual generation (CCG). The review is preceded by an extended introduction that 1- defines central notions regarding statistical causality, 2- familiarizes the reader with various architectures in generative modeling, and 3- discusses the notions of identifiability and disentanglement, and how these relate to causal representation learning. The reviewed methods are examined and tabulated with respect to the architectures they rely on, main assumptions on the actual data generating distribution as well as observed variables, and the identifiability results provided in the case of CRL. The paper finishes off with a summary of commonly used datasets, important applications of the reviewed methods, and open research questions.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Here, in support of the weaknesses subsection above, I present expanded comments, as well as mentioning some minor points. I consider it important, yet not critical, for the issues raised above and here to be addressed for the final paper:\n-  Identifiability and Disentanglement w.r.t. Causality\n\t- In Section 2.6.2, how these methods lead to identifiability can be discussed at least in the form of an example. This does not need to be exhaustive for all three methods presented therein, but still an example or high-level description should be present for completeness.\n\t- Section 3.1.4 can be expanded to more thoroughly describe the increase in complexity of the problem of identifiability as we transition to causal representation learning, with the assumption of independent factors representing a trivial SCM, and what kind of additional information and/or assumptions this might require.\n- Comparison of CRL and CCG with alternatives:\n\t- In CRL, this would take the form of comparing it with representation learning methods in general, wherein the advantages could include the authors describing in what ways CRL-produced representations are likely to be more robust to distribution shifts, and the disadvantages could include the additional requirements or assumptions on the data at hand (e.g. observed auxiliary labels, domain annotation, fixed SCM class), as well as increased computational costs whenever applicable. Although these dis/advantages are presented throughout the paper within specific methods, a more high level comparison and commentary would be beneficial for the reader.\n- Minor points:\n\t- The Example 1 on Pg. 10 is too abriged to be easily understandable.\n\t- Definition 4: \"domain of parameters\": Please be explicit that members of $\\Theta$ parametrize the mixing functions and densities.\n\t- \"Reduced form SCM\" is not defined.\n\t- Given the level of detail present in the paper, at least an informal definition of \"distribution shift\" should be provided.\n\t- Section 3.1.3: The beginning of this subsection is repetitive.\n\t- Definition 1 is referenced in Section 3.1.3, which does not exist. This must be a mix-up regarding the existence of Principles 1 and 2, as Definitions start with Definition 3.\n\t- The following sentence is unclear and would benefit from expansion: \"However, such an approach does not isolate the learned factors and is effectively semantically meaningless; hence, the impossibility implication for arbitrary unsupervised disentanglement (Locatello et al. (2019b)).\"\n\t- The following sentence is unclear: \"That is, the multi-domain setup \"completes the picture\" of the causal model in some sense since relations can arise from different domains, as shown in Figure 11.\"\n\t- Please be explicit about the notion of un/paired data in places where this is used as qualifier. \n\t- MorphMNIST is referred to without any context before its introduction in datasets section.\n\t- The same model is mentioned under two different names $\\beta$-VAE and BetaVAE", "broader_impact_concerns": "I believe the contents of this paper does not require a Broader Impact Statement.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a technical survey on causal generative modeling, focusing on causal representation learning and controllable counterfactual generation methods. Specifically, the authors categorize causal representation learning methods according to the type of data, and controllable counterfactual generation methods according to the type of used generative model, respectively. Then, they discuss the common metrics, datasets, applications, and future research directions.

All reviewers acknowledged this survey is informative, comprehensive, and well-written. Also, the authors successfully addressed reviewers’ comments during the revision, and all of the reviewers agreed to the acceptance. Therefore, AE recommends a clear acceptance of this paper.
Decision: Accept (poster)