Year: 2020

Abstract:
This paper proposes a Disentangled gEnerative cAusal Representation (DEAR) learning method. Unlike existing disentanglement methods that enforce independence of the latent variables, we consider the general case where the underlying factors of interests can be causally correlated. We show that previous methods with independent priors fail to disentangle causally correlated factors. Motivated by this finding, we propose a new disentangled learning method called DEAR that enables causal controllable generation and causal representation learning. The key ingredient of this new formulation is to use a structural causal model (SCM) as the prior for a bidirectional generative model. The prior is then trained jointly with a generator and an encoder using a suitable GAN loss incorporated with supervision. Theoretical justification on the proposed formulation is provided, which guarantees disentangled causal representation learning under appropriate conditions. We conduct extensive experiments on both synthesized and real datasets to demonstrate the effectiveness of DEAR in causal controllable generation, and the benefits of the learned representations for downstream tasks in terms of sample efficiency and distributional robustness.

1 INTRODUCTION
Consider the observed data x from a distribution qx on X ⊆ Rd and the latent variable z from a prior pz on Z ⊆ Rk. In bidirectional generative models (BGMs), we are normally interested in learning an encoder E : X → Z to infer latent variables and a generator G : Z → X to generate data, to achieve both representation learning and data generation. Classical BGMs include Variational Autoencoder (VAE) (Kingma & Welling, 2014) and BiGAN (Donahue et al., 2017). In representation learning, it was argued that an effective representation for downstream learning tasks should disentangle the underlying factors of variation (Bengio et al., 2013). In generation, it is highly desirable if one can control the semantic generative factors by aligning them with the latent variables such as in StyleGAN (Karras et al., 2019). Both goals can be achieved with the disentanglement of latent variable z, which informally means that each dimension of z measures a distinct factor of variation in the data (Bengio et al., 2013).
Earlier unsupervised disentanglement methods mostly regularize the VAE objective to encourage independence of learned representations (Higgins et al., 2017; Burgess et al., 2017; Kim & Mnih, 2018; Chen et al., 2018; Kumar et al., 2018). Later, Locatello et al. (2019) show that unsupervised learning of disentangled representations is impossible: many existing unsupervised methods are actually brittle, requiring careful supervised hyperparameter tuning or implicit inductive biases. To promote identifiability, recent work resorts to various forms of supervision (Locatello et al., 2020b; Shu et al., 2020; Locatello et al., 2020a). In this work, we also incorporate supervision on the ground-truth factors in the form stated in Section 3.2.
Most of these existing methods are built on the assumption that the underlying factors of variation are mutually independent. However, in many real world cases the semantically meaningful factors of interests are not independent (Bengio et al., 2020). Instead, semantically meaningful high-level variables are often causally correlated, i.e., connected by a causal graph. In this paper, we prove formally that methods with independent priors fail to disentangle causally correlated factors. Motivated by this observation, we propose a new method to learn disentangled generative causal representations called DEAR. The key ingredient of our formulation is a structured causal model (SCM) (Pearl et al.,
2000) as the prior for latent variables in a bidirectional generative model. With some background knowledge on the binary causal structure, the causal prior is then learned jointly with a generator and an encoder using a suitable GAN (Goodfellow et al., 2014) loss. We establish theoretical guarantees for DEAR to learn disentangled causal representations under appropriate conditions.
An immediate application of DEAR is causal controllable generation, which can generate data from any desired interventional distributions of the latent factors. Another useful application of disentangled representations is to use such representations in downstream tasks, leading to better sample complexity (Bengio et al., 2013; Schölkopf et al., 2012). Moreover, it is believed that causal disentanglement is invariant and thus robust under distribution shifts (Schölkopf, 2019; Arjovsky et al., 2019). In this paper, we demonstrate these conjectures in various downstream prediction tasks for the proposed DEAR method, which has theoretically guaranteed disentanglement property.
We summarize our main contributions as follows: • We formally identify a problem with previous disentangled representation learning methods using
the independent prior assumption, and prove that they fail to disentangle when the underlying factors of interests are causally correlated. • We propose a new disentangled learning method, DEAR, which integrates an SCM prior into a bidirectional generative model, trained with a suitable GAN loss. • We provide theoretical justification on the identifiability of the proposed formulation. • Extensive experiments are conducted on both synthesized and real data to demonstrate the effec-
tiveness of DEAR in causal controllable generation, and the benefits of the learned representations for downstream tasks in terms of sample efficiency and distributional robustness.
2 OTHER RELATED WORK
GAN-based disentanglement methods. Existing methods, including InfoGAN (Chen et al., 2016) and InfoGAN-CR (Lin et al., 2020), differ from our proposed formulation mainly in two folds. First they still assume an independent prior for latent variables, so suffer from the same problem with previous VAE-based methods mentioned above. Besides, the idea of InfoGAN-CR is to encourage each latent code to make changes that are easy to detect, which actually applies well only when the underlying factors are independent. Second, InfoGAN as a bidirectional generative modeling method further requires variational approximation apart from adversarial training, which is inferior to the principled formulation in BiGAN and AGES (Shen et al., 2020) that we adopt.
Causality with generative models. CausalGAN (Kocaoglu et al., 2018) and a concurrent work (Moraffah et al., 2020) of ours, are unidirectional generative models (i.e., a generative model that learns a single mapping from the latent variable to data) that build upon a cGAN (Mirza & Osindero, 2014). They assign an SCM to the conditional attributes while leave the latent variables as independent Gaussian noises. The limit of a cGAN is that it always requires full supervision on attributes to apply conditional adversarial training. And the ground-truth factors are directly fed into the generator as the conditional attributes, without an extra effort to align the dimensions between the latent variables and the underlying factors, so their models have nothing to do with disentanglement learning. Moreover their unidirectional nature makes it impossible to learn representations. Besides they only consider binary factors whose consequent semantic interpolations appear nonsmooth, as shown in Appendix D. CausalVAE (Yang et al., 2020) assigns the SCM directly on the latent variables, but built upon iVAE (Khemakhem et al., 2020), it adopts a conditional prior given the ground-truth factors so is also limited to fully supervised setting.
3 PROBLEM SETTING
3.1 GENERATIVE MODEL
We first describe the probabilistic framework of disentangled learning with supervision. We follow the commonly assumed two-step data generating process that first samples the underlying generative factors, and then conditional on those factors, generates the data (Kingma & Welling, 2014). During the generation process, the generator induces the generated conditional pG(x|z) and generated joint distribution pG(x, z) = pz(z)pG(x|z). During the inference process, the encoder induces the encoded conditional qE(z|x) which can be a factorized Gaussian and the encoded joint distribution
qE(x, z) = qx(x)qE(z|x). We consider the following objective for generative modeling: Lgen = DKL(qE(x, z), pG(x, z)), (1)
which is shown to be equivalent to the evidence lower bound used in VAEs up to a constant, and allows a closed form only with factorized Gaussian prior, encoder and generator (Shen et al., 2020).
Since constraints on the latent space are required to enforce disentanglement, it is desired that the distribution family of qE(x, z) and pG(x, z) should be large enough, especially for complex data like images. Normally more general implicit distributions are favored over factorized Gaussians in terms of expressiveness (Karras et al., 2019; Mescheder et al., 2017). Then minimizing (1) requires adversarial training, as discussed detailedly in Section 4.3.
3.2 SUPERVISED REGULARIZER
To guarantee disentanglement, we incorporate supervision when training the BGM, following the similar idea in Locatello et al. (2020b) but with a different formulation. Specifically, let ξ ∈ Rm be the underlying ground-truth factors of interests of x, following distribution pξ, and yi be some continuous or discrete observation of the underlying factor ξi satisfying ξi = E(yi|x) for i = 1, . . . ,m. For example, in the case of human face images, y1 can be the binary label indicating whether a person is young or not, and ξ1 = E(y1|x) = P(y1 = 1|x) is the probability of being young given one image x.
Let Ē(x) be the deterministic part of the stochastic transformation E(x), i.e., Ē(x) = E(E(x)|x), which is used for representation learning. We consider the following objective:
L(E,G) = Lgen(E,G) + λLsup(E), (2)
where Lsup = ∑m
i=1 Ex,y[CE(Ēi(x), yi)] if yi is the binary or bounded continuous label of the i-th factor ξi, where CE(l, y) = y log σ(l) + (1 − y) log(1 − σ(l)) is the cross-entropy loss with σ(·) being the sigmoid function; Lsup = ∑m i=1 Ex,y[Ēi(x) − yi]2 if yi is the continuous observation of ξi, and λ > 0 is the coefficient to balance both terms. We empirically find the choice of λ quite insensitive to different tasks and datasets, and hence set λ = 5 in all experiments. Estimating of Lgen requires the unlabelled dataset {x1, . . . , xN} while estimating Lsup requires a labeled dataset {(xj , yj) : j = 1, . . . , Ns} where Ns can be much smaller than N . In contrast, Locatello et al. (2020b) propose the regularizer Lsup = ∑m i=1 Ex,z[CE(Ēi(x), zi)] involving only the latent variable z which is a part of the generative model, without distinguishing from the ground-truth factor ξ and its observation y. Hence they do not establish any theoretical justification on disentanglement. Besides, they adopt a VAE loss for Lgen with an independent prior, which suffers from the unidentifiability problem described in the next section.
3.3 UNIDENTIFIABILITY WITH AN INDEPENDENT PRIOR
Intuitively, the above supervised regularizer aims at ensuring some alignment between factor ξ and latent variable z. We start with the definition of a disentangled representation following this intuition.
Definition 1 (Disentangled representation). Given the underlying factor ξ ∈ Rm of data x, a deterministic encoder E is said to learn a disentangled representation with respect to ξ if ∀i = 1, . . . ,m, there exists a 1-1 function gi such that Ei(x) = gi(ξi). Further, a stochastic encoder E is said to be disentangled wrt ξ if its deterministic part Ē(x) is disentangled wrt ξ.
As stated above, we consider the general case where the underlying factors of interests are causally correlated. Then the goal becomes to disentangle the causal factors. Previous methods mostly use an independent prior for z, which contradicts with the truth. We make this formal through the following proposition, which indicates that the disentangled representation is generally unidentifiable with an independent prior.
Proposition 1. Let E∗ be any encoder that is disentangled wrt ξ. Let b∗ = Lsup(E∗), a = minG Lgen(E
∗, G), and b = min{(E,G):Lgen=0} Lsup(E). Assume the elements of ξ are connected by a causal graph whose adjacency matrix A0 is not a zero matrix. Suppose the prior pz is factorized, i.e., pz(z) = ∏k i=1 pi(zi). Then we have a > 0, and either when b
∗ ≥ b or b∗ < b and λ < ab−b∗ , there exists a solution (E
′, G′) such that for any generator G, we have L(E′, G′) < L(E∗, G).
All proofs are given in Appendix A. This proposition directly suggests that minimizing (2) favors the solution (E′, G′) over one with a disentangled encoder E∗. Thus, with an independent prior we have no way to identify the disentangled solution with λ that is not large enough. However, in real applications it is impossible to estimate the threshold, and too large λ makes it difficult to learn the BGM. In the following section we propose a solution to this problem.
4 CAUSAL DISENTANGLEMENT LEARNING
4.1 GENERATIVE MODEL WITH A CAUSAL PRIOR
We propose to use a causal model as the prior pz . Specifically we use the generalized nonlinear Structural Causal Model (SCM) proposed by Yu et al. (2019) as follows
z = f((I −A⊤)−1h($)) := Fβ($), (3) where A is the weighted adjacency matrix of the directed acyclic graph (DAG) upon the k elements of z (i.e., Aij ∕= 0 if and only if zi is the parent of zj), $ denotes the exogenous variables following N (0, I), f and h are element-wise nonlinear transformations, and β = (f, h,A) denotes the set of parameters of f , h and A, with the parameter space B. Further let 1A = I(A ∕= 0) denote the corresponding binary adjacency matrix, where I is the element-wise indicator function.
When f is invertible, (3) is equivalent to f−1(z) = A⊤f−1(z) + h($) (4)
which indicates that the factors z satisfy a linear SCM after nonlinear transformation f , and enables interventions on latent variables as discussed later. The model structure is presented in Figure 1. Note that different from our model where z is the latent variable following the prior (3) with the goal of causal disentanglement, Yu et al. (2019) proposed a causal discovery method where variables z are observed with the aim of learning the causal structure among z.
In causal structure learning, the graph is required to be acyclic. Zheng et al. (2018) propose an equality constraint whose satisfaction ensures acyclicity and solve the problem with augmented Lagrangian method, which however leads to optimization difficulties (Ng et al., 2020). In this paper, to avoid dealing with the non-convex constraint but focus on disentangling, we assume to have some prior knowledge of the binary causal structure. Specifically, we assume the super-graph of the true binary graph 1A∗ is given, the best case of which is the true graph while the worst is that only the causal ordering is available. Then we learn the weights of the non-zero elements of the prior adjacency matrix that indicate the sign and scale of causal effects, jointly with other parameters using the formulation and algorithm described in later sections. To incorporate structure learning methods and jointly learn the structure from scratch with guarantee of identifiability could be explored in future work. An ablation study is done in Appendix B regarding this prior knowledge.
To enable causal controllable generation, we use invertible f and h and describe the mechanism to generate images from interventional distributions of latent variables. Note that interventions can be formalized as operations that modify a subset of equations in (4) (Pearl et al., 2000). Suppose we would like to intervene on the i-th dimension of z, i.e., Do(zi = c), where c is a constant. Once we have latent factors z inferred from data x, i.e., z = E(x), or sampled from prior pz , we follow the intervened equations in (4) to obtain z′ on the left hand side using ancestral sampling by performing (4) iteratively. Then we decode the intervened latent factor z′ to generate the sample G(z′). In Section 5.1 we define the two types of interventions of most interests in applications.
Another issue of the model is the latent dimension, to handle which we propose the so-called composite prior. Recall that m is the number of generative factors that we are interested to disentangle,
e.g., all the semantic concepts related to some field, where m tends to be smaller than the total number M of generative factors. The latent dimension k of the generative model should be no less than M to allow a sufficient degree of freedom in order to generate or reconstruct data well. Since M is generally unknown in reality, we set a sufficiently large k, at least larger than m which is a trivial lower bound of M . The role of the remaining k−m dimensions is to capture other factors necessary for generation whose structure is not cared or explicitly modeled. Then we propose to use a prior that is a composition of a causal model for the first m dimensions and another distribution for the other k −m dimensions to capture other factors necessary for generation, like a standard Gaussian.
4.2 FORMULATION AND IDENTIFIABILITY OF DISENTANGLEMENT
In this section, we present the formulation of DEAR and establish the theoretical justification of it. Compared with the BGM described in Section 3.1, here we have one more module to learn that is the SCM prior. Thus pG(x, z) becomes pG,F (x, z) = pF (z)pG(x|z) where pF (z) or pβ(z) denotes the marginal distribution of Fβ($) with $ ∼ N (0, I). We then rewrite the generative loss as follows
Lgen(E,G, F ) = DKL(qE(x, z), pG,F (x, z)). (5)
Then we propose the following formulation to learn causal generative causal representations:
min E,G,F L(E,G, F ) := Lgen(E,G, F ) + λLsup(E). (6)
In order to achieve causal disentanglement, we make two assumptions on the causal model. Assumption 1 supposes a sufficiently large capacity of the SCM in (3) to contain the underlying distribution pξ, which is reasonable due to the generalization of the nonlinear SCM. Assumption 2 states the identifiability of the true causal structure 1A0 of ξ, which is applicable given the true causal ordering under basic Markov and causal minimality conditions (Pearl, 2014; Zhang & Spirtes, 2011). Assumption 1 (SCM capacity). The underlying distribution pξ belongs to the distribution family {pβ : β ∈ B}, i.e., there exits β0 = (f0, h0, A0) such that pξ = pβ0 . Assumption 2 (Structure identifiability). For all β = (f, h,A) ∈ B with pβ = pβ0 , it holds that 1A = 1A0 .
The following theorem then guarantees that under appropriate conditions the DEAR formulation can learn the disentangled representations defined in Definition 1. Theorem 1. Assume the infinite capacity of E and G. Further under Assumption 1-2, DEAR formulation (6) learns the disentangled encoder E∗. Specifically, we have gi(ξi) = σ−1(ξi) if CE loss is used for the supervised regularizer, and gi(ξi) = ξi if L2 loss is used.
Note that the identifiability we establish in this paper differs from some previous work on the parameter identifiability, e.g., Khemakhem et al. (2020). We argue that to learn disentangled representations, the form in Definition 1, i.e., the existence but not the uniqueness of gi’s, is sufficient to identify the relation among the representations and the data. In contrast, parameter identifiability may not be achievable in many cases like over-parametrization. Thus the identifiability discussed here is more realistic in terms of the goal of disentangling. Later we provide empirical evidence to support the theory directly through the application in causal controllable generation.
4.3 ALGORITHM
In this section we propose the algorithm to solve the formulation (6). The SCM prior pF (z) and implicit generated conditional pG(x|z) make (5) lose an analytic form. Hence we adopt a GAN method to adversarially estimate the gradient of (5). We parametrize Eφ(x) and Gθ(z) by neural networks. Different from Shen et al. (2020), the prior also involves learnable parameters. We present in the following lemma the gradient formulas of (5). Lemma 1. Let r(x, z) = q(x, z)/p(x, z) and D(x, z) = log r(x, z). Then we have
∇θLgen = −Ez∼pβ(z)[s(x, z)∇xD(x, z) ⊤|x=Gθ(z)∇θGθ(z)], ∇φLgen = Ex∼qx [∇zD(x, z) ⊤|z=Eφ(x)∇φEφ(x)],
∇βLgen = −E$[s(x, z)(∇xD(x, z)⊤∇βG(Fβ(!)) +∇zD(x, z)⊤∇βFβ(!))| x=G(Fβ($))
z=Fβ($) ],
(7)
where s(x, z) = eD(x,z) is the scaling factor.
We then estimate the gradients in (7) by training a discriminator Dψ via empirical logistic regression: minψ′ [
1 |Se| ∑ (x,z)∈Se log(1 + e −Dψ′ (x,z)) + 1|Sg| ∑ (x,z)∈Sg log(1 + e Dψ′ (x,z))], where Se and Sg
are finite samples from qE(x, z) and pG(x, z) respectively, leading to a GAN approach.
Based on above, we propose Algorithm 1 to learn disentangled generative causal representation. Algorithm 1: Disentangled gEnerative cAusal Representation (DEAR) Learning Input: training set {x1, . . . , xN , y1, . . . , yNs}, initial parameters φ, θ,β,ψ, batch-size n
1 while not convergence do 2 for multiple steps do 3 Sample {x1, . . . , xn} from the training set, {!1, . . . , !n} from N (0, I) 4 Generate from the causal prior zi = Fβ(!i), i = 1, . . . n 5 Update ψ by descending the stochastic gradient:
1 n ∑n i=1 ∇ψ [ log(1 + e−Dψ(xi,Eφ(xi))) + log(1 + eDψ(Gθ(zi),zi)) ]
6 Sample {x1, . . . , xn, y1, . . . , yns}, {!1, . . . , !n} as above; generate zi = Fβ(!i) 7 Compute θ-gradient: − 1
n ∑n i=1 s(Gθ(zi), zi)∇θDψ(Gθ(zi), zi)
8 Compute φ-gradient: 1 n ∑n i=1 ∇φDψ(xi, Eφ(xi)) + 1 ns ∑ns i=1 ∇φLsup(φ;xi, yi) 9 Compute β-gradient: − 1 n ∑n i=1 s(G(zi), zi)∇βDψ(Gθ(Fβ(!i)), Fβ(!i))
10 Update parameters φ, θ,β using the gradients Return: φ, θ,β
Remark: without loss of generality, assume the first Ns samples in the training set and the first ns samples in each mini-batch has available labels; ns may vary across different iterations.
5 EXPERIMENTS
We evaluate our methods on two datasets. The first one is a synthesized dataset Pendulum similar to the one in Yang et al. (2020). As shown in Figure 3, each image is generated by four continuous factors: pendulum angle, light angle, shadow length and shadow position whose underlying structure is given in Figure 2(a) following physical mechanisms. To make the dataset realistic, we introduce random noises when generating the two effects from the causes, representing the measurement error. We further introduce 20% corrupted data whose shadow is randomly generated, mimicking some environmental disturbance. The sample sizes for training, validation and test set are all 6,724.1
The second one is a real human face dataset CelebA (Liu et al., 2015), containing 202,599 images with 40 labelled binary attributes. Among them we consider two groups of causally correlated factors shown in 2(b,c). We believe these two datasets are diverse enough to assess our methods. All the details of experimental setup and architectures are given in Appendix C.
pendulum_ ngle(1) light_angle(2)
shadow_length(3) shadow_position(4)
pendulum_ angle(0)
light_ angle(1)
shadow_ position(3) shadow_ length(2)
(a) Pendulum
smile(1)
checkbone(3) narrow_eye(5)mouth_open(4)
gender(2)
smile6
chubby(6)
smile(0) gender(1)
narrow_ eye(4)chubby(5) mouth_ open(3)
cheekbone(2)
(b) CelebA-Smile
Age6 young(1) gend r(2)
receding_hairline(3)make_up(4)chubby(5)
eye_bag(6)
young(0) gender(1)
receding_ hairline(2) make_ up(3)chubby(4)
eye_bag(2)
(c) CelebA-Attractive
Figure 2: Underlying causal structures.
5.1 CONTROLLABLE GENERATION
We first investigate the performance of our methods in disentanglement through applications in causal controllable generation (CG). Traditional CG methods mainly manipulate the independent generative factors (Karras et al., 2019), while we consider the general case where the factors are causally correlated. With a learned SCM as the prior, we are able to generate images from any desired interventional distributions of the latent factors. For example, we can manipulate only the
1The Pendulum dataset will be released as a causal disentanglement benchmark soon.
cause factor while leave its effects unchanged. Besides, the BGM framework enables controllable generation either from scratch or a given unlabeled image.
We consider two types of intervention. In traditional traversals, we manipulate one dimension of the latent vector while keep the others fixed to either their inferred or sampled values (Higgins et al., 2017). A causal view of such operations is an intervention on all the variables by setting them as constants with only one of them varying. Another interesting type of interventional distribution is to intervene on only one latent variable, i.e., Pdo(zi=c)(z). The proposed SCM prior enables us to conduct such intervention though the mechanism given in Section 4.1.
Figure 3-4 illustrate the results of causal controllable generation of the proposed DEAR and the baseline method with an independent prior, S-β-VAE (Locatello et al., 2020b). Results from other baselines including S-TCVAE, S-FactorVAE (which essentially make no difference due to the independence assumption) and CausalGAN are given in Appendix D. Note that we do not compare with unsupervised disentanglement methods because of fairness and their lack of justification.
In each figure, we first infer the latent representations from a test image in block (c). The traditional traversals of two models are given in blocks (a,b). We see that in each line when manipulating one latent dimension, the generated images from our model vary only in a single factor, indicating that our method can disentangle the causally correlated factors. It is worth pointing out that we are the first to achieve the disentanglement between the cause and its effects, while other methods tend to entangle them. In block (d), we show the results of intervention on the latent variables representing the cause factors, which clearly show that intervening on a cause variable changes its effect variables. Results in Appendix D further show that intervening on an effect node does not influence its cause.
Since the underlying factors are causally correlated, all previous quantitative metrics for disentanglement no longer apply. We provide more qualitative traversals in Appendix D to show the overall performance. A quantitative metric for causal disentanglement is worth exploring in future work.
5.2 DOWNSTREAM TASK
The previous section verifies the good disentanglement performance of DEAR. In this section, equipped with DEAR, we investigate and demonstrate the benefits of learned disentangled causal representations in sample efficiency and distributional robustness.
We state the downstream tasks. On CelebA, we consider the structure CelebA-Attractive in Figure 2(c). We artificially create a target label τ = 1 if young=1, gender=0, receding hairline=0, make up=1, chubby=0, eye bag=0, and τ = 0 otherwise, indicating one kind of attractiveness as a slim young woman with makeup and thick hair.2 On the pendulum dataset, we regard the label of data corruption as the target τ , i.e., τ = 1 if the data is corrupted and τ = 0 otherwise. We consider the downstream tasks of predicting the target label. In both cases, the factors of interests in Figure 2(a,c) are causally related to τ , which are the features that humans use to do the task. Hence it is conjectured that a disentangled representation of these causal factors tends to be more data efficient and invariant to distribution shifts.
5.2.1 SAMPLE EFFICIENCY
For a BGM including the previous state-of-the-art supervised disentangling methods S-VAEs (Locatello et al., 2020b) and DEAR, we use the learned encoder to embed the training data to the latent space and train a MLP classifier on the representations to predict the target label. Without an encoder, one normally needs to train a convolutional neural network with raw images as the input. Here we adopt the ResNet50 as the baseline classifier which is the architecture of the BGM encoder. Since disentangling methods use additional supervision of the generative factors, we consider another baseline that is pretrained using multi-label prediction of the factors on the same training set.
To measure the sample efficiency, we use the statistical efficiency score defined as the average test accuracy based on 100 samples divided by the average accuracy based on 10,000/all samples, following Locatello et al. (2019). Table 1 presents the results, showing that DEAR owns the highest sample efficiency on both datasets. ResNet with raw data inputs has the lowest efficiency, although multi-label pretraining improves its performance to a limited extent. S-VAEs have better efficiency than the ResNet baselines but lower accuracy under the case with more training data, which we think is mainly because the independent prior conflicts with the supervised loss as indicated in Proposition 1, making the learned representations entangled (as shown in the previous section) and less informative. Besides, we also investigate the performance of DEAR under the semi-supervised setting where only 10% of the labels are available. We find that DEAR with fewer labels has comparable sample efficiency with that in the fully supervised setting, with a sacrifice in accuracy that is yet still comparable to other baselines with more supervision.
2Note that the definition of attractiveness here only refers to one kind of attractiveness, which has nothing to do with the linguistic definition of attractiveness.
(b) Pendulum
5.2.2 DISTRIBUTIONAL ROBUSTNESS
We manipulate the training data to inject spurious correlations between the target label and some spurious attributes. On CelebA, we regard mouth open as the spurious factor; on Pendulum, we choose background color ∈ {blue(+), white(−)}. We manipulate the training data such that the target label is more strongly correlated with the spurious attributes, i.e., the target label and the spurious attribute of 80% of the examples are both positive or negative, while those of 20% examples are opposite. For example, in the manipulated training set, 80% smiling examples in CelebA have an open mouth; 80% corrupted examples in Pendulum are masked with a blue background. The test set however does not have these correlations, leading to a distribution shift.
Intuitively these spurious attributes are not causally correlated to the target label, but normal independent and identically distributed (IID) based methods like empirical risk minimization (ERM) tend to exploit these easily learned spurious correlations in prediction, and hence face performance degradation when the such correlation no longer exists during test. In contrast, causal factors are regarded invariant and thus robust under such shifts. Previous sections justify both theoretically and empirically that DEAR can learn disentangled causal representations. We then apply those representations by training a classifier upon them, which is conjectured to be invariant and robust. Baseline methods include ERM, multi-label ERM to predict target label and all the factors considered in disentangling to have the same amount of supervision, and S-VAEs that can not disentangle well in the causal case.
Table 2 shows the average and worst-case (Sagawa et al., 2019) test accuracy to assess both the overall classification performance and distributional robustness, where we group the test set according to the two binary labels, the target one and the spurious attribute, into four cases and regard the one with the worst accuracy as the worst-case, which usually owns the opposite correlation to the training data. We see that the classifiers trained upon DEAR representations outperform the baselines in both metrics. Particularly, when comparing the worst-case accuracy with the average one, we observe a slump from around 80 to around 60 for other methods on CelebA, while DEAR enjoys an acceptable small decline. These results support the above conjecture and the benefits of causal disentanglement in distributional robustness.
6 CONCLUSION
This paper showed that previous methods with the independent latent prior assumption fail to learn disentangled representation when the underlying factors of interests are causally correlated. We then proposed a new disentangled learning method called DEAR with theoretical guarantees. Extensive experiments demonstrated the effectiveness of DEAR in causal generation, and the benefits of the learned representations for downstream tasks.
REFERENCES
Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprint arXiv:1907.02893, 2019.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 35(8):1798–1828, 2013.
Yoshua Bengio, Tristan Deleu, Nasim Rahaman, Rosemary Ke, Sébastien Lachapelle, Olexa Bilaniuk, Anirudh Goyal, and Christopher Pal. A meta-transfer objective for learning to disentangle causal mechanisms. In ICLR, 2020.
Christopher P. Burgess, Irina Higgins, Arka Pal, Loı̈c Matthey, Nicholas Watters, Guillaume Desjardins, and Alexander Lerchner. Understanding disentangling in beta-vae. NIPS Workshop of Learning Disentangled Features, 2017.
Tian Qi Chen, Xuechen Li, Roger B. Grosse, and David K. Duvenaud. Isolating sources of disentanglement in variational autoencoders. In NeurIPS, 2018.
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. In Advances in neural information processing systems, pp. 2172–2180, 2016.
Jeff Donahue, Philipp Krähenbühl, and Trevor Darrell. Adversarial feature learning. In ICLR, 2017.
Vincent Dumoulin, Ishmael Belghazi, Ben Poole, Alex Lamb, Martı́n Arjovsky, Olivier Mastropietro, and Aaron C. Courville. Adversarially learned inference. In ICLR, 2017.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural information processing systems, pp. 2672–2680, 2014.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770–778, 2016.
Irina Higgins, Loı̈c Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with a constrained variational framework. In ICLR, 2017.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4401–4410, 2019.
Ilyes Khemakhem, Diederik Kingma, Ricardo Monti, and Aapo Hyvarinen. Variational autoencoders and nonlinear ica: A unifying framework. In International Conference on Artificial Intelligence and Statistics, pp. 2207–2217, 2020.
Hyunjik Kim and Andriy Mnih. Disentangling by factorising. In ICML, 2018.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. In ICLR, 2014.
Murat Kocaoglu, Christopher Snyder, Alexandros G Dimakis, and Sriram Vishwanath. Causalgan: Learning causal implicit generative models with adversarial training. In ICLR, 2018.
Abhishek Kumar, Prasanna Sattigeri, and Avinash Balakrishnan. Variational inference of disentangled latent concepts from unlabeled observations. In ICLR, 2018.
Felix Leeb, Yashas Annadani, Stefan Bauer, and Bernhard Schölkopf. Structural autoencoders improve representations for generation and transfer. arXiv preprint arXiv:2006.07796, 2020.
Zinan Lin, Kiran K Thekumparampil, Giulia Fanti, and Sewoong Oh. Infogan-cr and modelcentrality: Self-supervised model training and selection for disentangling gans. In ICML, 2020.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of the IEEE international conference on computer vision, pp. 3730–3738, 2015.
F. Locatello, S. Bauer, M. Lucic, G. Raetsch, S. Gelly, B. Schölkopf, and O. Bachem. Challenging common assumptions in the unsupervised learning of disentangled representations. In Proceedings of the 36th International Conference on Machine Learning (ICML), volume 97 of Proceedings of Machine Learning Research, pp. 4114–4124. PMLR, June 2019. URL http://proceedings.mlr.press/v97/locatello19a.html.
Francesco Locatello, Ben Poole, Gunnar Rätsch, Bernhard Schölkopf, Olivier Bachem, and Michael Tschannen. Weakly-supervised disentanglement without compromises. In ICML, 2020a.
Francesco Locatello, Michael Tschannen, Stefan Bauer, Gunnar Rätsch, Bernhard Schölkopf, and Olivier Bachem. Disentangling factors of variation using few labels. In ICLR, 2020b.
Lars Mescheder, Sebastian Nowozin, and Andreas Geiger. Adversarial variational bayes: Unifying variational autoencoders and generative adversarial networks. In Proceedings of the 34th International Conference on Machine Learning-Volume 70, pp. 2391–2400. JMLR. org, 2017.
Mehdi Mirza and Simon Osindero. Conditional generative adversarial nets. arXiv preprint arXiv:1411.1784, 2014.
Raha Moraffah, Bahman Moraffah, Mansooreh Karami, Adrienne Raglin, and Huan Liu. Can: A causal adversarial network for learning observational and interventional distributions. arXiv preprint arXiv:2008.11376, 2020.
Ignavier Ng, AmirEmad Ghassami, and Kun Zhang. On the role of sparsity and dag constraints for learning linear dags. arXiv preprint arXiv:2006.10201, 2020.
Judea Pearl. Probabilistic reasoning in intelligent systems: networks of plausible inference. Elsevier, 2014.
Judea Pearl et al. Models, reasoning and inference. Cambridge, UK: CambridgeUniversityPress, 2000.
Ali Razavi, Aaron van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vae-2. In Advances in Neural Information Processing Systems, pp. 14866–14876, 2019.
Shiori Sagawa, Pang Wei Koh, Tatsunori B Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. arXiv preprint arXiv:1911.08731, 2019.
Bernhard Schölkopf. Causality for machine learning. arXiv preprint arXiv:1911.10500, 2019.
Bernhard Schölkopf, Dominik Janzing, Jonas Peters, Eleni Sgouritsa, Kun Zhang, and Joris Mooij. On causal and anticausal learning. In ICML, 2012.
Xinwei Shen, Tong Zhang, and Kani Chen. Bidirectional generative modeling using adversarial gradient estimation. arXiv preprint arXiv:2002.09161, 2020.
Rui Shu, Yining Chen, Abhishek Kumar, Stefano Ermon, and Ben Poole. Weakly supervised disentanglement with guarantees. In ICLR, 2020.
Mengyue Yang, Furui Liu, Zhitang Chen, Xinwei Shen, Jianye Hao, and Jun Wang. Causalvae: Structured causal disentanglement in variational autoencoder. arXiv preprint arXiv:2004.08697, 2020.
Yue Yu, Jie Chen, Tian Gao, and Mo Yu. Dag-gnn: Dag structure learning with graph neural networks. In ICML, 2019.
Han Zhang, Ian Goodfellow, Dimitris Metaxas, and Augustus Odena. Self-attention generative adversarial networks. In International Conference on Machine Learning, pp. 7354–7363. PMLR, 2019.
Jiji Zhang and Peter Spirtes. Intervention, determinism, and the causal minimality condition. Synthese, 182(3):335–347, 2011.
Shengjia Zhao, Jiaming Song, and Stefano Ermon. Learning hierarchical features from generative models. In ICML, 2017.
Xun Zheng, Bryon Aragam, Pradeep K Ravikumar, and Eric P Xing. Dags with no tears: Continuous optimization for structure learning. In Advances in Neural Information Processing Systems, pp. 9472–9483, 2018.
APPENDIX A PROOFS
A.1 PROOF OF PROPOSITION 1
Proof. On one hand, by the assumption that the elements of ξ are connected by a causal graph whose adjacency matrix is not a zero matrix. Then exist i ∕= j such that ξi and ξj are not independent, indicating that the probability density of ξ cannot be factorized. Since E∗ is disentangled wrt ξ, by Definition 1, ∀i = 1, . . . ,m there exists gi such that E∗i (x) = gi(ξi). This implies that the probability density of E∗(x) is not factorized.
On the other hand, notice that the distribution family of the latent prior is {pz : pz is factorized}. Hence the intersection of the marginal distribution families of z and E∗(x) is an empty set. Then the joint distribution families of (x,E∗(x)) and (G(z), z) also have an empty intersection.
We know that Lgen(E,G) = 0 implies qE(x, z) = pG(x, z) which contradicts the above. Therefore, we have a = minG Lgen(E∗, G) > 0.
Let (E′, G′) be the solution of the optimization problem min{(E,G):Lgen=0} Lsup(E). Then we have L′ = L(E′, G′) = λb, and L∗ = L(E∗, G) ≥ a + λb∗ > λb∗ for any generator G. When b∗ ≥ b we directly have L′ < L∗. When b∗ < b and λ is not large enough, i.e., λ < ab−b∗ , we have L′ < L∗.
A.2 PROOF OF THEOREM 1
Proof. Assume E is deterministic.
On one hand, for each i = 1, . . . ,m, first consider the cross-entropy loss Lsup,i(E) = E(x,y)[CE(Ei(x), yi)] = ∫ p(x)p(yi|x)(yi log σ(Ei(x))+(1−yi) log(1−σ(Ei(x))))dxdyi,
where p(yi|x) is the probability mass function of the binary label yi given x, characterized by P(yi = 1|x) = E(yi|x) and P(yi = 0|x) = 1− E(yi|x). Let
∂Lsup,i ∂σ(Ei(x)) =
∫ p(x)p(yi|x) ( yi
1 σ(Ei)(1− σ(Ei)) − 1 1− σ(Ei)
) dxdyi = 0.
Then we know that E∗i (x) = σ −1(E(yi|x)) = σ−1(ξi) minimizes Lsup,i.
Consider the L2 loss
Lsup,i(φ) = E(x,y)[Ēi(x)− yi]2 = ∫ p(x)p(yi|x)‖Ei(x)− yi‖2dxdyi.
Let ∂Lsup,i ∂Ei(x) = 2 ∫ p(x)p(yi|x)(Ei(x)− yi)dxdyi = 0.
Then we know that E∗i (x) = E(yi|x) = ξi minimizes Lsup,i in this case. On the other hand, by Assumption 1 there exists β0 = (f0, h0, A0) such that pξ = pβ0 . Then the distribution of E∗(x) is given by pβ∗ with β∗ = (g ◦ f0, h0, A0). Assumption 2 ensures that there is no β′ = (f ′, h′, A′) such that A′ ∕= A0 but pβ′ = pβ∗ . Let F ∗ = Fβ∗ . Further due to the infinite capacity of G, we have the distribution family of pG,F∗(x, z) contains qE∗(x, z). Then by minimizing the loss in (6) over G, we can find G∗ such that pG∗,F∗(x, z) matches qE∗(x, z) and thus Lgen(E∗, G∗, F ∗) reaches 0.
Hence minimizing L = Lgen +λLsup, which is the DEAR formulation (6), leads to the solution with E∗i (x) = gi(ξi) with gi(ξi) = σ
−1(ξi) if CE loss is used, and gi(ξi) = ξi if L2 loss is used, and the true binary adjacency matrix.
For a stochastic encoder, we establish the disentanglement of its deterministic part as above, and follow Definition 1 to obtain the desired result.
A.3 PROOF OF LEMMA 1
We follow the same proof scheme as in Shen et al. (2020) where the only difference lies in the gradient wrt the prior parameter β. To make this paper self-contained, we restate some proof steps here using our notations.
Let ‖ · ‖ denote the vector 2-norm. For a scalar function h(x, y), let ∇xh(x, y) denote its gradient with respect to x. For a vector function g(x, y), let ∇xg(x, y) denote its Jacobi matrix with respect to x. Given a differentiable vector function g(x) : Rk → Rk, we use ∇ · g(x) to denote its divergence, defined as
∇ · g(x) := k∑
j=1
∂[g(x)]j ∂[x]j ,
where [x]j denotes the j-th component of x. We know that ∫
∇ · g(x)dx = 0
for all vector function g(x) such that g(∞) = 0. Given a matrix function w(x) = (w1(x), . . . , wl(x)) : Rk → Rk×l where each wi(x), i = 1 . . . , l is a k-dimensional differentiable vector function, its divergence is defined as ∇ · w(x) = (∇ · w1(x), . . . ,∇ · wl(x)). To prove Lemma 1, we need the following lemma which specifies the dynamics of the generator joint distribution pg(x, z) and the encoder joint distribution pe(x, z), denoted by pθ(x, z) and pφ(x, z) here.
Lemma 2. Using the definitions and notations in Lemma 1, we have
∇θpθ,β(x, z) = −∇xpθ,β(x, z)⊤gθ(x)− pθ,β(x, z)∇ · gθ(x), (8) ∇φqφ(x, z) = −∇zqφ(x, z)⊤eφ(z)− qφ(x, z)∇ · eφ(z), (9) ∇βpθ,β(x, z) = ∇xpθ,β(x, z)⊤f̃β(x)−∇zpθ,β(x, z)⊤fβ(z)− pθ,β(x, z)∇ · ( f̃β(x)
fβ(z)
) , (10)
for all data x and latent variable z, where gθ(Gθ(z, $)) = ∇θGθ(z, $), eφ(Eφ(x, $)) = ∇φEφ(x, $), fβ(Fβ($)) = ∇βFβ($), and f̃β(G(Fβ($))) = ∇βG(Fβ($)).
Proof of Lemma 2. We only prove (10) which is the distinct part from Shen et al. (2020).
Let l be the dimension of parameter β. To simplify notation, let random vector Z = Fβ($) and X = G(Z) ∈ Rd and Y = (X,Z) ∈ Rd+k, and let p be the probability density of Y . For each i = 1, . . . , l, let ∆ = δei where ei is a l-dimensional unit vector whose i-th component is one and all the others are zero, and δ is a small scalar. Let Z ′ = Fβ+δ($), X ′ = G(Z ′) and Y ′ = (X ′, Z ′) so that Y ′ is a random variable transformed from Y by
Y ′ = Y +
( f̃β(X)
fβ(Z)
) ∆+ o(δ).
Let p′ be the probability density of Y ′. For an arbitrary y′ = (x′, z′) ∈ Rd+k, let y′ = y +(f̃β(x) fβ(z) ) ∆+ o(δ) and y = (x, z). Then we have
p′(y′) = p(y)| det(dy′/dy)|−1
= p(y)| det(Id + (∇f̃β(x),∇fβ(z))⊤∆+ o(δ))|−1
= p(y)(1 +∆⊤∇ · (f̃β(x), fβ(z))⊤ + o(δ))−1
= p(y)(1−∆⊤∇ · (f̃β(x), fβ(z))⊤ + o(δ)) = p(y)−∆⊤p(y′)∇ · (f̃β(x′), fβ(z′))⊤ + o(δ) = p(y′)−∆⊤(f̃β(x′), fβ(z′)) ·∇x′p(x′, z)−∆⊤p(y′)(∇ · f̃β(x′),∇ · fβ(z′))⊤ + o(δ).
Since y′ is arbitrary, above implies that
p′(x, z) = p(x, z)−∆⊤(f̃β(x), fβ(z)) · (∇xp(x, z),∇zp(x, z))⊤ ·∇xp(x, z) −∆⊤p(x, z)(∇ · f̃β(x′),∇ · fβ(z′))⊤ + o(δ)
for all x ∈ Rd, z ∈ Rk and i = 1, . . . , l, leading to (10) by taking δ → 0, and noting that p = pβ and p′ = pβ+∆. Similarly we can obtain (8) and (9).
Proof of Lemma 1. Recall the objective DKL(q, p) = ∫ q(x, z) log(p(x, z)/q(x, z))dxdz. Denote its integrand by ℓ(q, p). Let ℓ′2(q, p) = ∂ℓ(q, p)/∂p. We have
∇βℓ(q(x, z), p(x, z)) = ℓ′2(q(x, z), p(x, z))∇βpθ,β(x, z) where ∇βpθ,β(x, z) is computed in Lemma 2. Besides, we have
∇x · [ℓ′2(q, p)p(x, z)f̃β(x)] = ℓ′2(q, p)p(x, z)∇ · f̃β(x) + ℓ′2(q, p)∇xp(x, z) · f̃β(x) +∇xℓ′2(q, p)p(x, z)f̃β(x), ∇z · [ℓ′2(q, p)p(x, z)fβ(z)] = ℓ′2(q, p)p(x, z)∇ · fβ(z) + ℓ′2(q, p)∇p(x, z) · fβ(z) +∇ℓ′2(q, p)p(x, z)fβ(z).
Thus,
∇βLgen = ∫ ∇βℓ(q(x, z), p(x, z))dxdz = ∫ p(x, z)[∇xℓ′2(q, p)f̃β(x) +∇zℓ′2(q, p)fβ(z)]
where we can compute ∇xℓ′2(q, p) = s(x, z)∇xD(x, z) and ∇xℓ′2(q, p) = s(x, z)∇zD(x, z). Hence
∇βLgen = −E(x,z)∼p(x,z) [ s(x, z)(∇xD(x, z)⊤f̃β(x) +∇zD(x, z)⊤fβ(z)) ]
= −E' [ s(x, z)(∇xD(x, z)⊤∇βG(Fβ($)) +∇zD(x, z)⊤∇βFβ($))| x=G(Fβ('))
z=Fβ(')
] .
where the second equality follows reparametrization.
Lemma 3. For any a, b ∈ R (a < b), the set of continuous piece-wise linear function P is dense in C[a, b] where the metric d(f, g) = supx∈[a,b] |f(x)− g(x)|. Note that P is defined as
P = ∪h∈{(b−a)/n|n∈N+}Ph
Ph =
 
k + (b−a)/h−1∑
i=0
wi(x− a− ih)1(x ≥ a+ ih) ∣∣∣∣wi, k ∈ R
 
 ,
where [·] here is floor function.
Proof. Since [a, b] is compact, any function f ∈ C[a, b] is uniform continuous, i.e., ∀$ > 0, there exists δ > 0 such that |x− y| < δ =⇒ |f(x)− f(y)| < $/2. Let [a, b] = ∪N−1n=0 [an, bn], and gn(x) be a linear function, such that
an = a+ nh,
bn = a+ (n+ 1)h,
gn(an) = f(an),
gi(bn) = f(bn),
Nh = b− a.
Assume that h < δ. For any x ∈ [an, bn], we have |f(x)− gi(x)| ≤ min {|f(x)− f(an)|+ |gi(x)− gi(an)|, |f(x)− f(bn)|+ |gi(x)− gi(bn)|}
≤ |gi(an)− gi(bn)|+min {|f(x)− f(an)|, |f(x)− f(bn)|} ≤ |f(an)− f(bn)|+min {|f(x)− f(an)|, |f(x)− f(bn)|} < $.
Thus, sup
x∈[an,bn] |f(x)− gn(x)| < $.
We define
g(x) =
N−1∑
n=1
gn(x)1(x ∈ [an, bn])
which is obvious that g(x) ∈ Ph ⊂ P . And we have sup
x∈[a,b] |f(x)− g(x)| < $
Therefore, P is dense in C[a, b] and Ph is $-dense.
APPENDIX B LEARNING THE STRUCTURE
As mentioned in Section 4.1, our DEAR algorithm requires the prior knowledge on the super-graph of the true graph over the underlying factors of interests. The experiments shown in the main text are all based on the assumption that the true graph is given. In this section we investigate the performance of the learned weighted adjacency matrix and present an ablation study on different extents of prior knowledge on the structure.
B.1 GIVEN THE TRUE GRAPH
Figure 5 shows the learned weighted adjacency matrices when the true binary structure is given, whose weights show sensible signs and scalings consistent with common knowledge. For example, smile and its effect mouth open are positively correlated. The corresponding element in the weighted adjacency A03 of (a) turns out positive, which makes sense. Also gender (the logit of male) and its effect make up are negatively correlated. Then A13 of (b) turns out negative.
B.2 GIVEN THE TRUE CAUSAL ORDERING
Consider the Pendulum dataset, whose ground-truth structure is given in Figure 2(a). Consider a causal ordering pendulum angle, light angle, shadow position, shadow length, given which we start with a full graph whose elements are randomly initialized around 0 as shown in Figure 6(a). Figure 6 presents the adjacency matrices learned by DEAR at different training epochs, from which we see that it eventually obtains the learned structure that nearly coincides with the one learned given the true graph shown in Figure 5(c). This experiment shows the potential of DEAR to incorporate structure learning methods to learn the latent causal structure from scratch, which will be explored in future research.
APPENDIX C IMPLEMENTATION DETAILS
In this section we state the details of experimental setup and the network architectures used for all experiments.
Preprocessing and hyperparameters. We pre-process the images by taking a center crops of 128 × 128 for CelebA and resizing all images in CelebA and Pendulum to the 64 × 64 resolution. We adopt Adam with β1 = 0, β2 = 0.999, and a learning rate of 1× 10−4 for D, 5× 10−5 for E, G and F , and 1 × 10−3 for the adjacency matrix A. We use a mini-batch size of 128. For adversarial training in Algorithm 1, we train the D once on each mini-batch. The coefficient λ of the supervised regularizer is set to 5. We use CE supervised loss for both CelebA with binary observations of the underlying factors and Pendulum with bounded continuous observations. Note that L2 loss works comparable to CE loss on Pendulum. In downstream tasks, for BGMs with an encoder, we train a two-level MLP classifier with 100 hidden nodes using Adam with a learning rate of 1× 10−2 and a mini-batch size of 128. Models were trained for around 150 epochs on CelebA and 600 epochs on Pendulum on NVIDIA RTX 2080 Ti.
Network architectures. We follow the architectures used in Shen et al. (2020). Specifically, for such realistic data, we adopt the SAGAN (Zhang et al., 2019) architecture for D and G. The D network consists of three modules as shown in Figure 7 and detailed described in (Shen et al., 2020). Details for newtork G and Dx are given in Figure 7 and Table 3. The encoder architecture is the ResNet50 (He et al., 2016) followed by a 4-layer MLP of size 1024.
Implementation of the SCM. Recall the nonlinear SCM as the prior
Z = f((I −A⊤)−1h($)) := Fβ($). We find Gaussians are expressive enough as unexplained noises, so we set h as the identity mapping. As mentioned in Section 4.1 we require the invertibility of f . We implement both linear and nonlinear ones. For a linear f , we formally refer to
f(z) = Wz + b,
where W and b are learnable weights and biases. Note that W is a diagonal matrix to model the element-wise transformation. Its inverse function can be easily computed by
f−1(z) = W−1(z − b). For a non-linear f , we use piece-wise linear functions defined by
f (i)(z(i)) = w (i) 0 z (i) +
Na∑
t=1
w (i) t (z (i) − ai)1(z(i) ≥ ai) + b(i)
where ·(i) denote the i-th dimension of a vector or a vector-function, a0 < a1 < · · · < aNa are points of division, and 1(·) is the indicator function. From its denseness shown in lemma 3, the family of such piece-wise linear functions is expressive enough to model general element-wise nonlinear invertible transformations.
Experimental details for baseline methods. We reproduce the S-VAEs including S-VAE, S-βVAE and S-TCVAE using E and G with the same architecture as ours and adopt the same optimization algorithm for training. The coefficient for the independence regularizer is set to 4 since we
notice that setting a larger independence regularizer hurts disentanglement in the correlated case. For the supervised regularizer, we use λ = 1000 for a balance of generative model and supervision. The ERM ResNet is trained using the same optimizer with a learning rate of 1× 10−4.
APPENDIX D ADDITIONAL RESULTS OF CAUSAL CONTROLLABLE GENERATION
In this section we present more qualitative results of causal controllable generation on two datasets using DEAR and baseline methods, including S-VAEs (Locatello et al., 2020b) and CausalGAN (Kocaoglu et al., 2018). We consider three underlying structures on two datasets: Pendulum in Figure 2(a), CelebA-Smile in Figure 2(b), and CelebA-Attractive in Figure 2(c).

Review summaries:
This paper presents a latent variable model where the variables in the latent space are causally disentangled, i.e. the disentanglement is ensured according to a structural causal model (SCM). The resulting model is made up of two parts. The first one, a generative unsupervised part, is essentially a VAE and is defined with the VAE ELBO loss. The second part is supervised and accounts for the causal disentanglement of the factors that are assumed to underlie the distribution; the authors claim the fewer supervised samples are required to estimate the second part of the loss alone. The two parts are then combined with a hyperparameter.