Disentangled Generative Causal Representation Learning
Authors: Xinwei Shen
Furui Liu
Hanze Dong
Qing LIAN
Zhitang Chen
Tong Zhang
Venue:  (2021)
Keywords: disentanglement
causality
representation learning
generative model

Abstract: This paper proposes a Disentangled gEnerative cAusal Representation (DEAR) learning method. Unlike existing disentanglement methods that enforce independence of the latent variables, we consider the general case where the underlying factors of interests can be causally correlated. We show that previous methods with independent priors fail to disentangle causally related factors. Motivated by this finding, we propose a new disentangled learning method called DEAR that enables causal controllable generation and causal representation learning. The key ingredient of this new formulation is to use a structural causal model (SCM) as the prior for a bidirectional generative model. A generator is then trained jointly with an encoder using a suitable GAN loss. Theoretical justification on the proposed formulation is provided, which guarantees disentangled causal representation learning under appropriate conditions. We conduct extensive experiments on both synthesized and real datasets to demonstrate the effectiveness of DEAR in causal controllable generation, and the benefits of the learned representations for downstream tasks in terms of sample efficiency and distributional robustness.

Under review as a conference paper at ICLR 2021
DISENTANGLED GENERATIVE CAUSAL REPRESENTATION LEARNING
Anonymous authors
Paper under double-blind review
ABSTRACT
This paper proposes Disentangled gEnerative cAusal Representation (DEAR) learning method. Unlike existing disentanglement methods that enforce indepen- dence of the latent variables, we consider the general case where the underlying factors of interests can be causally correlated. We show that previous methods with independent priors fail to disentangle causally correlated factors. Motivated by this finding, we propose a new disentangled learning method called DEAR that enables causal controllable generation and causal representation learning. The key ingredient of this new formulation prior for bidirectional generative model. The prior is then trained jointly with a generator and an encoder using a suitable GAN loss incorporated with super- vision. Theoretical justification the proposed formulation is provided, which guarantees disentangled causal representation learning under appropriate condi- tions. We conduct extensive experiments both synthesized and real datasets demonstrate the effectiveness of DEAR in causal controllable generation, and the benefits of the learned representations for downstream tasks in terms of sample efficiency and distributional robustness.
use a structural causal model (SCM) as the
INTRODUCTION
Consider the observed data from a distribution 9N on X' C Rd and the latent variable a from a prior p= on R In bidirectional generative models (BGMs), we are normally interested in learning an encoder E X' + Z to infer latent variables and generator G Z + X' to generate data, to achieve both representation learning and data generation. Classical BGMs include Variational Autoencoder (VAE) (Kingma & Welling, 2014) and BiGAN (Donahue et al., 2017). In representation learning, was argued that an effective representation for downstream learning tasks should disentangle the underlying factors of variation (Bengio et al., 2013). In generation, it is highly desirable if one can control the semantic generative factors by aligning them with the latent variables such in StyleGAN (Karras et al., 2019). Both goals can be achieved with the disentanglement of latent variable z, which informally means that each dimension of = measures a distinct factor of variation in the data (Bengio et al., 2013).
Earlier unsupervised disentanglement methods mostly regularize the VAE objective to encourage independence of learned representations (Higgins et al., 2017: Burgess et al., 2017: Kim & Mnih, 2018; Chen et al., 2018; Kumar et al., 2018). Later, Locatello et al. (2019) show that unsupervised learning of disentangled representations is impossible: many existing unsupervised methods are actually brittle, requiring careful supervised hyperparameter tuning implicit inductive biases. To promote identifiability, recent work resorts various forms of supervision (Locatello al., 2020b; Shu et al., 2020; Locatello et al., 2020a). In this work, we also incorporate supervision on the ground-truth factors in the form stated in Section 3.2.
Most of these existing methods are built on the assumption that the underlying factors of variation are mutually independent. However, in many real world cases the semantically meaningful factors of interests are not independent (Bengio et al., 2020). Instead, semantically meaningful high-level vari- ables are often causally correlated, i.e., connected by a causal graph. In this paper, we prove formally that methods with independent priors fail to disentangle causally correlated factors. Motivated by this observation, we propose a new method to learn disentangled generative causal representations called DEAR. The key ingredient of our formulation is structured causal model (SCM) (Pearl et al.,

---

Under review as a conference paper at ICLR 2021
2000) the prior for latent variables in bidirectional generative model. With some background knowledge on the binary causal structure, the causal prior is then learned jointly with generator and an encoder using suitable GAN (Goodfellow et al., 2014) loss. We establish theoretical guarantees for DEAR to learn disentangled gled causal representations under appropriate conditions.
An immediate application of DEAR is causal controllable generation, which can generate data from any desired interventional distributions of the latent factors. Another useful application of disen- tangled representations is use such representations in downstream tasks, leading to better sample complexity (Bengio et al., 2013; Schölkopf et al., 2012). Moreover, it is believed that causal disen- tanglement is invariant and thus robust under distribution shifts (Schölkopf, 2019; Arjovsky et al., 2019). In this paper, we demonstrate these conjectures in various downstream prediction tasks for the proposed DEAR method, which has theoretically guaranteed disentanglement property.
We summarize our main contributions as follows:
We formally identify a problem with previous disentangled representation learning methods using the independent prior assumption, and prove that they fail to disentangle when the underlying factors of interests are causally correlated.
We propose new disentangled learning method, DEAR, which integrates SCM prior into bidirectional generative model, trained with suitable GAN loss.
We provide theoretical justification on the identifiability of the proposed formulation.
Extensive experiments are conducted on both synthesized and real data to demonstrate the effec- tiveness of DEAR in causal controllable generation, and the benefits of the learned representations for downstream tasks in terms of sample efficiency and distributional robustness.
2 OTHER RELATED WORK
GAN-based disentanglement methods. Existing methods, including InfoGAN (Chen et al., 2016) and InfoGAN-CR (Lin et al., 2020), differ from our proposed formulation mainly in two folds. First they still assume an independent prior for latent variables, so suffer from the same problem with previous VAE-based methods mentioned above. Besides, the idea of InfoGAN-CR is to encourage each latent code to make changes that are easy to detect, which actually applies well only when the underlying factors are independent. Second, InfoGAN as a bidirectional generative modeling method further requires variational approximation apart from adversarial training, which is inferior to the principled formulation in BiGAN and AGES (Shen et al., 2020) that we adopt.
Causality with generative models. CausalGAN (Kocaoglu al., 2018) and concurrent work (Moraffah et al., 2020) of ours, are unidirectional generative models (i.e., a generative model that learns a single mapping from the latent variable data) that build upon cGAN (Mirza & Osin- dero, 2014). They assign an SCM to the conditional attributes while leave the latent variables as independent Gaussian noises. The limit of a cGAN is that it always requires full supervision on attributes to apply conditional adversarial training. And the ground-truth factors are directly fed into the generator as the conditional attributes, without an extra effort align the dimensions between the latent variables and the underlying factors, so their models have nothing to do with disentangle- ment learning. Moreover their unidirectional nature makes it impossible to learn representations. Besides they only consider binary factors whose consequent semantic interpolations appear non- smooth, as shown in Appendix D. CausalVAE (Yang et al., 2020) assigns the SCM directly on the latent variables, but built upon iVAE (Khemakhem et al., 2020), it adopts conditional prior given the ground-truth factors so also limited to fully supervised setting.
PROBLEM SETTING
3.1 GENERATIVE MODEL
We first describe the probabilistic framework of disentangled learning with supervision. We follow the commonly assumed two-step data generating process that first samples the underlying generative factors, and then conditional on those factors, generates the data (Kingma & Welling, 2014). Dur- ing the generation process, the generator induces the generated conditional pa(a (2) and generated joint distribution (a( 2) = P=(2) (2))pa()))) During the inference process, the encoder induces the encoded conditional qm (=(2 (r) which can be factorized Gaussian and the encoded joint distribution
2

---

Under review as a conference paper at ICLR 2021
(2 2) (2())) We consider the following objective for generative modeling:
(I) which is shown to be equivalent to the evidence lower bound used in VAEs up constant, and allows a closed form only with factorized Gaussian prior, encoder and generator (Shen et al., 2020).
Lgen = DrL (q, =), Pa(x, x)),
Since constraints on the latent space are required to enforce disentanglement, it is desired that the distribution family of qe (z, 2) and pa 2) should be large enough, especially for complex data like images. Normally more general implicit distributions terms of expressiveness (Karras et al., 2019; Mescheder et al., 2017). Then minimizing (1) requires adversarial training. as discussed detailedly in Section 4.3.
favored over factorized Gaussians in
3.2 SUPERVISED REGULARIZER
To guarantee disentanglement, incorporate supervision when training the BGM, following the similar idea in Locatello et al. (2020b) but with a different formulation. Specifically, let & E R be the underlying ground-truth factors of interests of æ, following distribution pş. and y be some continuous or discrete observation of the underlying factor & satisfying & = E(ys(z) for = I,..., m. For example, the case of human face images, 1/1 can be the binary label indicating whether person is young or not, and & [x))lz) = Ply 1/x) is the probability of being young given one image x.
Let E(r)) be the deterministic part of the stochastic transformation E(x), ie., B(x) = ((E())))), which is used for representation learning. We consider the following objective:
L(E, G) = Lgen (E, G) + XLap(E))
(2)
where Luu = > === (E,, 34] if ys is the binary or bounded continuous label of the i-th factor εx, where CE(1, v) - y log or (1) (1 y) log(1 a()) is the cross-entropy loss with (-) being the sigmoid function; Tup Ee= m E Er,y [x(x) y:12 if is the continuous observation of E,, and À > is the coefficient to balance both terms. We empirically find the choice of À quite insensitive to different tasks and datasets, and hence set A 5 in all experiments. Estimating of Lygen requires the unlabelled dataset {a (x.",...,." xV\ while estimating Luup requires a labeled dataset {(x), (P))))) : = N, N
where N₂ can be much smaller than N.
In contrast, Locatello et al. (2020b) propose the regularizer Lsap Ei= Ex,:/CE(E,,, (E(E,()), 2t) in- volving only the latent variable & which is a part of the generative model, without distinguishing from the ground-truth factor & and its observation y. Hence they do not establish any theoretical jus- tification on disentanglement. Besides, they adopt a VAE loss for Tagn with an independent prior, which suffers from the unidentifiability problem described in the next section.
3.3 UNIDENTIFIABILITY WITH AN INDEPENDENT PRIOR
Intuitively, the above supervised regularizer aims at ensuring some alignment between factor & and latent variable z. We start with the definition of a disentangled representation following this intuition.
Definition 1 (Disentangled representation). Given the underlying factor & E R. of data x, a deter- ministic encoder E is said to learn disentangled representation with respect to & if 1,..., ......, m, there exists 1-1 function such that E,(x) = (E.).. Further, stochastic encoder E is said to be disentangled wrt if its deterministic part E (x) is disentangled wrt E.
As stated above, consider the general case where the underlying factors of interests are causally correlated. Then the goal becomes to disentangle the causal factors. Previous methods mostly use an independent prior for which contradicts with the truth. We make this formal through the following proposition, which indicates that the disentangled representation is generally unidentifiable with an independent prior.
Proposition 1. Let E* be any encoder that is disentangled wrt &. Let b' = Lamp (E*), a ming Legn(EE G), and b (in Leup (E)) Assume the elements of & are con- nected by a causal graph whose adjacency matrix Ao not a zero matrix. Suppose the prior pa is factorized, i.e, pz(z)= II pz(z.). Then we have a > O, and either when 2 or b" < b and X < a there exists solution (E', G') such that for any generator G, we have T(F''G") I(F",G).
3

---

Under review as a conference paper at ICLR 2021
All proofs are given in Appendix A. This proposition directly suggests that minimizing (2) favors the solution (E", G") over one with a disentangled encoder E". Thus, with an independent prior we have no way to identify the disentangled solution with À that is not large enough. However, in real applications it is impossible to estimate the threshold, and too large À makes it difficult to learn the BGM. In the following section we propose a solution to this problem.
4 CAUSAL DISENTANGLEMENT LEARNING
4.1 GENERATIVE MODEL WITH A CAUSAL PRIOR
We propose to use causal model as the prior ps. Specifically we use the generalized nonlinear Structural Causal Model (SCM) proposed by Yu et al. (2019) as follows
2 =(I A ( == Fs(c).
(3)
where A is the weighted adjacency matrix of the directed acyclic graph (DAG) upon the k elements of = (i.e., Ag # O if and only if 2, is the parent of 2,), c denotes the exogenous variables following (0, f and h are element-wise nonlinear transtormations, and 3 (f, h, A) denotes the set of parameters of f. h and A. with the parameter space B. Further let 14 = I(A o 0) denote the corresponding binary adjaceney matrix, where I is the element-wise indicator function.
When f is invertible, (3) is equivalent to -= + h(c)
(4) which indicates that the factors z satisfy linear SCM after nonlinear transformation f. and enables
interventions on latent variables as discussed later. The model structure is presented in Figure 1. Note that different from our model where is the latent variable following the prior (3) with the goal of causal disentanglement, Yu et al. (2019) proposed a causal discovery method where variables are observed with the aim of learning the causal structure among 2.
Inference
Generation
SCM
X
Encoder
Generator
-X
Data
I Latent
Data
Figure 1: Model structure of a bidirectional generative model (BGM) with an SCM prior.
In causal structure learning, the graph is required to be acyclic. Zheng et al. (2018) propose an equality constraint whose satisfaction ensures acyclicity and solve the problem with augmented Lagrangian method, which however leads to optimization difficulties (Ng et al., 2020). In this paper, to avoid dealing with the non-convex constraint but focus disentangling, we assume to have some prior knowledge of the binary causal structure. Specifically, we assume the super-graph of the true binary graph A- is given, the best case of which is the true graph while the worst that only the causal ordering is available. Then we learn the weights of the non-zero elements of the prior adjacency matrix that indicate the sign and scale of causal effects, jointly with other parameters using the formulation and algorithm described in later sections. To incorporate structure learning methods and jointly learn the structure from scratch with guarantee of identifiability could be explored in future work. An ablation study is done in Appendix B regarding this prior knowledge.
To enable causal controllable generation, we use invertible f and and describe the mechanism to generate images from interventional distributions of latent variables. Note that interventions can be formalized as operations that modify a subset of equations in (4) (Pearl et al., 2000). Suppose we would like to intervene have latent factors z inferred from data x, i.e., z = E(x), or sampled from prior pa, we follow the intervened equations in (4) to obtain z' the left hand side using ancestral sampling by performing (4) iteratively. Then we decode the intervened latent factor z' to generate the sample G(z'). In Section 5.1 we define the two types of interventions of most interests in applications.
the i-th dimension of i.e., Do( 2; = c). where is a constant. Once
Another issue of the model is the latent dimension, to handle which we propose the so-called com- posite prior. Recall that m is the number of generative factors that we interested to disentangle,
4

---

Under review as a conference paper at ICLR 2021
e.g., all the semantic concepts related to some field, where m tends to be smaller than the total num- ber Mf of generative factors. The latent dimension k of the generative model should be no less than M to allow sufficient degree of freedom in order to generate or reconstruct data well. Since M is generally unknown in reality, we set a sufficiently large k; at least larger than m which is a trivial lower bound of M. The role of the remaining k dimensions is to capture other factors necessary for generation whose structure is not cared or explicitly modeled. Then we propose to use prior that is a composition of a causal model for the first m dimensions and another distribution for the other h m dimensions to capture other factors necessary for generation, like standard Gaussian.
4.2 FORMULATION AND IDENTIFIABILITY OF DISENTANGLEMENT
In this section, we present the formulation of DEAR and establish the theoretical justification of it. Compared with the BGM described in Section 3.1, here we have one more module to learn that is the SCM prior. Thus pG(z, 2) becomes PG, (z. 2) = pr pr(2) (2)p)(=)2) where pr or (2(2) denotes the marginal distribution of Fs(c) with c ~ (0, 7). We then rewrite the generative loss as follows
Lgen( (E, F) Dxr (g 2), PG, (x, (z):
(5)
Then we propose the following formulation to leam causal generative causal representations:
min E.G.F
L(E,G,F) Lgen (E, F) (E).
In order achieve causal disentanglement, we make two assumptions the causal model. Assump- tion 1 supposes a sufficiently large capacity of the SCM in (3) to contain the underlying distribution Pe. which reasonable due the generalization of the nonlinear SCM. Assumption 2 states the identifiability of the true causal structure LA, of E, which is applicable given the true causal order- ing under basic Markov and causal minimality conditions (Pearl, 2014; Zhang & Spirtes, 2011).
Assumption 1 (SCM capacity). The underlying distribution Pg belongs to the distribution family {ps : 3 E B}, i.e., there exits 30 = ho. Ao) such that Pe = Pß.
Assumption 2 (Structure identifiability). For all 3 (f. (f, e B with ps Ppo- it holds that 1 1
The following theorem then guarantees that under appropriate conditions the DEAR formulation can learn the disentangled representations defined in Definition
Theorem 1. Assume the infinite capacity of E and G. Further under Assumption 1-2, DEAR for- mulation (6) learns the disentangled encoder E Specifically, we have 9r(E.) = o -E if CE loss used for the supervised regularizer, and (E.))) = & if La loss is used.
Note that the identifiability we establish in this paper differs from some previous work on the pa- rameter identifiability, e.g., Khemakhem al. (2020). We argue that learn disentangled repre- sentations, the form in Definition I, i.e., the existence but not the uniqueness of gi ss is sufficient to identify the relation among the representations and the data. In contrast, parameter identifiability may not be achievable in many cases like over- parametrization. Thus the identifiability discussed here is more realistic in terms of the goal of disentangling. Later we provide empirical evidence to support the theory directly through the application in causal controllable generation.
4.3 ALGORITHM
In this section we propose the algorithm to solve the formulation (6). The SCM prior PF(>) and implicit generated conditional pa (z|z) make (5) lose analytic form. Hence we adopt GAN method to adversarially estimate the gradient of (5). We parametrize Ex(x) and Gz(z) by neural networks. Different from Shen al. (2020), the prior also involves learnable parameters. We present in the following lemma the gradient formulas of (5).
Lemma 1. Let r(x, 2) = q 2) ( 2) and D(x, 2) logr(x, 2). Then we have (I Co voGo(=)),
V Jagen = - -B [s(a,, V, =)
V. Jagen = Ex V D II V.
(7)
VgLgen = - [, 2) " (FF(() + V(D(a,)) V
where s(x,2) 2) P(e()) is the scaling factor.
5

---

Under review as a conference paper at ICLR 2021
We then estimate the gradients in (7) by training discriminator D via empirical logistic regression:
ming [S.] (a,,)es, log(L (D)((())) [s L(,,)es, log(L + (ar((.))) where Se and So are finite samples from gr(2, 2) and pa z) respectively, leading to a GAN approach.
Based above, propose Algorithm I to learn disentangled generative causal representation. Algorithm 1: Disentangled gEnerative cAusal Representation (DEAR) Learning
training set (#i..... W. M1..... MN. initial parameters Φ, 0, B, vb batch-size r 1 while not convergence do for multiple steps do Sample [#1,...,Zr] ] from the training fci, fe,,..., En from () Generate from the causal prior 2, (e())) .....7
Input:
2
5
Update d by descending the stochastic gradient:
- V loo(l -
log(1 + (ePlle(to)))))))
6.
Sample {#1.... Jn, 91, $1....,, Dn .e.,...,,6 fn) as above; generate 2; = Pa(es)
Compute 0-gradient:
Σ Di=r (o
Compute d-gradient: En VaDa(us, (x:)) - EE= Valum(d;z,,
Compute 3-gradient: - - a(G(z,)
Fs(e))
10
Update parameters (b, , using the gradients
Return: Φ 8
Remark: without loss of generality, assume N, samples the training set and the first samples in each mini-batch has available labels; may vary across different iterations.
EXPERIMENTS
We evaluate our methods on two datasets. The first one is a synthesized dataset Pendulum similar the one in Yang al. (2020). As shown in Figure each image is generated by four contin- uous factors: pendulum Langle Liglle shadow_length and structure is given in Figure 2(a) following physical mechanisms. To make the dataset realistic, we introduce random noises when generating the two effects from the causes, representing the mea- surement error. We further introduce 20% corrupted data whose shadow randomly generated, mimicking some environmental disturbance. The sample sizes for training, validation and test set are all 6,724.'
whose underlying
The second one is real human face dataset CelebA (Liu et al., 2015), containing 202,599 images with 40 labelled binary attributes. Among them we consider two groups of causally correlated factors shown in 2(b,c), We believe these two datasets are diverse enough to assess methods. All the details of experimental setup and architectures are given Appendix C.
penduluma angle(t)
light angle(1)
smile(D)
gender(1)
young(()
gonder(1)
chock- bano(2)
eye bag(2)
shadow: length(()
shadow position(3)
open(3)
chubby(S)
eyo(4)
chubby(4)
make up(3)
receding hairlino(2)
(a) Pendulum
(b) CelebA-Smin
(c) CelebA
Figure 2: Underlying causal structures.
5.1
CONTROLLABLE GENERATION
We first investigate the performance of our methods in disentanglement through applications in causal controllable generation (CG). Traditional CG methods mainly manipulate the independent generative factors (Karras al., 2019), while we consider causally correlated. With a learned SCM as the prior, we are able to generate images from any desired interventional distributions of the latent factors. For example, we can manipulate only the
general case where the factors are
Thhe Pendulum dataset will be released causal disentanglement benchmark

---

Under review as a conference paper at ICLR 2021
cause factor while leave its effects unchanged. Besides, the BGM framework enables controllable generation either from scratch or a given unlabeled image.
We consider types of intervention. In traditional traversals, manipulate one dimension of the latent vector while keep the others fixed to either their inferred or sampled values (Higgins et al., 2017). A causal view of such operations is intervention all the variables by setting them as constants with only one of them varying. Another interesting type of interventional distribution is intervene on only one latent variable, i.e., Fao(= (2). The proposed SCM prior enables to conduct such intervention though the mechanism given in Section 4.1.
Figure 3-4 illustrate the results of causal controllable generation of the proposed DEAR and the baseline method with an independent prior, S-3-VAE (Locatello et al., 2020b). Results from other baselines including S-TCVAE, S-FactorVAE (which essentially make no difference due to the inde- pendence assumption) and CausalGAN are given in Appendix D. Note that we do not compare with unsupervised disentanglement methods because of fairness and their lack of justification.
In each figure, we first infer the latent representations from test image in block (c). The traditional traversals of two models are given in blocks (a.b). We see that each line when manipulating one latent dimension, the generated images from our model vary only in single factor, indicating that our method can disentangle the causally correlated factors. It is worth pointing out that we are the first to achieve the disentanglement between the cause and its effects, while other methods tend to entangle them. In block (d), we show the results of intervention on the latent variables representing the cause factors, which clearly show that intervening Results in Appendix D further show that intervening on an effect node does not influence its cause.
cause variable changes its effect variables.
Since the underlying factors causally correlated, all previous quantitative metrics for disentan- glement no longer apply. We provide more qualitative traversals in Appendix D to show the overall performance. A quantitative metric for causal disentanglement is worth exploring in future work.
single
with others fixed
16
1
v
x
x
1
4
Single factor aftecten
Muttpee factora effected
!t
x
1
v
r
1
1
Disentangled
(a) Traversal S.A.
(b) Traversal DEAR
Intervone on pandulum_angge
shardow Vengtti Eposition
Intervere Jiht
shadow Jengihé position affected
(c) Test data
Intervention
Figure 3: Results of causal controllable generation on Pendulum.
Traverse A
single
laternt with
others foxe
Muliipe
hout
factors
Disentangled
affected
No factos
affected
(8) Traversal of S-P-VAE
(b) Traversal of DEAR
Intervene on smmle
affected
gender
(6) Test data
Intervention cause
Figure Results of causal controllable generation on CelebA.
7

---

Under review as a conference paper at ICLR 2021
Table 1: Sample efficiency and test accuracy with different training sample sizes. DEAR-lin and -nir denote the model with linear and nonlinear Line is unsupervised: 2-3 are semi-supervised; others supervised.
(a) CelehA
(b) Pendulum
Method
())
10,000(%)
Eff(%
100( %)
all( (6)
EM(()
ResNet
68. 18
39.51
79.71 85,59=0.27
10.64
87.97+211
DEAR- lin- 10%
78.09
79.54.10.1
08.18
18.9
95,43 1
DEAR-nlr- 10%
80.30
80.87:10.1
99.29.0.23
87.65
91.27 2.221
96.03.1229
ResNet-pretrain
76.84=2.0
83.754
91.74
79.59
89.16=1.6
89.28
V-VAE
77.07 1.42
79.87-11.67
96,49
84.16
90,89
92.60
S-8-VAE
71.78=#1.99
76.63+024
93,67
+9.95
87.87
90.98+147
S-TC-VAE
77.10=2.08
81.63++200
94.45+2272
85.36+1111
90.33
94.51+111
DEAR-lin
83.51
84.9220.1
98.342208
90.21
93.31
96.682089
DEAR-nir
84.44
85.10
99.23=0.51
90.6220.1
92.57=0.08
97.931029
5.2 DOWNSTREAM TASK
The previous section verifies the good disentanglement performance of DEAR. In this section, equipped with DEAR, investigate and demonstrate the benefits of learned disentangled causal representations in sample efficiency and distributional robustness.
We state the downstream tasks. On CelebA, ure 2(c). We artificially create a target label т = 1 if young=1, gender=0, receding hairline=O,, make up=1, chubby=0, eye_ bag=O, and т 0 otherwise, indicating slim young woman with makeup and thick hair. On the pendulum dataset, we regard the label of data corruption as the target T, i.e., 7 1 if the data is corrupted and the downstream tasks of predicting the target label. In both cases, the factors of interests in Figure 2(a.c) are causally related т, which are the features that humans use do the task. Hence is conjectured that disentangled representation of these causal factors tends to be more data efficient and invariant to distribution shifts.
consider the structure CelebA-Attractiv in Fig-
kind of attractiveness
otherwise. We consider
5.2.1 SAMPLE EFFICIENCY
For BGM including the previous state-of-the-arr supervised disentangling methods S-VAEs (Lo- catello et al., 2020b) and DEAR, we use the learned encoder to embed the training data the latent space and train a MLP classifier on the representations to predict the target label. Without en- coder, one normally needs to train a convolutional neural network with raw images as the input. Here we adopt the ResNet50 as the baseline classifier which is the architecture of the BGM encoder. Since disentangling methods use additional supervision of the generative factors, we consider an- other bascline that is pretrained using multi-label prediction of the factors the same training set.
To measure the sample efficiency, we use the statistical efficiency score defined as the average test accuracy based on 100 samples divided by the average accuracy based on 10,000/all samples, fol- lowing Locatello et al. (2019). Table presents the results, showing that DEAR owns the highest sample efficiency on both datasets. ResNet with raw data inputs has the lowest efficiency, although multi-label pretraining improves its performance to a limited extent. S-VAES have better efficiency than the ResNet baselines but lower accuracy under the case with more training data, which we think is mainly because the independent prior conflicts with the supervised loss as indicated in Proposition 1, making the learned representations entangled (as shown in the previous section) and less infor- mative. Besides, we also investigate the performance of DEAR under the semi-supervised setting where only 10% of the labels are available. We find that DEAR with fewer labels has comparable sample efficiency with that in the fully supervised setting, with a sacrifice in accuracy that is yet still comparable to other baselines with more supervision.
Note that the definition of attractiveness here only refers to one kind of attractiveness, which has nothing do with the linguistic definition of attractiveness.
8

---

Under review as a conference paper at ICLR 2021
Table 2: Distributional robustness. The worst-case and average test accuracy
(a) CelebA
(b) Pendulum
Method
WorstAcc(%)
AvgAce(%)
WorstAcc(%4 (%)
AvgAcc(%)
ResNet
59.12:1178
82.12.10.6
60.48
87.40
DEAR-lin-le 10%
71.40:047
81.04.:014
63.93
89.70:tme3
DEAR- 10%
70.44-11.02
81.94.10.1
65.59
90.19440.3
ResNet-mulli
59.17
82
61.70
87.20:1.00
S-VAE
60.54:148
79.51
20.78:4445
8.,26+131
VAB
63.85
80.8240.19
4-.73
86,99
S-TC- VAE
64.93
81.58 H0.14
35.50+5.57
86.64+
DEAR-lin
76.05 +:070
43,56
74.95+126
93,61 +0.13
DEAR-nir
71.37
83.8
72.481074
13
5.2.2 DISTRIBUTIONAL ROBUSTNESS
We manipulate the training data inject spurious correlations between the target label and some spurious attributes. On CelebA, we regard mouth_open as the spurious factor; on Pendulum, we choose background
E (blue(+), white(-)). We manipulate the training data such that the target label is more strongly correlated with the spurious attributes. i.e., the target label and the spurious attribute of 80% of the examples are both positive or negative, while those of 20% examples are opposite. For example, in the manipulated training set, 80% smiling examples in CelebA have an open mouth; 80% corrupted examples in Pendulum are masked with blue background. The test set however does not have these correlations, leading to distribution shift.
Intuitively these spurious attributes are not causally correlated to the target label, normal in- dependent and identically distributed (IID) based methods like empirical risk minimization (ERM) tend to exploit these casily Iearned spurious correlations in prediction, and hence face performance degradation when the such correlation no longer exists during test. In contrast, causal factors regarded invariant and thus robust under such shifts. Previous sections justify both theoretically and empirically that DEAR can learn disentangled causal representations. We then apply those represen- tations by training a classifier upon them, which is conjectured to be invariant and robust. Baseline methods include ERM, multi-label ERM to predict target label and all the factors considered in dis- entangling to have the same amount of supervision, and S- VAEs that can not disentangle well in the causal case.
Table 2 shows the average and worst-case (Sagawa et al.. 2019) test accuracy to assess both the over- all classification performance and distributional robustness, where group the test set according to the two binary labels, the target one and the spurious attribute, into four cases and regard the one with the worst accuracy as the worst-case, which usually owns the opposite correlation the train- ing data. We see that the classifiers trained upon DEAR representations outperform the baselines both metrics. Particularly, when comparing the worst-case accuracy with the average one, we observe slump from around 80 around 60 for other methods CelebA, while DEAR enjoys an acceptable small decline. These results support the above conjecture and the benefits of causal disentanglement in distributional robustness.
6 CONCLUSION
This paper showed that previous methods with the independent latent prior assumption fail to learn disentangled representation when the underlying factors of interests causally correlated. We then proposed a new disentangled learning method called DEAR with theoretical guarantees. Extensive experiments demonstrated the effectiveness of DEAR causal generation, and the benefits of the learned representations for downstream tasks.
REFERENCES
Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization.
arXiv preprint arXiv: 1907.09000000 002893902,3 2019.

---

Under review as a conference paper at ICLR 2021
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 35(8):1798-1828, 2013.
Yoshua Bengio, Tristan Deleu, Nasim Rahaman, Rosemary Ke, Sébastien Lachapelle, Olexa Bila- niuk, Anirudh Goyal, and Christopher Pal. meta-transfer objective for learning disentangle causal mechanisms. In ICLR, 2020.
Christopher P. Burgess, Irina Higgins, Arka Pal, Loic Matthey, Nicholas Watters, Guillaume Des- jardins, and Alexander Lerchner. Understanding disentangling in beta-vae. NIPS Workshop of Learning Disentangled Features, 2017.
Tian Qi Chen, Xuechen Li, Roger B. Grosse, and David K. Duvenaud. Isolating sources of disen- tanglement in variational autoencoders. In NeurlPS, 2018.
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. In Advances in neural information processing systems, pp. 2172-2180, 2016.
Jeff Donahue, Philipp Krähenbühl, and Trevor Darrell. Adversarial feature learning. ICLR, 2017.
Vincent Dumoulin, Ishmael Belghazi, Ben Poole, Alex Lamb, Martín Arjovsky, Olivier Mastropi- etro, and Aaron C. Courville. Adversarially learned inference. In ICLR, 2017.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural infor- mation processing systems, pp. 2672-2680, 2014.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with constrained variational framework. In ICLR, 2017.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4401-4410, 2019.
Ilyes Khemakhem, Diederik Kingma, Ricardo Monti, and Aapo Hyvarinen. Variational autoen- coders and nonlinear ica: A unifying framework. In International Conference Artificial Intel- ligence and Statistics, pp. 2207-2217, 2020.
Hyunjik Kim and Andriy Mnih. Disentangling by factorising. ICML, 2018
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. In ICLR, 2014.
Murat Kocaoglu, Christopher Snyder, Alexandros G Dimakis, and Sriram Vishwanath. Causalgan: Learning causal implicit generative models with adversarial training. In ICLR, 2018.
Abhishek Kumar, Prasanna Sattigeri, and Avinash Balakrishnan. Variational inference disentan- gled latent concepts from unlabeled observations. In ICLR, 2018.
Felix Leeb, Yashas Annadani, Stefan Bauer, and Bernhard Schölkopf. Structural autoencoders im- prove representations for generation and transfer. arXiv preprint 2.XX::0006607796 2020.
Zinan Lin, Kiran K Thekumparampil, Giulia Fanti, and Sewoong Oh. Infogan-er and modelcentral- ity: Self-supervised model training and selection for disentangling gans. In ICML, 2020.
Ziwei Liu, Ping Luo. Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of the IEEE international conference computer vision, pp. 3730-3738, 2015.
10

---

Under review as a conference paper at ICLR 2021
F. Locatello, S. Bauer, M. Lucic, G. Raetsch, S. Gelly, B. Schölkopf, and O. Bachem. Chal- lenging common assumptions in the unsupervised learning disentangled representations. In Proceedings of the 36th International Conference Machine Learning (ICML). volume 97 of Proceedings of Machine Learning Research, pp. 4114-4124. PMLR, June 2019. URL. ht.p::/ http:://prrceedingge mll prw.ssppeessseem https/////////a//////e/// 1o1 9a.html.
Francesco Locatello, Ben Poole, Gunnar Rätsch, Bernhard Schölkopf, Olivier Bachem, and Michael Tschannen. Weakly- disentanglement without compromises. In ICML, 2020a.
Francesco Locatello, Michael Tschannen, Stefan Bauer, Gunnar Rätsch, Bernhard Schölkopf, and Olivier Bachem. Disentangling factors of variation using few labels. In ICLR, 2020b.
Lars Mescheder, Sebastian Nowozin, and Andreas Geiger. Adversarial variational bayes: Unifying variational autoencoders and generative adversarial networks. In Proceedings of the 34th Inter- national Conference on Machine Learning- Volume 70, pp. 2391-2400. JMLR. org 2017.
Mehdi Mirza and Simon Osindero. Conditional generative adversarial nets. arXiv preprint arXiv:1411.1 2014.
Raha Moraffah, Bahman Moraffah, Mansooreh Karami, Adrienne Raglin, and Huan Liu. Can A causal adversarial network for learning observational and interventional distributions. arXiv preprint arXiv:2008. / 1376, 2020.
Ignavier Ng, AmirEmad Ghassami, and Kun Zhang. On the role of sparsity and dag constraints for learning linear dags. arXiv preprint arXiv:2006.10201, 2020.
Judea Pearl. Probabilistic reasoning in intelligent systems: networks of plausible inference. Elsevier, 2014.
Judea Pearl et al. Models, reasoning and inference. Cambridge, UK: ContrridgeUnivversittpress.. 2000.
Ali Razavi, Aaron van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vae-2. In Advances in Neural Information Processing Systems, pp. 14866-14876. 2019.
Shiori Sagawa, Pang Wei Koh, Tatsunori Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generaliza- tion. arXiv preprint 1911v:911100770 2019.
Bernhard Schölkopf. Causality for machine learning. arXiv preprint arXiv: 2019.
Bernhard Schölkopf, Dominik Janzing, Jonas Peters, Eleni Sgouritsa, Kun Zhang, and Joris Mooij. On causal and anticausal learning. In ICML, 2012.
Xinwei Shen, Tong Zhang, and Kani Chen. Bidirectional generative modeling using adversarial gradient estimation. arXiv preprint 1611 2020.
Rui Shu, Yining Chen, Abhishek Kumar, Stefano Ermon, and Ben Poole. Weakly supervised disen- tanglement with guarantees. In ICLR, 2020.
Mengyue Yang, Furui Liu, Zhitang Chen, Xinwei Shen, Jianye Hao, and Jun Wang. Causalvae: Structured causal disentanglement in variational autoencoder. arXiv preprint arXiv:2004.08697 2020.
Yue Yu, Jie Chen, Tian Gao, and Mo Yu. Dag-gun: Dag structure learning with graph neural networks. In ICML, 2019.
Han Zhang, Ian Goodfellow, Dimitris Metaxas, and Augustus Odena. Self-attention generative adversarial networks. In International Conference on Machine Learning, pp. 7354-7363, PMLR, 2019.
Jiji Zhang and Peter Spirtes. Intervention, determinism, and the causal minimality condition. Syn- these, 18243))335-347 201
11

---

Under review as a conference paper at ICLR 2021
Shengjia Zhao, Jiaming Song, and Stefano Ermon. Learning hierarchical features from generative models. In ICML, 2017.
Xun Zheng, Bryon Aragam, Pradeep K Ravikumar, and Eric P Xing. Dags with no tears: Continuous optimization for structure learning. In Advances in Neural Information Processing Systems, pp. 9472-9483, 2018.
12

---

Under review as a conference paper at ICLR 2021
APPENDIX A PROOFS
A.1
PROOF OF PROPOSITION 1
Proof. On one hand, by the assumption that the elements of & are connected by a causal graph whose adjacency matrix is not a zero matrix. Then exist * j such that & and { are not independent, indicating that the probability density of cannot factorized. Since E is disentangled wrt &, by Definition 1, vi 1, .....: there exists 94 such that (x) (.))). This implies that the probability density of Ex (x) is not factorized.
On the other hand, notice that the distribution family of the latent prior is {p= Ps is factorized). Hence the intersection of the marginal distribution families of z and E (a)) is an empty set. Then the joint distribution families of (zz, E" (x)) and (G(2), 2) also have empty intersection.
We know that Lgen (E, G) = 0 implics qa q z) = Pa (2, 2) which contradicts the above. Therefore, have ming Lgenn (L", > 0.
Let (E')'') the solution of the optimization problem min((E, (E) Then we have L' L(E', G") Ab, and L = L(E", G) 2 a + Ab > Xb for any generator G. When ba 2 directly have L' L'. When b* b and X is not large enough, i.e., X 6-6 have L' L
A.2 PROOF OF THEOREM 1
Proof. Assume deterministic.
, m. first consider the cross-entropy loss On one hand, for each i = I,...,
Lsu,, (E)= (r.))) (CE(E(()), v.31
where p(ya the probability mass function of the binary label yt given z, characterized by P(y: 11x) ((uilar) and (P(s= = 0f=:) B(yila)) Let
Laup. (F) = ff | (i o 1 1 (
1
dady: = 0.
Then we know that (x) - o (E(()z)) = o- (Ex) minimizes Lsup.
Consider the L2 loss
/ - ysll? drdy;-
(0) = E((x.)) (Fxxt))F((xt) vi12 = p(e)p(ul)))1(((())
Let
@L - y; O.
p(x))(ul)))
(E,
Then we know that E (x) E(y1(z) = & minimizes sau. in this case.
On the other hand, by Assumption there exists 3g = (fo, ho: Ao) such that Ps = Ps. Then the distribution of E² (x:) is given by P3. with 3 = (g fo. ho, Ag). Assumption 2 ensures that there no 3' = (f', Mi, A') such that A' * Ao but p3- = p3.. Let F = F3.. Further due to the infinite capacity of G, we have the distribution family of Pc. r- (z. 2) contains qr- (x. 2). Then by minimizing the loss in (6) over G, we can find G* such that pa-.Fe (x, z) matches qr (x, z) and thus Legen E G* F*) reaches O.
Hence minimizing T. = Tagen + Aouup which is the DEAR formulation (6), leads to the solution with (x) = (Es) with 9=(E,) = σ (E,) if CE loss used, and gi(Ex) = E, if L2 loss used, and the true binary adjacency matrix.
For a stochastic encoder, we establish the disentanglement of its deterministic part as above, and follow Definition to obtain the desired result.
13

---

Under review as a conference paper at ICLR 2021
A.3 PROOF OF LEMMA 1
We follow the same proof scheme as in Shen et al. (2020) where the only difference lies in the gradient wrt the prior parameter B. To make this paper self-contained, we restate some proof steps here using our notations.
Let denote the vector 2-norm. For a scalar function h(x, y), let Vzh(x, y) denote its gradient with respect to xr For a vector function g(æ, y), let V Vzg(z, y) denote its Jacobi matrix with respect to a. Given differentiable vector function g(x) I2 - R we use V g(x) denote its divergence, defined as
h d[g(z}]
(() =
a|z]
where [xl, denotes the j-th component of x. We know that
v g(z)d===))
for all vector function g(x) such that g(oo) 0. Given matrix function w(x) (wi (x)........ wx(x)) I Rext where each wi (2)x) 1..., is a k-dimensional differentiable vector function, its divergence is defined as V w(x) = (V w (z), V wi())).
To prove Lemma 1, we need the following lemma which specifies the dynamics of the generator joint distribution Py (2, 2) and the encoder joint distribution (e 2), denoted by po(x, 2) and (1 2) here.
Lemma 2. Using the definitions and notations in Lemma we have
Vopo,e(E, 2) = - va,,, 2) g0(x)) p0,s( 2)V V. go(x),
(8)
V Vouol1, 2) = -- -- 2) ea(z) (s(x, z) c<(=),
(9)
(falz))
V VsPo,e(x,2) 2) Va,o,oo 2) fa - VePo,o(æ, 2) f5(2) Po.
(10)
ff(2))
for all data and latent variable where go(Go(z,e) VoGa(z,e), e cs(Es(z,e)) V Fala,, )) (a(Fa(c)) FFFe(e), and (G(((((())) VFG((F()))
Proof of Lemma 2. We only prove (10) which the distinct part from Shen al. (2020).
Let be the dimension of parameter B. To simplify notation, let random vector 2 = Es(e) and X = (()) E IRd and Y = (X, Z) E Rd+k and let p be the probability density of Y. For each i= 1, 2, let A de, where e; I-dimensional unit vector whose i-th component is and all the others are zero, and is a small scalar. Let Z' = F3+5(E), X' = G(Z") and Y' = (X', Z') so that Y is random variable transformed from by
(faLX) (a(2) 4+ o(8).
Y'
Let p' be the probability density of Y'. For arbitrary o' (a',z") E Rd+k let yf = y (fa(a)) A + o()) and y = =(x, =x,,)). Then we have
fs(e)
p'()')
p(y) dot /dy)1
p(y) det + (T)()) +
p1u)(1 + T (ff(l)).f(()) +o())-
= =P(y)(1 A T (()(
=p(V) - A P (fole), +o(0)
=P(v') A(fo(2'),ff((2)) Vap(a', A p(po'(( fs('), V. f=((()) + +
14

---

Under review as a conference paper at ICLR 2021
Since of is arbitrary, above implies that
pl (z, = A T (fa(z), fa(z)) (V.p(a, V v=p(a, Vzp(z,2) A p(z (V fa(z'), V fa(=)) +
for all x E IR IR and l..... ,., leading to (10) by taking + 0, and noting that = and p' P3+a- Similarly
obtain (8) and (9).
Proof of Lemma 1. Recall the objective (g p) I q 2) log(p(æ, 2)/ 2)dzz its integrand by Egg. p). Let (g) p)/ppp We have
Denote
v Vsl((<2, p(r, =2((((x ().
where V, (r, 2) is computed in Lemma
Besides, we have
V
V. (GG4, = V.1
+ +((av)\VVla.a)111((
+ (E((0.)))(,,)))((()))
Thus,
Vglgen /
(5)((((, =))drd2 = /1::::////66001/1004444444000000
where compute Vzl(ggp)) slx, and 3.
Hence
V VaLgon - -E(x,s)
Ee + [b(=))
V VV, VpF()))) (=((((()
where the second equality follows reparametrization.
Lemma 3. For any a, R (a b), the of continuous piece-wise linear function P is dense in Cla, b] where the metric d(f.9) g) = sup,e((.M) [f(x) g(x)]. Note that P is defined as
P ((-a)//nnn/e
(b- (o-a)/t /h-1
Pn
+
wilx - a (a--ih)(e 1h) 2 a ih)
wi, & E
1=
where [:] here is floor function.
Proof. Since Ja, b) compact, any function E C(a, b] uniform continuous, i.e. Ve > there
exists > O such that
x vl = ff()) < e/2.
Let (a, b] = - (an, bn]. and (x) be a linear function, such that
a, nh,
b, = a + (n + 1)h,
In (a,) f(an),
gr(b,) ) = f(bb),
Nh = b
15

---

Under review as a conference paper at ICLR 2021
Assume that h < 8. For any (an, bnl) have
If(x) (1))) _ min {1f(x) [(an) + (g:(z) 9i (n) [f(x)) f(b, [(b))l + (9:(x)) 24 (b())))
< (a)( (=,b,)+++mmm) min (1f(x) f(an)), 1f(x) f(bn))
[[(a)) [(b.)) min (1[(x) - ((an)):1(()) - 1(b))l
c.
Thus,
sup 1 9m)( <e.
zE (an..b.)
We define
g(x) Σ In (n(x)1(x Jan bul)
which is obvious that g(x) E Pn C P. And we have
sup
[f(x) - g(x)l <e
xE(a,b]
Therefore, P is dense Cla b) and Pa eensee.
APPENDIX B LEARNING THE STRUCTURE
As mentioned Section 4.1, our DEAR algorithm requires the prior knowledge on the super-graph of the true graph over the underlying factors of interests. The experiments shown in the main text are all based on the assumption that the true graph is given. In this section we investigate the performance of the learned weighted adjacency matrix and present an ablation study on different extents of prior knowledge on the structure.
B.1 GIVEN THE TRUE GRAPH
Figure 5 shows the learned weighted adjacency matrices when the true binary structure is given, whose weights show sensible signs and scalings consistent with common knowledge. For example, smile and its effect mouth open are positively correlated. The corresponding element in the weighted adjacency Aus of (a) turns out positive, which makes sense. Also gender (the logit of male) and its effect make up are negatively correlated. Then A13 (b) turns out negative.
B.2 GIVEN THE TRUE CAUSAL ORDERING
Consider the Pendulum dataset, whose ground-truth structure is given in Figure 2(a). Consider a causal ordering pendulum aendulummangle, light .aggle, shadow shadow-position, shadow_lee length, given which we start with full graph whose elements are randomly initialized around 0 as shown in Figure 6(a). Figure presents the adjacency matrices learned by DEAR at different training epochs, from which we see that it eventually obtains the learned structure that nearly coincides with the one learned given the true graph shown in Figure 5(c). This experiment shows the potential of DEAR to incorporate structure learning methods to Iearn the latent causal structure from scratch, which will be explored in future research.
(a) CelebA-Smile
(b) CelebA-Attractive
(c) Pendulum
Figure 5: Learned adjacency matrices for different underlying structures.
16

---

Under review as a conference paper at ICLR 2021
APPENDIX C IMPLEMENTATION DETAILS
In this section we state the details of experimental setup and the network architectures used for all experiments.
Preprocessing and hyperparameters. We pre-process the images by taking a center crops of 128 128 for CelebA and resizing all images CelebA and Pendulum to the 64 64 resolution. We adopt Adam with 31 = 0, 32 0.999, and learning rate of x 10 for D, x 10 for E, G and F, and 10-2 for the adjacency matrix A. We use mini-batch size of 128. For adversarial training in Algorithm 1, we train the D once on each mini-batch, The coefficient À of the supervised regularizer is set to We use CE supervised loss for both CelebA with binary observations of the underlying factors and Pendulum with bounded continuous observations. Note that L2 loss works comparable to CE loss on Pendulum. In downstream tasks, for BGMs with an encoder, we train a two-level MLP classifier with 100 hidden nodes using Adam with learning rate of x 10-2 and mini-batch size 128. Models were trained for around 150 epochs on CelebA and 600 epochs on Pendulum NVIDIA RTX 2080 Ti.
Network architectures. We follow the architectures used Shen et al. (2020). Specifically, for such realistic data, we adopt the SAGAN (Zhang et al., 2019) architecture for D and G. The D network consists of three modules as shown in Figure and detailed described (Shen et al., 2020). Details for newtork G and D2 are given in Figure 7 and Table 3. The encoder architecture is the ResNet50 (He et al., 2016) followed by 4-layer MLP size 1024.
Implementation of the SCM. Recall the nonlinear SCM as the prior
= ((T >")==((())=))>>> Fe(c)-
We find Gaussians expressive enough as unexplained noises, set h the identity map- ping. As mentioned Section 4.1 we require the invertibility of f. We implement both linear and nonlinear ones. For a linear f, we formally refer
f(2) W2 +
where W and b are learnable weights and biases. Note that W is diagonal matrix to model the element-wise transformation. Its inverse function can easily computed by
(=) W- b).
For non-linear f, we use piece-wise linear functions defined by
Na m
(z0())))) = w (0 (0)
- a;) + b00
where .04) denote the i-th dimension of points of division, and 1(-) is the indicator function. From its denseness shown in lemma 3, the family of such piece-wise linear functions is expressive enough model general element-wise non- linear invertible transformations.
vector-function, Ap a1 aN,
Experimental details for baseline methods. We reproduce the S- 8-VAEs including S- S-VAE, S-3- VAE and S-TCVAE using and with the same architecture as ours and adopt the same opti- mization algorithm for training. The coefficient for the independence regularizer is set to 4 since we
in
680
16
(a) Epoch
(b) Epoch 100
(c) Epoch 200
(d) Epoch 500
Figure 6: Learned adjacency matrices at different training epochs, starting from a random initializa- tion.
17

---

Under review as a conference paper at ICLR 2021
notice that setting larger independence regularizer hurts disentanglement in the correlated case. For the supervised regularizer, we use À = 1000 for a balance of generative model and supervision. The ERM ResNet is trained using the same optimizer with learning rate of 1 x 10
Batchwnom
ReLL
ReLU
3x3 Cony
Upsample
Upsample
Conv
Data x
D.
1xt Conv
3kd Comv
Average pooling
Batch-oomm
ReLU
Score (ALS 2
RaLU
3e3 Conv
Sxd
Average pooking
(b)
Figure 7: (a) Architecture of the discriminator D(x, 2): (b) A residual block (up scale) in the SAGAN generator where we use nearest neighbor interpolation for Upsampling; (c) A residual block (down scale) in the SAGAN discriminator.
Table 3: SAGAN architecture (A: = 100 and ch = 32).
(a) Generator
(b) Discriminator module D₁
Input: = E R ~ (0,
Input: RGB image 22
Linear + 4 x 4 x 16ch
ResBlock down ch - 2ch
ResBlock up 16ch >> 16ch
Non-Local Block (64 x 64)
ResBlock up 16ch >>
ResBlock down 2ch + 4ch
ResBlock up 8ch 4ch
ResBlock down 4ch 8ch
Non-Local Block (64 x 64)
ResBlock down 8ch -> 16ch
ResBlock up 4ch -> 2ch.
ResBlock 16ch + 16ch
BN, ReIUU, 3 x 3 Conv 2ch - 3
ReL.U, Global average pooling (fz)
Tanh
Lincar (8x)
APPENDIX D ADDITIONAL RESULTS OF CAUSAL CONTROLLABLE GENERATION
In this section we present more qualitative results of causal controllable generation on two datasets using DEAR and baseline methods, including S-VAEs (Locatello et al., 2020b) and CausalGAN (Kocaoglu et al., 2018). We consider three underlying structures on two datasets: Pendulum in Figure 2(a), SelebA-Smile in Figure 2(b), and CelebA-Atrrattie in Figure 2(c).
18

---

Under review as a conference paper at ICLR 2021
Traversal
(b) Intervention PelebA-Smile
(c) Traversal
(d) Intervention GedlebA-Atracttinnn
1
1
J
J
}
}
(c) Traversal (Pendulum)
(f) Intervention (Pendulum)
Figure 8: Results DEAR. Note that the ordering of the representations matches that of the indices in Figure On the left we show the traditional latent traversals (the first type of intervention stated in Scction 5) On the right show the results of intervening one latent variable from which we the consequent changes of the others (the first type of intervention). Specifically intervening the cause variable influences the effect variables while intervening on effect variables makes no difference to causes.
19

---

Under review as a conference paper at ICLR 2021
(a) S-TCVAE (CelebA-Smile)
(b) S-TCVAE (CelebA-Attractive)
(c) FactorVAE (CelebA Smile)
(d) S- V-FactorVAE (cttat--Atractivee)
U
70
2
2
7
(e) 3-3- VAE (CelebA-Smil -mile)
(1) 8-8-VAE (Celeb--Attractive)
Figure 9: Traversal results of baseline methods. We see that (1) entanglement occurs; (2) some factors are not detected (traversing some dimensions of the latent vector makes difference in the decoded images.) Besides, the generated images from VAES are blurry.
20

---

Under review as a conference paper at ICLR 2021
(a) CausalGAN (CelebA-Smill)
(b) CausalGAN (CelebA-Atrractvve)
1
1
(c) S-TCVAE (Pendulum)
(d) S-FactorVAE (Pendulum)
Figure 10: Traversal results of baseline methods. CausalGAN uses the binary binary factors as the conditional attributes, so the traversals appear some sudden changes. contrast, we regard the logit of binary labels as the underlying factors and hence enjoy smooth manipulations. In addition, the controllability of CausalGAN also limited, since entanglement still exists. Results of S- VAEs are explained in Figure 9.
21


Reviews:
[{"review_id": "UdQ41APz-HT", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper propose to learning causal disentangled representation which conforms human's cognition. The learned representation can be applied in causal controllable generation and benefit downstream tasks. The uses of this method is interesting.\nThe paper has some advantages:\nIt only requires part of sample is supervised, the previous methods need fully supervied information.\nThe previous method unreasonably force the latent factors to be independent. However, this method incorporate a SCM as prior distribution of latent representations. Therefore, the learned factors can be causally correlated which is consistent to practice.\nThe experiments are extensively conducted. The results demonstrate the effectiveness of this method.\n\nI have some questions about this method:\nIn section 4.3, the gradient in Algorithm 1 is not consistent with the gradient in Lemma 1. Which one is correct?\nIn section 5.2.2, the test set is grouped into 4 groups according to the two binary labels, the target one and the spurious one. It is not clear what is the meaning of target one and spurious one.\nThe paper require the information about SCM as prior, which is a strong requirement. I think it is of great significance to investigate how to learn causal disentanlged representation with weaker prior knowledge in the future. I understant it is very difficult, but it is worth researching.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "TghOScY6Ygw", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a novel and interesting method to formulate learning of correlated disentangled causal factors as a part of encode-decoder framework using a Gan-style learning approach. The proposed idea is interesting but I think the wording of the paper as well as the theoretical and experimental results could be improved. \n\nPlease find my detailed comments:\n1- In equation (6), I suggest it to be modified to L(E, G, F)=L_{gen}(E,F,G)+\\lambda L_{sup}(E). In other words, it would make sense to remove min_{E, G, F}.\n\n2- Proposition 1: I think a necessary condition for a to be not equal to zero is that if the ground truth latent factors are correlated. Therefore, I suggest this assumption to be added to the assumptions of the proposition. Regarding the proof, I found the following typos. First, in the third line of the proof, I think x_{j} should be \\xi_{j} instead. In the last two lines of the proof, L(E^*, G)\\geq a+b^* should be replaced with L(E^*, G)\\geq a+\\lambda b^*.\n\n3- Regarding theorem 1, I found the sentence \"Then DEAR learns the disentangled encoder E*\" not accurate. It is because assuming E, G and f does not guarantee that this framework is able to actually learn E*. Therefore, I suggest to change this statement to be more like the following statement: The encoder for the optimal solution of the proposed loss fn will be disentangled encoder E*. I found the following typos in the proof of this theorem. First, in the third line, inside the parenthesis, y should be replaced with y_i. In the 8th line, the derivative should be with respect to E_i(x) and not \\sigma(E_i(x)). In the 11th line, p(x,z) should be replaced with p_G(x, z).\n\n4- In the section 4.3 Algorithm, I found the following typos. First, in the second line, \"implicit generated conditional\" should be \"implicit generative conditional\". In the 5th line, \"theorem\" should be replaced with \"Lemma\". In Lemma 1, in the presented gradients, L_{gen} should be L.\n\n5- Regarding the Experiments section, I did not find the qualitative experiments to clearly present the superiority of this method to disentangle correlated causal factors over the previous methods with a similar objective.  \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "Q7__ULkQPlX", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a latent variable model where the variables in the latent space are causally disentangled, i.e. the disentanglement is ensured according to a structural causal model (SCM). The resulting model is made up of two parts. The first one, a generative unsupervised part, is essentially a VAE and is defined with the VAE ELBO loss. The second part is supervised and accounts for the causal disentanglement of the factors that are assumed to underlie the distribution; the authors claim the fewer supervised samples are required to estimate the second part of the loss alone. The two parts are then combined with a hyperparameter.\n\nThe authors motivate their approach by comparing it to models using an independent prior for the latent factors and to ones using structural causal models in the latent space. For the former case, they claim that in reality the latent factors are frequently dependent, which their model is able to capture. For the latter, they argue that the competing models use SCMs for conditional factors rather than unconditional as in the proposed approach.\n\nSubsequently, disentangled representation is defined as one where a 1-to-1 function can be established between the data and the underlying factors corresponding to the nodes of the SCM (Definition 1). The authors then propose to use the general nonlinear SCM model in the latent space which concludes the actual definition of their model (part 2 of the loss mentioned before). Finally, they note that the model ensures disentanglement as defined before and describe a GAN-like algorithm for minimizing the compound loss (part 1 + part 2). The paper is concluded with a series of experiments which include a quantitative comparison of accuracy, sample efficiency and distributional robustness against a number of VAE-based disentanglement methods.\n\n\n*****Strengths:*****\n\nThe incorporation of structural causal models to generative modelling, especially in the context of disentanglement and the resulting explainability, is an important topic and this paper is certainly relevant in this area.\n\nThe experiments show a quantitative improvement over a number of competing disentanglement methods.\n\n\n*****Weaknesses:*****\n\nGenerative modelling with SCMs, being an important area of research, was addressed before. The review of related work provided here is rather limited. Moreover, the authors decided to contrast their approach mostly to disentanglement strategies with an independence prior. I think the paper would be much more convincing if more weight were put on direct comparison with other methods using SCMs for generative modelling such as causalGAN and cognate methods (which is absent from the experiment section at the moment). As a matter of fact even the limited high-level comparison to such models provided in section 2 left me confused (what is the “unidirectional nature” of the other models compared to DARE? What is meant by “direct access to attributes” in the other methods – labels? Factors?).\n\n\n*****Questions / feedback:*****\n\nSee weaknesses.\n\nWhat is the role of the hyperparameter lambda (equations 2, 6)? Can we alter the sample complexity (wrt labels y) by varying it, or make the model less dependent on the assumed generalized SCM?\n\nWhy is there a difference in notation between Z and \\xi? Is the DAG defined by Z not the same as the SCM defined by \\xi?\n\n\n*****Typos / minor comments:*****\n\nSection 4.2: justification on -> justification of\n\nSection 4.1: interventional distribution is never defined\n\nI found Figure 1 unreadable.\n\n\n*****Post Rebuttal*****\n\nI would like to thank the authors for the rebuttal. While I think more discussion / comparison to approaches without an explicit independence prior would further improve the paper, the authors have clarified many of my doubts. I have therefore decided to raise my final score.", "extra_scores": {}, "extra_text": {}}, {"review_id": "2loUJM3auk8", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Summary of the paper**\nThis submission addresses the topic of causal representation learning/causal disentanglement, i.e., learning explanatory latent factors which are not mutually independent, but instead connected via an underlying causal model. The authors propose the DEAR framework to address this task. DEAR consists of an encoder-decoder architecture (similar to VAEs) which is combined with a separate discriminator. The key learning procedure is to match two joint distributions over data x and latents z: (i) the joint induced by pushing the (empirical?) data distribution q(x) through the encoder q(z|x); and (ii) the joint induced by pushing the (model?) prior p(z) through the generator p(x|z). The task of distribution matching is done using a separate discriminator trained to distinguish q(x,z) from p(x,z) while encoder and decoder are trained to fool the discriminator. \nAnother key ingredient is the form of the model prior p(z) which does not factorise, but instead takes a particular form inspired by structural causal models (SCMs).\nIn order to learn the dependent latent variables, two additional strong assumptions are made: (i) full supervision in the form of observations of all ground truth causal factors y for (a subset of) the training data; (ii) ground truth knowledge of the causal ordering or full causal graph. The latter (ii) is used to construct the prior p(z) to allow for the same DAG structure as the true factors, and the former (i) is used to augment the joint distribution matching objective with a supervised regulariser that penalises deviation of the encoded factors from the groundtruth factors.\nThe authors claim to prove non-identifiability for models assuming a factorised prior, as well as identifiability for their model, but I have some reservations and concerns about the level of detail and technical correctness of the proofs (more details below).\nExperiments are performed on three datasets with causally related latent variables: Pendulum, CelebA-Smile and CelebA-Attractive. The proposed DEAR approach is compared against a baseline (S-$\\beta$-VAE) in terms of latent traversals, controllable generation, and two downstream tasks (sample efficiency and distributional robustness) and outperforms the baseline on these.\n\n**Summary of evaluation**\nWhile the paper addresses a very important problem and contributes some interesting ideas and results, the submission has too many technical flaws and misses too many important details to merit publication in the current form. \n\n**Pros**\n- the setting of learning causally related latent factors considered in the paper is a very interesting and highly relevant\n- the approach of matching the two joint distributions induced by encoder and decoder in an adversarial fashion is well-motivated, interesting, and (as far as I can tell) novel\n- the authors (attempt to) provide some theoretical insights along with the proposed method which is appreciated\n- the datasets seem suitable for the task, and some of the results (e.g., Fig.3) are quite interesting and promising\n\n**Cons**\n- the main part of the paper (Sections 3 & 4) is quite hard to parse due to inconsistencies, non-standard use of notation (CE, q(x) for data distribution, ...), and important missing details (e.g., the assumed data generating process in terms of $\\xi$ is never specified)\n- the model relies on known ground truth causal ordering AND supervision in the form of annotated ground truth factors which is a very strong assumption and too unrealistic to be useful for causal representation learning in practice: learning the causal variables and their structure is precisely the task in causal disentanglement and it seems that most information is assumed given. At the same time, the importance of these assumptions is not stressed enough in the abstract and introduction and formulations like \"using a suitable GAN loss\" are misleading since they suggest an unsupervised approach.\n- only linear relationships between the causal variables $V=f^{-1}(Z)$ with additive noise are allowed (before a component-wise nonlinearity f is applied) and it is not clear why this restriction is necessary or why the component wise nonlinearity f cannot simply by absorbed into the decoder\n- the theoretical results claimed in the paper are not backed up by rigorous proofs (I checked proofs for Proposition 1 and Theorem 1, but not for the derivations of the gradients/Lemmas): there are several mathematical errors, missing assumptions, and unclear steps brushed under the carpet\n- there is no information about the choice of two key quantities, the hyperparameter $\\lambda$ determining the weight of the supervised regulariser, and the fraction/amount of labelled examples $N_S$ (nor any ablation on how performance depends on these) which harms transparency and reproducibility\n\n**Comments regarding the Proof of Proposition 1**\n- 2nd sentence: E should be E*.\n- The assumption of a non-factorising distribution over $\\xi$ is not stated as part of the proposition. This should be stated as a clear assumption in which case the first paragraph of the proof becomes obsolete. \n- 2nd paragraph, 2nd sentence refers to distributions of z and E(x), but I believe E*(x) is meant. Otherwise the claim that \"the intersection of their distribution families\" (could also be clarified) is incorrect or at least not justified.\n- 3rd paragraph refers to $p_E(x,z)$ which is not defined. I assume $q_E(x,z)$ is meant? \n- A key problem I see with the definition of b as the minimum of $L_{sup}$ subject to the constraint $L_gen=0$. It is not clear why this minimum should exist, i.e., why it should be possible to satisfy the constraint if the prior $p(z)$ factorises, but the true generative process is based on dependent latents. Certainly, there are not sufficient details provided to assess the correctness of the statement.\n- there are missing $\\lambda$'s in $L^*$, should be $L^*$ $\\geq a+\\lambda b^* > \\lambda b^*$\n\n**Comments regarding the proof of Theorem 1**\n- as stated before, cross entropy is between two distributions does not involve a sigmoid function\n- what is the distribution $p(y_i|x)$? this was not defined in the paper\n- the partial derivative is incorrect, should be $y/\\sigma(E_i)-(1-y_i)/(1-\\sigma(E_i))$ inside the bracket\n- the second partial derivative misses a factor 2\n- the main part of the proof is only informally sketched in natural language: this is not sufficiently rigorous for a prove that claims to show identifiability; more detail is needed to be able to assess the correctness of the claim. It is also not clear whether infinite data is assumed.\n\n**More detailed comments and questions**\n- the use of $q_x(x)$ for the data distribution is unconventional: usually this is denoted $p_{\\text{data}}(x)$ or simply $p(x)$; also it would be good to distinguish between the empirical distribution provided by the training data and the true distribution which is unknown at different parts in the paper\n- the related work section is very short and key references are not described in any detail: it would make the paper more accessible if key building blocks of the proposed method (e.g., Locatello et al., Yu et al.) were explained in more detail here\n- I do not follow the logic  of the first sentence at the top of page 3, please clarify or rephrase\n- the assumed true data generating process to be captured is never specified: in particular, what is the generative process for the ground truth factors $\\xi$ and how are these factors decoded to give rise to the observations?\n- I do not understand the need to consider separate $\\xi$ and $y$? what does the addition of $y$ add? also what distribution is the expectation in the definition of $y$ with respect to: $\\xi_i=\\mathbb{E}[y_i|x]$ ?\n- the definition of cross entropy stated in 3.2 does not coincide with the universally accepted form of cross entropy (which does not have the sigmoid term)\n- what is $\\phi$ in $L_{sup}(\\phi)$ in 3.2? it is not defined at this point, and why does it not appear as argument to the other $L_{sup}$'s?\n- Definition 1 seems strange to me as it does not allow for permutations of the ground truth factors. I would have expected that for all i there is a j s.t. $E_i(x)=g_j(\\xi_j)$. Can you elaborate?\n- last paragraph in Section 3: The claim made here seems incorrect: (E',G') as constructed in Proposition 1 minimises neither (1) nor (2), but $L_{sup}(E)$ subject to $L_{gen}=0$; see also my point on the feasibility of this constraint in the context of the proof of proposition 1\n-Sec.4.1: if f is a component wise tranformation, it is not clear what it's addition adds, and why you do not consider a causal model over $V=f^{-1}(z)$ instead. the particular choice of a linear SCM is not motivated further which makes it hard to understand. Also, it would help to specify $p(\\epsilon)$ earlier as otherwise one wonders what the point of $h$ is.\n- you refer to the \"direction\" and scale of causal effects, but the direction is given by assumption in the form of causal order. do you mean the \"sign\" of the linear effect?\n- the procedure described for computing interventions is not motivated from the latent SCM, and in fact it appears to be closer in nature to a counterfactual (rung 3) rather than an interventional (rung 2) quantity since the noise variables are kept fixed when performing the intervention as far as I understand. For an intervention, the exogenous variables would have to be re-sampled.\n- I do not understand the reasoning in the last paragraph of Sec. 4.1.: the claim that k=m fails due to capacity issues is not explained and does not make sense to me. I do not understand why one would consider different sets of latent, since independent factors can also be incorporated within the latent SCM, these would simply have zeros in the corresponding rows and columns in the adjacency matrix.\n- Theorem 1 assumes that \"the true binary adjacency matrix\" can be learned. This is too vague: what does \"can be learnt\" mean exactly? Also, why is the adjacency matrix binary all of the sudden?\n- 4.3 states that unlike prior work, the prior distribution has learnable parameters, but there are a number of works considering learning the prior. you should either rephrase or reference such works. \n- Algorithm 1: $\\psi$ is not defined at this point. I assume it refers to the discriminator parameters?\n- 5.1 you state that you can only manipulate the causal factors whilr leaving their effects unchanged, but this is incorrect. Effects will change unless you also intervene on those and fix them (which I believe is what you mean, given the experiments).\n- There is a reference to Figure 3-9 which confused me.\n- 5.2 I find the choice of defining \"attractiveness as a slim young woman with makeup and thick hair\" very questionable from an ethical perspective. I think that we should try our best in academia to overcome such outdated notions, and given that this was a design choice you made, I think it would be nice to consider a more neutral/less controversial example. (This is just a personal note and has no bearing on my evaluation!)\n\n**Post rebuttal comments**\nI thank the authors for the detailed response. I think that some points have been clarified and corrected, and I have increased my score slightly to reflect this. I still think the paper needs another iteration before publication though.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a method to formulate learning of causally disentangled representation as a part of the encode-decoder framework. Although the reviewers agree that the paper presents some interesting ideas, they feel the paper is not ready for publication yet.  In particular, I encourage the authors to take the feedback of reviewer R2 into account, which is quite detailed and provides substantive ways of improving the work. After all, I recommend rejection.


Decision: Reject