Identifying Coarse-grained Independent Causal Mechanisms with Self-supervision
Authors: Xiaoyang Wang
Klara Nahrstedt
Oluwasanmi O Koyejo
Venue:  (2021)
Keywords: Causal Mechanisms
Identifiability
Disentangled Representations

Abstract: Current approaches for learning disentangled representations assume that independent latent variables generate the data through a single data generation process. In contrast, this manuscript considers independent causal mechanisms (ICM), which, unlike disentangled representations, directly model multiple data generation processes (mechanisms) in a coarse granularity. In this work, we aim to learn a model that disentangles each mechanism and approximates the ground-truth mechanisms from observational data. We outline sufficient conditions under which the mechanisms can be learned using a single self-supervised generative model with an unconventional mixture prior, simplifying previous methods. Moreover, we prove the identifiability of our model w.r.t. the mechanisms in the self-supervised scenario. We compare our approach to disentangled representations on various downstream tasks, showing that our approach is more robust to intervention, covariant shift, and noise due to the disentanglement between the data generation processes.

Under review as a conference paper at ICLR 2021
IDENTIFYING COARSE-GRAINED CAUSAL MECHANISMS WITH SELF-SUPERVISION
INDEPENDENT
Anonymous authors
Paper under double-blind review
ABSTRACT
Current approaches for learning disentangled representations assume that inde- pendent latent variables generate the data through single data generation process. contrast, this manuscript considers independent causal mechanisms (ICM), which, unlike disentangled representations, directly model multiple data gener- ation processes (mechanisms) a coarse granularity. this work, we aim to learn model that disentangles each mechanism and approximates the ground- truth mechanisms from observational data. We outline sufficient conditions un- der which the mechanisms can be learned using single self-supervise genera- tive model with an unconventional mixture prior, simplifying previous methods. Moreover, prove the identifiability of our model w.r.. the mechanisms in the self-supervised scenario. We compare our approach to disentangled representa- tions on various downstream tasks, showing that our approach more robust intervention, covariant shift, and noise due to the disentanglement between the data generation processes.
INTRODUCTION
The past decade witnessed the great success of machine learning (ML) algorithms, which achieve record- record-breaking performance in various tasks. However, most of the successes are based on dis- covering statistical regularities that are encoded in the data, instead of causal structure. As a con- sequence, standard MI. model performance may decrease significantly under minor changes to the data, such as color changes that are irrelevant for the task, but which affect the statistical associa- tions, On the other hand, human intelligence is more robust against such changes (Szegedy et al.) 2013). For example, a baby learns to recognize a digit, the baby can recognize the digit regardless of color, brightness, or even some style changes. Arguably, it is because human intelligence relies on causal mechanisms (Schölkopf et al. [0122] entailed data distribution (Parascandolo et al.) 2018). The independent causal mechanisms (ICM) principle (Schölkopf al. 2012) Peters al. 2017) assumes that the data generating process is composed of independent and autonomous modules that do not inform influence each other. The promising capability of causal mechanisms grows an activate subfield (Parascandolo et al.] 2018] Locatello al. 2018a b. Bengio et al. 2019). Recent works define the mechanisms to be: func- tions that generate a variable from the cause (Bengio et al., 2019). 2) functions that transform the data (e.g. rotation) (Parascandolo et al. 2018). and disentangled mixture of independent gen- crative models that generate data from distinct causes (Locatello et al. [2018a b). Throughout this paper, we refer type mechanisms shared mechanisms and type 3) mechanisms as generative mechanisms.
|0122|Petess Peters el. al. (017) which make sense beyond a particular
Despite the recent progress, unsupervised learning of the generative and shared mechanisms from complex observational data (c.g. images) remains difficult and unsolved task. In particular, pre- vious approaches (Locatello al. 2018a b) for disentangling the generative mechanisms rely on competitive training, which does not directly enforce the disentanglement between generative mech- anisms. The empirical results show entanglement. Additionally, Parascannolo et al. (2018) proposed ciittur--f--xxpert--beesee method to learn the shared mechanisms using canonical distribution and a reference distribution, which contains the transformed data from the canonical distribution. Such reference distribution is generally unavailable in real-world datasets. To create reference distribution. we need to use the shared mechanisms that we aim the learn. This causes a chicken-egg problem. Besides, the unsupervised learning of the deep generative model is proved to be unidentifi-

---

Under review as a conference paper at ICLR 2021
able (Locatello et al.
2019 Khemakhem et al. 2020). Lacking identifiability makes it impossible to
learn the right disentangled model (Locatello et al., 2019). Recent methods (Locatello et al. 2020;
Khemakhem et al. 2020) leverage weak-supervisson or auxiliary variables to identify the right deep
generative model. However, such weak-supervision or auxiliary variables still do not exist in con-
ventional datasets (e.g. MNIST).
We, therefore, seek a practical algorithm with identifiability result that disentangles the mecha- nisms from i.i.d data without manual supervision. To this end, we propose single self-supervised generative model with an unconventional mixture prior. In the following sections, we refer to our model as the ICM model. Using a single selff-upprrvied generative model would allow to lever- age the recent progress in deep generative clustering (Mukherjee et al. 2019), which would en- force the disentanglement between the generative mechanisms. We use the following example to illustrate the relationship between the generative model and the mechanisms. Let us assume we have a generative model
Z - X, two generative mechanisms Mo : 2M, - XMo. M1 2M; XM and one shared mechanism Mg XM, Zs X' where 2 = [Z ZM, Zs] and XM = XMa U AM. We have G[[=Mo 0, =s]) = (M(MM(ZMM)) æg) and G(O, RM, =s]) = Ms(Mi (ZM, ), zs). Our mixture prior is unconventional because the mixture components To keep the omit the normalization factor. We disentangle the generative mechanisms by
(O I). 0, I)
I). [))} instead of {W o²), ( (H1,
notations clear, disentangling the type of variations (causes) carried by each ZM,, vk E (1,2,..., N) where N is the number of generative mechanisms. The disentanglement between the generative mechanisms and the shared mechanisms will be guaranteed by the prior itself. Furthermore, we theoretically prove that the ICM model is identifiable w.r.t. the mechanisms without accessing any label. The key contributions of this paper are:
We propose simpler method learn the mechanisms with only self- sulf-suue
We design an unconventional mixture prior that enforce disentanglement.
We prove the first identifiability result w.r.t. the mechanisms in the self-supervised scenario.
We develop a novel method to quantitatively evaluate the robustness of ML models under covariant shift using the covariant that is naturally encoded in the data.
We conduct extensive experiments to show that our ICM model is more robust against intervention, covariant shift, and noise compared to disentangled representations.
2 RELATED WORK
Functional Causal Model In functional causal model (FCM), the relationships between variables are expressed through deterministic, functional equations: 2; = f( u4, I,... N. The uncertainty in FCM is introduced via the assumption that variables u,, i = 1,.., N, are not observed (Pearl et al., 2000p. each function in FCM represents an autonomous mechanism, such FCM is called a structural model. Moreover, if each mechanism determines the value of one and only one variable, then the model is called a structural causal model (SCM). The SCMs form the basis for many statistical methods (Mooij & Heskes 2013 Mooij et al. (016) that aim at inferring knowledge of the underlying causal structure from data (Bongers et 2016). Taking the view from the SCM's perspective, we want to learn mixture of causal models whose inputs pure latent variables and whose output is a single high-dimensional variable that describes complex data such as images. Different from other SCM approaches, where the unobserved variables only introduce uncertainty to the model, the latent variables in our model carries distinct variations in the dataset.
Independent Component Analysis Discovering independent components of the data generating process has been studied intensively (Hyvärinen & Ojal 2000 Hyvarinen et al., 2019). A recent work (Khemakhem et al. 2020) bridges the gap between the nonlinear independent component analysis (ICA) and the deep generative model. 'Tee nonlinear ICA with auxiliary variables brings parameter-space identifiability variational auto-encode. The nonlinear ICA tackles the parameter- space identifiability of deep generative models. However, the parameter-space identifiability does not guarantee the disentanglement between causes. We will discuss the difference in section
Disentangled Representations Disentangled representations assume that the data is generated using a set of independent latent explanatory factors (Bengio al. 2013). Previous works (Higgins)
2

---

Under review as a conference paper at ICLR 2021
M
ZC
S
D
M
Ms o MN
DN
MM0
ZMd
M
zs
Ms.
M
My o Mo
Du
D
x²
(a)
(b)
(c)
(d)
Figure I: (a) The graphical model of deep generative models and disentangled representations. (b) The assumed data generation process in previous works (Parascandolo al. 2018 Locatello et al. 2018b). where zM, is the cause for the hthh generative mechanism, M2 is the generative mechanism and Ms, the shared mechanism. (c) The graphical model of our approach, where zc is a confounding variable that controls which group of causes will take effect. (d) The assumed data generation processes in our approach. Each cluster D2 in the data set D is generated by its associated generative mechanism M2 and the shared mechanisms Ms. RM, and zs are the causes for the generative mechanism Mg and the shared mechanism Ms, respectively.
et al. 2017 Kumar et al., 2017) proposed various unsupervised methods to disentangle the latent explanatory factors. Later, unsupervised disentanglement proved to be impossible (Locatello) et al. 2019). Recent works (Shu et al. 2019 Locatello et al. (2020) leverage weak-supervision (c.g. pair-wisc label) to identify the right disentangled model. Compared to disentangled represtations, ICM model takes different assumption. We assume there are groups of latent explanatory factors that generate clusters in the data distribution, respectively.
3 GENERATIVE MODEL FOR DISENTANGLED MIXTURE OF MECHANISMS
3.1 PRELIMINARIES
Many generative models (Kingma Welling 2013 Goodfellow al. [014 Locatello et al., 2020)
assume that the data is generated through two-step procedure: (1) A sample z is drawn from an unobserved continuous prior distribution p(z), which is usually assumed to be A f). (2) An observational data sample x is drawn from unknown conditional distribution p(æ x) or is generated by a function f Z - X. Figure ((a) and [(b) further visualize this procedure from a causal mechanisms' perspective.
Generative Adversarial Network Generative Adversarial Network (GAN) learns the data gener- ation process via a two-player minmax game between a generator 2 + X' and discriminator D X - y: ming max n V(G, D) = (e)) [(z)] + Rg<p(()) [log(1 (=)()))) The goal is to minimize the divergence between the generated data and the real data. Recent research has shown that the Wasserstein distance (Arjovsky et al.) 2017) is good choice of divergence in practice. Thus, we use Wasserstein GAN with gradient penalty (WGAN-GP) (Gulrajani et al. 2017) our generative model. GAN less popular in unsupervised learning of disentangled representa- tions as is difficult to approximate the posterior p( x). However, recent work (Mukherjee et al.) 2019) showed its advantages in self-supervised deep clustering, which is highly relevant disentan- gling the data generation process. Thus, in addition to the generator network and the discriminator network of conventional GAN, we add encoder network to perform self-supervision, which is discussed in section 3.3
3.2 THE UNCONVENTIONAL MIXTURE PRIOR AND STRUCTURAL LATENT SPACE
Unlike most generative models which assume p(z) = I[=_ p(2;), we use an unconventional mix- ture prior:
3

---

Under review as a conference paper at ICLR 2021
E sc
I
P(ZMa 120)
N
Ph. (k((2s))
p(z) = -p(zs)
=p(zs)
Z
7.
k=
p(zMa (20) =
I)
FC =k ootherwise.
where Z = Σk S210 pk (P (ZM)) p(zs) I p(zs,), p(zm, II p(zas,,), and aM [ZM, 2My: > 2s represents the causes for the shared mechanisms and 2M,, represents the causes for the bth generative mechanisms. We convert the sum of the conditional joint distribution to our unconventional mixture distribution by defining P1 (zM) = [W(0, O). O), P2(ZM) = [0, (). ..., 0] and so on. Figure I(c) and [(d] further visualize the graphical model and the assumed data generation process with the unconventional mixture prior.
Our prior disentangles the mixture components by encoding the components in orthogonal latent subspaces. The mixture components of Gaussian mixture prior, however, are shown to entangle in the latent space (Mukherjee al. 2019) because
perfectly fine for mixture components NNo, oğ) and [(,,, oi) to have overlap. Such an overlap entangles the causes of different mechanisms. For the shared mechanisms, we encode all causes, such as rotation and brightness, in 2ş and do not further disentangle them because such disentanglement is proved to be impossible without supervision (Locatello et al.) 2019). Although we do not disentangle each shared cause, usually does not hurt the performance of downstream tasks because most of the predictive tasks associated with the generative mechanisms. Thus, we only need to disentangle the shared mecha- nisms from the generative mechanisms. In section 4 we show such disentanglement guaranteed, provably.
Besides the Gaussian mixture prior, we also discuss the advantage of unconventional mixture prior over the existing disentangled prior (Higgins et al. 2017] Locatello et al. 2020). To see this, consider that by definition, each 24 disentangled prior z, which admits a density p(z)
I p(=),, represents one type of variations in disentangled representations. With the disentangled prior, can generate data through the generative model by setting each 2; an arbitrary value. If we use each z; to represent each cause, the prior would allow the generative mechanism to generate data by arbitrarily combining the causes. This contradicts our assumption that each cluster in the data distribution has its distinct cause and none of the data is generated by the combined causes. To resolve this conflict, we need either create a new dataset or adopt the unconventional mixture prior.
3.3 THE SELF-SUPERVISED MODEI.
Our model is composed of generator network G 2 - X', discriminator network D X' ), and an auxiliary encoder network X + 2. The auxiliary encoder is a basic deterministic encoder that tries to invert the generator. The self-supervision means predicting which generative mechanism does the generated data come from using the auxiliary encoder network E. Such self- supervision would encourage the data samples from the same generative mechanism to be similar to each other and vice versa. Thus, we can disentangle generated data from different mechanisms in the sense that they are separable and their sources predictable. Once the data samples disentangled into clusters, the generative mechanisms are disentangled as well. The loss function of method is:
L = W(P. + Bc H(z E(G())c) + B, Ex)(()))) G(E(x))11
(2)
+ BMr B | E(G(z)) M 113 + 3sE z8 - E(G(z))sl13
where Py denotes the distribution covered by the generator network G, P, denotes the real data dis- tribution, ((P, [PP, denotes the Wasserstein distance between two distributions, ze is a categorical variable represents the index of the generative mechanisms, zM and øy are the same as is defined in section 3.2 and H denotes the cross-entropy loss. We have tried replace the
[zc. ZM, øs],
4

---

Under review as a conference paper at ICLR 2021
Euclidean distance with Cosine distance in the loss function, but both methods yield similar results. As the readers may have noticed, we add #c to & in our implementation. The purpose is to eliminate the ambiguity when 2M z 0 and the generator does not know which generative mechanism to use (Antoran & Miguel 2019). As we are only fixing a corner case, we leave it as an implementtiion detail and keep the unconventional mixture prior definition unchanged.
The loss function above can be decomposed into four parts: 1) W(P, is the WGAN loss. 2) Ex) H(c,, E) is the self-supervision loss that enforces the disentanglement between generated data. 3) Ex- [ G(E(2)) is the forward cycle-consistent loss, which enforces = G(E(a))). The encoder takes the generated data from the generator input. However, the generated data distribution may differ from the real data distribution. Thus, certain data in the real distribution may become cattof-distribution (OOD) for the encoder. Such distribution diver- gence hurts the performance of downstream tasks. The forward cycle-consistency mitigates the distribution divergence by encouraging the ICM model cover the whole real distribution. 4) Rzz<p(>) zM E(G(z)) M 112 + B5l Bsppz=p(s)) - F(G(=))sslī is the backward cycle-consistent loss, which enforces 2 xe E(G(=)). The backward cycle-consistency prevents the generator from discarding certain causes in the generation process.
4 IDENTIFIABLE DISENTANGLEMENT
Identifiability is crucial for disentangling the causal mechanisms. We can construct infinitely many generative models that have the same marginal distribution (Locatello et al. 2019 Khemakhem al. 2020). Without identifiability, any one of these models could be the true causal generative model for the data, and the right model cannot be identified given only the observational data (Pe- ters et al., 2017). Prior work (Locatello al.) (2019) shows that we can not identify the disentangled generative model using the marginal distribution. On learning the causal mechanisms, we are in- terested in identifying the right generative model that disentangles the causes in the latent space. Specifically, each ΣM₃ and #ş should represent distinct causes when is paired with the right gen- crative model g. Before beginning the proof, we introduce the impossible result in unsupervised learning of disentangled representations, where they aim to let each latent explanatory factor 2; carries a distinct type of variations.
Theorem 1. (Locatello et al. 2019) For d > 1, let 2 ~ P denote any distribution which admits density p(z) II:= p( 2:). Then, there exists an infinite family of bijective functions h. supp(z) supp(z) such that (h.(w) entangled) and P(z S u) P(h(z) < u) for all a E ssup((2) (i.e., they have the same marginal distribution).
Su, + 0 almost everywhere for all i and j (i.e., z and h(z) are completely
Theorem Findicatess that we can not identify the disentangled model because we can not tell whether the latent explanatory factors are entangled by h or not from the marginal distribution. Compared to the parameter-space identifiability in nonlinear ICA (Khemakhem et al., (020), identifying a disentangled model is more challenging because even if we identify the right distribution, the function h may still entangle the latent explanatory factors. Another difference that the ground- truth distribution of the latent variables in disentangled representations is assumed to be isotropic Gaussian, which diminishes the importance of the parameter-space identifiability. Therefore, we formulate our identifiability in the function space:
Definition 1. Let ~h be an equivalence relation on g + X as follows:
g m g + (
(()),Vac Vo E X
(3)
where h. E + Z is smooth invertable function.
Definition 2. We say that g Z + X is identifiable up ~ (or ~ -oonnnnndttiibble if:
(x(x)) Pg(x)) ~
(4)
Under the definitions and 2 the following lemma shows that the generative model is ~M identifiable:
5

---

Under review as a conference paper at ICLR 2021
5
55555555
9 9
1999999
99
7 7
6
C
5
3
3.3 33333333
3 3
2
3
2 2
0
(a) Generative Mechanisms Sub-
(b) Shared Mechanism Subspace,
(c) Shared Mechanism Subspace. Dim #1, Stroke Thickness and Width
spaces, Distinct Variation for Each
Dim #0, Width and Rotation
Subspace/Row
Figure 2: Latent Space Traversal of ICM model on MNIST
Lemma 2.
Let G be the space of smooth invertable functions with smooth inverse (i.e., diffeomor- Z a smooth invertable function. Then, any function g E g
phism) that map to X, and h can be represented as 9 = 9" h, where 9" Z - X is the assumed ground-truth disentangled generative model in the function space G. Formally, we have g ~ g¹ Vg E 9 and the model g is -asidentifiable
The proof is in Appendix A The following theorem further shows that the function h in ~ disen- tangles the groups of causes for different mechanisms with the unconventional mixture prior.
Theorem 3. Let p(z) denote the uncomenniiona mixture prior. We use M to denote the manifold where the ground-truth latent variable z lies on. Let k E 21. NN y be the index of generative mechanisms M. We assume each 2M, lies on Mas, and zş lies on Ms- We let & h(2) which lies on M. g" is the ground- -rund disentangled model. Then, if there exists an smooth invertable function h Z + Z such that g g" h maps Z to X, we have h maps each Ma. disjoint sub-manifold MMs and maps Mg to Ms. which is disjoint from all Mas.
We leave the proof of Theorem 3 in Appendix B. By slight abuse of notation, we define the disentangled as hp. Then, the disentangled ICM model is ~nD -identifiable.
5 EXPERIMENT
For various downstream tasks, this section, we first examine the robustness of our model against interventions. As we only apply interventions on the root node, such interventions visualize the result by latent space traversal. We then evaluate the robustness of our model under the covariant shift by measuring the downstream ML model performance. The covariate shift the change in the distribution of the covariates, that is, the independent variables. More specifically, we consider disentangled single covariant shifts (e.g. rotation) and entangled multiple covariant shifts (e.g. brightness and width). Unlike previous works (Arjovsky et al. 2019 Locatello et al.) 2020), which rely on manual annotation, we propose new method to extract the latent covariants from the datasets and use them to create experiments. Finally, we measure the robustness of our model against uniform noise and Gaussian noise by measuring the downstream ML model performance. Through experiments,
the encoder network E to extract data representations. In
equivalent to conditional generation. We
use: 1) Two datasets, which are MNIST and FashionMNIST. 2) Three competitors, which are VAE (Kingma & Welling 2013), B-VAE (Higgins et al., 2017), and Ada- GVAE (Locatello et al. 2020). 3) Four covariant shifts, which are implicitly carried by the datasets.
5.1 INTERVENTIONAL ROBUSTNESS AND LATENT SPACE TRAVERSAL
We show the latent space traversal of the causal mechanisms on MNIST in Figure 2 In this exper- iment, each generative mechanism M₁ has an one dimensional MM₁- The shared mechanisms has a four dimensional 2s. For the th generative mechanism subspace traversal, we set the 2o to and manually change ZM, During this process, all the irrelevant dimensions are set to 0. As Figure

---

Under review as a conference paper at ICLR 2021
Table 1: Average Accuracy of Downstream Classifiers under Different Shift Strength
MODEL
MNIST (T)
MNIST (W&R)
MNIST (R)
FASHIONMNIST (D&W)
ICM
74.63% %
56.91
43.71%
42.92 %
VAE
55.91% %
41.92%
39.90%
37.08%
B 3-VAE
73.34%
46.33%
46.05%
41.60%
ADA-GVAE
68.09%
47.16%
47.58% %
40.85%
Table Normalized Accuracy Variations of Downstream Classifiers under Different Shift Strength
MODEL
MNIST (T)
MNIST (W&R)
MNIST (R)
FASHIONMNIST (D& W)
ICM
2.43%
3.70%
7.10%
5.02 %
VAE
4.32%
6.30%
7.33%
9.02%
8-VAE
2.71%
4.98%
7.80%
7.83%
ADA-GVAE
2.96%
4.90%
7.31%
7.98%
Table Shift Distance Needed for 10% Relative Accuracy Drop under Covariant Shift
MODEL
MNIST (T)
MNIST (W&R)
MNIST (R)
FASHIONMNIST (D& W)
ICM
1.4
0.8
0.6
0.4
VAE
0.6
0.4
0.4
0.2
3 3-VAE
1.1
0.5
0.4
0.3
ADA-GVAE
1.1
0.5
0.4
0.3
2(a) shows, each row represents a traversal of an generative mechanism subspace. For the shared mechanism, first let ze 0. Then, we do traversal for dimension in the shared mechanism subspace. After this traversal, we increase 2c by and repeat the procedure. Figures [ and 2(c) each figure represents a traversal for dimension in the shared mechanism subspace and each row in the same figure represents a traversal of a fixed dimension with different zc. Due to the limited space, we report more visualization results in the Appendix.
We say our ICM model is robust against intervention in the sense that no matter how we change MMa and 2s while keeping zc and z M. fixed, the generator does not generate data that does not belong to mechanism Mr (e.g. digit 0 does not change to other digits during the traversal). Furthermore, we show that the traversal of each generative mechanism yields a distinct, mechanisms-speiine type of variation. The traversal of the shared mechanism yields the same type of variation. Compared with conditional generative models (Kingma al. 2014: Klys et al. 2018), our work do not pose any requirement on the label during training.
Finally, we note that the number of generative mechanisms chosen to be larger than the number of classes, which means ground-trun mechanism may be split into two estimated mechanisms. Previous works (Parascandolo al. 2018 Locatello et al., (2018b) also adopt the same setting and do not find it triggers any issue in the following experiment.
5.2 ROBUSTNESS UNDER COVARIANT SHIFT
To quantitatively measure the robustness under the covariant shift, first partition the dataset using the learned representations as the covariant. For convince, we use the encoder of the NVAE model (Antoran & Miguel 2019) to extract the representations. Using the shared mechanism in our ICM model would yield the same result. More specifically, we partition the dataset into subsets {o E X,
E(x); [Clowery Cupper)) using the predicted value in dimension i. For the training set, we set [Clower, Cupper) to [0, x0). For each test set, we set [Ctower, Cupper) to [-0.1, 0). 1-0.2,-0000 -0.1), .... [-3.0, -2.9). We use the Crreei Ctest upper to represent the strength of the covariant shift, which is also called the shift distance in Table In this experiment, consider

---

Under review as a conference paper at ICLR 2021
0.75
VAE
teta-ViE
0.2
Deta VAG
VAE
0.1
bela-VAE Ada-OVAE
0.50
Gras
0.50
Adl GUAS
0.5
0.0
0.25
-0.1
0.00
0
2
2
(a)
Accuracy under (b) Accuracy Differences (c) Accuracy under Rota- (d) Accuracy Differences between ICM and Others
Witth&Rotation Shift
between ICM and Others tion Shift
Figure 3: Accuracy and Accuracy Differences of Downstream Classifiers under covariant Shift on MNIST dataset. If ICM outperforms another method, the accuracy difference is positive.
Table 4: Accuracy of Downstream Classifiers under Noise No, I)
MODEL
MNIST (T)
MNIST (W&R)
MNIST (R)
FASHIONMNIST (D&W) W)
ICM
52.64% 2.03 %
39.36% +: 2.58%
53.30% % + 2.08
46.69 +: 1.75% %
VAE
34.13% 1.48%
32.40% + 2.07%
27.14% 2.32%
33.19% 2.52%
B D-VAE
39.98% 2.74%
39.93% = 2.23%
37.35% + 3.16%
34.79% + 2.86%
ADA-GVAE
46.41% + 1.66%
43.10% + 0.86 %
46.25% + 0.77%
43.92% + 1.61%
stroke thickness (T), width and rotation (W&R), rotation (R) and, darkness and width (D&W) as covariants. Figure fin the Appendix further visualizes these covariants.
gradient boosting classifier (GBT) from Scikit-learn with default parameters as the downstream ML. model, which the same as previous works (Locatello et al. 2019 2020).
Then,
We train the classifier with 1000 samples, which is sufficient for producing good accuracy, and test its accuracy on a sequence of test sets. For each experiment, we evaluate the classifier 10 times and collect the average accuracy well the accuracy variations. Tables and 2 show the average of average accuracy and the average of normalized variation of the classifier using each model's 's representation as input under different covariant shift strengths, respectively. Our ICM model achieves the best average accuracy across all the experiments except for MNIST (R). For the normalized accuracy variations, which the standard deviation over the accuracy, our ICM model always achieves the lowest variation. It means that the ICM model is less sensitive to the choice of training samples as the GBT classifier does not contribute many variations.
We further investigate the issue of our model under MNIST (R) covariant shift. Figure 3 shows the accuracy changes as the shift strength increases. both experiments, our method shows more robustness when the shift strength is low as its accuracy decreases slower and its advantage grows bigger. However, after a threshold, our method begins to lose its advantage. There are two possible reasons: 1) The test data shifts too far from the training data and the base generative model can not generalize to the test sets. After the test set shifts too far away, none of the methods perform well. It's hard to conclude that model with 40% accuracy is better than model with 35% accuracy. 2) The test set there contains too few samples, which is just around tens or hundreds, and makes the evaluation inaccurate. As we can see from Figure 3 the larger the shift, the bigger the accuracy variations. To eliminate this interference, we instead measure how much distance the covariant shift needs to go decrease the accuracy by a percentage relatively. Such evaluation will put more weight on the test sets which have more samples and yield reasonable accuracy. Table 3 and Tables 9 10 in Appendix DD2 show our method can tolerate more covariant shift before the accuracy relatively drops by 10%, 20% and 40%.
5.3 ROBUSTNESS AGAINST NOISE
We evaluate the robustness of our ICM model under noise using similar setting as the previous section. In this experiment, we use the train the GBT with 1000 samples and test the GBT accuracy on whole train set. Table 3.3 shows our ICM model is generally more robust against Gaussian noise
8

---

Under review as a conference paper at ICLR 2021
Table Accuracy of Downstream Classifiers under Noise + NN 1) on MNIST (W&R)
MODEL
0.25
c 0.5
c= 0.75
ICM
85.04% = 0.57 %
69.59% 2.05% %
51.61% + 4.68%
VAE
74.97% 1 0.66%
58.57% + 1.44%
42.90% 1 1.42%
8-VAE
80.08% 1 0.77%
64.94% 1 2.53%
49.51% 1 2.02%
ADA-GVAE
81.64% 1.29%
68.38% + 1.76%
53.90% + 1.68%
(O, I). Table 5.3
further shows that although there methods that perform slightly better than our method on MNIST (W&R), our method still performs better before the noise becomes too large and decrease the accuracy by high percentage. We further show the results under uniform noise Uniform(O, 1) in Appendix D.3
CONCLUSION
In this paper, we present self-supervised method to learn the independent causal mechanisms. We show the presents of generative mechanisms and shared mechanisms. Our model can learn these two types of mechanisms through an unconventional mixture prior. Furthermore, we outline the suffi- cient conditions for theoretically identifying the mechanisms from observational data. Experiments show that our ICM model is generally more robust against interventions, covariant shifts, and noise.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Javier Antoran and Antonio Miguel. Disentangling and learning robust representations with natural clustering. 20/9 18th IEEE International Conference On Machine Learning And Applications (ICMLA), pp. 694-699. IEEE, 2019.
Martin Arjovsky, Soumith Chintala, and Léon Bottou. arXiv:1701.07875. 2017.
Wasserstein
gan. arXiv preprint
Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprint www.:: 2019.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 178811179--1888 2013.
Yoshua Bengio, Tristan Deleu, Nasim Rahaman, Rosemary Ke, Sébastien Lachapelle, Olexa Bila- niuk, Anirudh Goyal, and Christopher Pal. A meta- ceets-trannfer objective for learning to disentangle causal mechanisms. arXiv preprint arXiv: 1991, 2019.
Stephan Bongers, Patrick Forré, Jonas Peters, Bernhard Schölkopf, and Joris M Mooij. Foundations of structural causal models with cycles and latent variables. arXiv preprint 2016.
lan Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley. Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in Neural Infor- mation Processing Systems 27, pp. 2672-2680. Curran Associates, Inc., 2014.
Ishaan Guirajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. Im- proved training of wasserstein gans. Advances in neural information processing systems, pp- 5767-5777, 2017.
I. Higgins, Loic Matthey, A. Pal, C. Burgess, Xavier Glorot, M. Botvinick, S. Mohamed, and Alexander Lerehner. beta-vae: Learning basic visual concepts with constrained variational framework. In International Conference on Learning Representations, 2017.
Aapo Hyvärinen and Erkki Oja. Independent component analysis: algorithms and applications.
Neural networks, 13(4-)):411-430 2000.
Aapo Hyvarinen, Hiroaki Sasaki, and Richard Turner. Nonlinear ica using auxiliary variables and generalized contrastive learning, In The 22nd International Conference Artificial Intelligence and Statistics, pp. 859-868, 2019.
Ilyes Khemakhem, Diederik Kingma, Ricardo Monti, and Aapo Hyvarinen. Variational autoen- coders and nonlinear ica: A unifying framework. In International Conference on Artificial Intel- ligence and Statistics, pp. 2207-2217, 2020.
Diederik P Kingma and Max Welling. 1, 2013.
Auto-encoding variational bayes.
arXiv preprint
Durk P Kingma, Shakir Mohamed, Danilo Jimenez Rezende, and Max Welling, Semi-superve learning with deep generative models. In Advances in neural information processing systems, pp- 3581-3589, 2014.
Jack Klys, Jake Snell, and Richard Zemel. Learning latent subspaces variational autoencoders. In Advances in Neural Information Processing Systems, pp.
2018.
Abhishek Kumar, Prasanna Sattigeri, and Avinash Balakrishnan. Variational inference disentan-
gled latent concepts from unlabeled observations. arXiv preprint arXiv:1711.00848 2017.
Francesco Locatello, Damien Vincent, Ilya Tolstikhin, Gunnar Ratsch, Sylvain Gelly, and Bernhard Scholkopf. Clustering meets implicit generative models. 20.
10

---

Under review as a conference paper at ICLR 2021
Francesco Locatello, Damien Vincent, Ilya Tolstikhin, Gunnar Rätsch, Sylvain Gelly, and Bern- hard Schölkopf. Competitive training mixtures of independent deep generative models. arXiv preprint 11XY::18044111300 2018b.
Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Raetsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentangled representations. International Conference on Machine Learning, pp. 4114 4124, 2019.
Francesco Locatello, Ben Poole, Gunnar Rätsch, Bernhard Schölkopf, Olivier Bachem, and Michael Tschannen. Weakly-supervised disentanglement without compromises. International Conference on Machine Learning. 2020.
Joris Mooij and Tom Heskes. Cyclic causal discovery from continuous equilibrium data. arXiv preprint arXiv:1309.666 6849,,684928880 2013.
Joris M Mooij, Jonas Peters, Dominik Janzing, Jakob Zscheischler, and Bernhard Schölkopf. Dis- tinguishing cause from effect using observational data: methods and benchmarks. The Journal of Machine Learning Research, 17(1): 103 1204, 2016.
Sudipto Mukherjee, Himanshu Asnani, Eugene Lin, and Sreeram Kannan. Clustergan: Latent space clustering in generative adversarial networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 4610 6610-4617, 2019.
Giambattista Parascandolo, Niki Kilbertus, Mateo Rojas-Carulla, and Bernhard Schölkopf. Learning independent causal mechanisms. In International Conference on Machine Learning, pp. 4036- 4044. PMLR, 2018
Judea Pearl al. Models, reasoning and inference. Cambridge, UK: ConbrridgeUniiversittPPesssss 2000.
Jonas Peters, Dominik Janzing, and Bernhard Schölkopf. Elements of causal inference. The MIT Press, 2017.
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. In Advances in neural information processing systems, pp. 2234-2242. Curran Associates, Inc., 2016.
Bernhard Schölkopf, Dominik Janzing, Jonas Peters, Eleni Sgouritsa, Kun Zhang, and Joris Mooij. On causal and anticausal learning. arXiv preprint 1206:v:2006.477 6471, 2012.
Rui Shu, Yining Chen, Abhishck Kumar, Stefano Ermon, and Ben Poole. Weakly supervised disen- tanglement with guarantees. arXiv preprint arXiv: 111.071910 2019.
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv:1312.61999 2013.
Jun- Yan Zhu Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired amage-to-image translation using cycle-consistent adversarial networks. Proceedings of the IEEE international conference on computer vision, pp. 2223-2232 2017.


Reviews:
[{"review_id": "xPx_ThysrHk", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Authors propose a new method to learn independent causal mechanisms from data, which is achieved by designing a mixture prior consisting of shared mechanisms and Independent Causal Mechanisms (ICM) conditioned mechanisms. Specific orthogonal structure or separability is imposed on the ICM-conditioned mechanisms. A single generative model is used with such a mixture prior to learn mechanisms from data. Authors also prove the subspace of the mechanisms are separated and thus identifiable. Experiments on MNIST dataset demonstrate that the method is able to learn independent mechanisms and it improves the robustness against intervention, co-variant shift and noise. \n\nDetailed Comments:\n1. I am not quite clear about the meaning of ICM-conditioned mechanisms. Could authors elaborate it a little bit what “conditioned” means.\n\n2. The Independent Causal Mechanisms first proposed by (Scholkopf, 2019) is referring to the independence of each sub-modules in a whole generating process, e.g. for a cause-effect pair (x, y), ICM usually means that the process of generating the cause which is p(x) is independent of the process that maps the cause to the effect which is p(y|x). However, in this paper, authors are considering a very different setting where they argue that for different data groups, the generative mechanisms are different. I am not very sure they are the same thing. Could authors explain the connection between the two?\n\n3. Authors use single generative model which is claimed to be one of the advantage of their method. However, in Fig 1(d) they use M_N, …, M_0 to indicate that for different data groups, the generating mechanisms are different. Could authors explain this discrepancy?\n\n4. It appears to me that the method can only work in the scenario where we know beforehand the number of different generating mechanisms in the data. However, for real applications it is difficult to have such prior information. Can authors also comment about the applicability of the method?\n\n5. It appears to me that the proposed method is incremental to existing disentangled representation learning methods except for the mixture prior.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "myg1Q3SFo9C", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Summary**\nThe paper considers the task of coarsely-disentangling a data-generating process into independent and shared modules. In more detail, it is assumed that each observation in a given dataset was generated by exactly one out of a set of independent generative processes referred to as independent causal mechanisms (ICM) in combination with a global mechanism that is shared across modules. The paper proposed the use of a mixture prior to model separate ICMs within a single architecture/generative model, chosen to be a GAN (Wasserstein-GAN with gradient penalty). Learning is performed by combining the GAN loss with self-supervision by using a separate encoder and adding loss terms resemblant of cycle-consistency-losses. The paper claims to prove a notion of identifiability of the coarse-grained modules in the sense of separation into disjoint manifolds and conducts some experiments on (variations of) MNIST and Fashion-MNIST, comparing their method against vanilla and disentangled VAE variants on some downstream tasks.\n\n**Pros**\n- the paper tackles a very interesting and highly-relevant topic (disentanglement with non-factorising latent space and connections to causality)\n- the coarse-grained view and the separation into independent and shared mechanisms seems like a useful abstraction and (to the best of my knowledge) has not been investigated before in this form\n- the authors support their proposed method with some theoretical analysis\n- the latent traversal in Figure 2 and some of the quantitative results look promising \n\n**Cons**\n- even though I am very familiar with the topic of the submission, I found the paper very hard to follow\n- many of the statements relating to causality and ICMs are inaccurate, misleading, or wrong (see detailed comments below)\n- the notion of identifiability used in the paper is very unconventional and not consistent with prior work\n- the assumed generative model is never fully specified (in particular, the mapping from latents to observations and the role of the mechanisms is not described in the text or given as formula) and the mixture prior is inconsistent with the graphical model given in Fig.1\n- some of the cited works are misrepresented or wrongly protrayed (see detailed comments)\n\n\n**Evaluation**\nI think the paper tackles a very important problem and presents some interesting ideas. However, the paper contains too many errors, unclear parts, and inconsistencies in the current form which makes it very difficult to gain a clear picture of the proposed method and to assess its correctness. Significant improvements in terms of clarity and exposition are needed before publication and I therefore recommend rejection at the current stage.\n\n**Detailed Comments and Questions**\n- The ICM principle states that \"the data generating process is composed of independent and autonomous modules that do not inform of influence each other\" (Peters et al., 2017). I believe you should cite this. Note that this is different from the notion of isolated, as different ICMs may feed into each other. In fact this is how ICMs are conventionally understood in causality: each conditional distribution of a variable given its causal parents $P(X_i|PA_{X_i})$ is interpreted as an ICM, and together they form the data generating process for the joint $P(X_1, ..., X_n)=\\prod_{i=1}^n P(X_i|PA_{X_i})$. ICMs are thus traditionally understood as independent modules that can be (re-)combined.\n- I do not understand why the term \"ICM-conditioned mechanism\" is used: what is an \"independent causal mechanism-conditioned mechanism\"? \n- Related Work: you mention three aspects but only two are given.\n- Functional Causal Models: there are several key differences between FCMs and Bayesian networks (BN), so I think the presentation here is misguided. Firstly, BNs have no causal connotation to them but are only a way to represent a particular factorisation of a probability distribution. Causal graphical models or causal BNs on the other hand are endowed with a notion of intervention and are thus closer to FCMs. Perhaps this is what you intended to say? Moreover, it is unusual to refer to $P(X|PA_X)$ as a posterior---this is precisely what an ICM refers to and other common names include (causal) conditional or Markov kernel. \n- the entire discussion of causality and FCMs misses the notion of intervention and counterfactual which is crucial for defining causal concepts\n- The description of FCMs is incorrect: The deterministic relationships (structural equations) in SCMs are a set of assignments of the form $X_i:=f_i(PA_{X_i}, U_i)$ where $U_i$ are the unobserved variables and $f_i$ are indeed deterministic. \n- I have never heard or read about this and would like to ask about the source of this characterisation: \"If each function in FCM represents an autonomous mechanism, such FCM is called a structural model. Moreover, if each mechanism only determines the value of one and only one variable, such a structural model is called a structural causal model (SCM).\" ? This seems strange to me.\n- (Cai et al., 2019; Monti et al., 2020) are cited incorrectly: actually these aim to discover the graph which is different from learning an SCM as the former supports interventional reasoning, while the latter supports counterfactuals. \n- Figure 1: the rectangles M_i are not described in the caption.\n- ICA: the description of recent advances in ICA is incorrect: it refers to \"requiring a relatively large number of independent components\" when this should read \"... large number of environments\" or more generally values of the auxiliary variable which renders sources conditionally independent\n- the specification of the mixture prior at the end of page 3 seems incorrect as it does not integrate to 1. Perhaps you meant to include mixture weights? However, this is also inconsistent with Fig.1: (c) would suggest a prior which factorises as $p(z)=p(z_C)p(z_S)\\prod_{i=0}^N p(z_{M_i}|z_C)$, while (d) suggests $p(z)=p(z_S)\\prod_{i=0}^N p(z_{M_i})$ both of which are different from the text. As I infer it from reading between the lines, $z_C$ is a categorical variable that switches between different mechanisms, and $z_M$, $z_S$ are both inputs to the generative process. Why the need for the many $z_M$'s? Would a simple mixture distribution on $z_M$ with weights and parameters depending on $z_C$ not do the job? If not, can you explain why?\n- How is the isolation constraint implemented, i.e., how does it translate mathematically? Can you please specify the full generative process $p(z, x)$?\n- the last paragraph in 3.2 is very hard to follow\n- 3.3 suddenly mentions the encoder, but no encoder has been introduced up to this point\n- can you justify the loss function in more detail?\n- Wasserstain --> Wasserstein\n- 4. The stated version of identifiability does not seem to make sense to me. It is almost always possible to learn the ground truth model, the question of identifiability is whether we are *guaranteed* to learn the right model. The cited work of Khemakem et al (2020) provides a nice and intuitive definition of identifiability in terms of equivalence classes in parameter space that seems more principled than the notion used here. \n- Thm.3 This is the first time M_k is mentioned \n- I am not sure what the point of the paragraph at the end of page 5 is? This does not really provide any intuition on the Theorem or its proof. Moreover, you assume $h:Z\\rightarrow Z$ and then have $h(z)\\not\\in Z$ which seems to contradict each other. Please clarify.\n- the paper refers to \"co-variant shift\" throughout; to the best of my knowledge \"covariate shift\" is the accepted terminology. is the different notation intentional, and if so can you clarify the difference?\n- some experiments are not described in sufficient detail (e.g., what is being predicted in 5.2? why are there no error bars in Tables 1 or 3)\n- the paper would benefit considerably from some professional proofreading and grammar checking (though this is not a key factor in my evaluation)\n\n\n\nPOST-REBUTTAL UPDATE:\nI thank the authors for the detailed response. Based on the proposed changes I will slightly increase my score, but I still believe the paper needs additional work before meriting publication.", "extra_scores": {}, "extra_text": {}}, {"review_id": "B1BGt-XFWOJ", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "I hardly agree with the assumption made in the graphical model c) and d) that each cluster of data has a different generating mechanism from others. First, it is named the ICM-conditional mechanism in the paper, which is very confusing since it contradicts the definition in the literature of ICM [1]. The ICM, in the literature, originally describes the independent autonomous for generating a sequence of variables in the causal graph, e.g., p(v_1,...,v_k) = \\Pi_k p(v_k | Pa(k)), based on the assumption that the exogenous variables are independent with each other. Besides, the difference in terms of variation for digits 1 and 2, in the example made for supporting this assumption, should be encoded in the latent variables z, rather than the generating process. Since the latent variable z is defined to describe the high-level abstractions or concepts, including but not limited to the thickness, width, length in the example of MNIST. Therefore, the graphical model should be y -> z -> x, which is aligned with existing literature in nonlinear ICA.  Besides, even if the assumption is correct, the posterior distribution of p(z_Mk | x_Mk) should be varied across k, which hence cannot be learned by the same encoder E. \n\nAlthough the experimental results yield good results, it can not be interpreted in this way. The high-level spirit of this paper is based on an invalid assumption, which makes the promising results in experimental parts non-important. \n\n\n[1] Schölkopf, Bernhard. \"Causality for machine learning.\" arXiv preprint arXiv:1911.10500 (2019).", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes to learn representations in an unsupervised manner using a generative model in which observations are generated by combining independent causal mechanisms (ICMs), in combination with a global mechanism. The authors introduce an unconventional mixture prior for the shared and independent components of the representation and train an encoder, discriminator and generator using a Wasserstein GAN with additional terms that enforce consistency in the data and latent space. Experiments consider variations of MNIST and Fashion-MNIST and perform comparisons against a standard VAE, a β-VAE, and the Ada-GVAE. 

Reviewers are broadly in agreement that this submission is not ready for publication in its current form. R4 in particular has left very detailed comments regarding clarity. The authors were able to in part address these comments, and R4 raised their score in response. That said, from a read of the manuscript in its latest form, the metareviewer (who is very familiar with literature on disentangled representations) is inclined to agree with the reviewers that this is work that has value, but is very difficult to follow in its current form. The metareviewer would like to suggest that the authors regroup, think carefully about how to improve clarity (in addition to addressing concrete points raised in reviews) and resubmit to a different venue. 
Decision: Reject