On the interventional consistency of autoencoders
Authors: Giulia Lanzillotta
Felix Leeb
Stefan Bauer
Bernhard Schölkopf
Venue: ICLR 2022 Submitted (2022)
Keywords: causal representation learning
disentanglement
autoencoders

Abstract: Autoencoders have played a crucial role in the field of representation learning since its inception, proving to be a flexible learning scheme able to accommodate various notions of optimality of the representation. The now established idea of disentanglement and the recently popular perspective of causality in representation learning identify modularity and robustness to be essential characteristics of the optimal representation. In this work, we show that the current conceptual tools available to assess the quality of the representation against these criteria (e.g. latent traversals or disentanglement metrics)  are inadequate. In this regard, we introduce the notion of \emph{interventional consistency} of a representation and argue that it is a desirable property of any disentangled representation. We develop a general training scheme for autoencoders that takes into account interventional consistency in the optimality condition. We present empirical evidence toward the validity of the approach on three different autoencoders, namely standard autoencoders (AE), variational autoencoders (VAE) and structural autoencoders (SAE).
Another key finding in this work is that differentiating between information and structure in the latent space of autoencoders can increase the modularity and interpretability of the resulting representation.

Under review as a conference paper at ICLR 2021
ON THE INTERVENTIONAL Y OF AUTOEN- CODERS
Anonymous authors Paper under double-blind review
ABSTRACT
Autoencoders have played a crucial role in the field of representation learning since its inception, proving be a flexible learning scheme able to accommo- date various notions of optimality of the representation. The now established idea of disentanglement and the recently popular perspective of causality represen- tation learning identify modularity and robustness to be essential characteristics of the optimal representation. In this work, we show that the current concep- tual tools available to assess the quality of the representation against these criteria (e.g. latent traversals or disentanglement metrics) are inadequate. In this regard, we introduce the notion of interventional consistency of representation and ar- gue that it is a desirable property of any disentangled representation. We develop general training scheme for autoencoders that takes into account interventional consistency in the optimality condition. We present empirical evidence toward the validity of the approach on three different autoencoders, namely standard autoen- coders (AE), variational autoencoders (VAE) and structural autoencoders (SAE). Another key finding this work is that differentiating between information and structure in the latent space of autoencoders can increase the modularity and in- terpretability of the resulting representation.
INTRODUCTION
Representation learning is the problem of finding a low dimensional description of the data. The characteristics of 'good representation' have long since been a matter of debate, often depending on the context in which the representation has to be employed. Multiple works (cf. Bengio al. (2013), Schölkopf et al. (202T) identify modularity, robustness to distribution shifts and interpretability as erucial features of 'goodd representation', motivating the quest toward disentanglement (Bachman let al. 2019). Locatello et al. (2020), Ridgeway & Mozer (2018) and causal representations (Suter al. (2019). Träuble et al. (021)). The insight shared by both these fields is that complex world phenomena arise from the rrich interaction of many sources', and thus enable compact descriptions terms of the basic components participating in their generative process. More formally, the hy- pothesis is that there exist a set of semantically meaningful variables S and an arbitrarily non-linear function G such that X G(SI; ..., Sa), where X are the observations.
Autoencoders have played crucial role in the field of representation learning since its inception. Their success largely due to their simplicity and surprising effectiveness. Mathematically autoen- coders can be represented as the tuple (E I + R²,, D IR" + Rd). Capacity constraints (e.g. n < d) force the latent space to prioritize certain information in the input, thus yielding a useful representation. Additional requirements may be imposed either structurally tion. Intuitively, consistency means that an input generated by the decoder can be mapped back to the point in the latent space that produced it. The central idea of Cemgil et al. (2020) is making the encoder and the decoder consistent both on the training data P(X) and on the auxiliary observations generated by the decoder P(X').
through regularisa-
Our contribution. recent paper Locatello al. (2019) prove that 'the unsupervised learning of disentangled represen- tations is fundamentally impossible without inductive biases on both the models and the data'. Mo- tivated by this result, in this work we, investigate the structure and in particular the self-consistency
Unsupervised causal representation learning is largely unsolved today. In

---

Under review as a conference paper at ICLR 2021
of the learned autoencoder, rather than focusing the representation's relation to the "groundtruth" factors. In this context, we formulate an inductive bias for autoencoders, that we call interventional consistency, and link it to the solution space of the disentanglement problem. Additionally, introduce a new architectural module for the latent space of autoencoders which unifies the disen- tanglement objective and the, more structured, causal perspective.
METHOD
Consider the 'artificial" generative process implemented by the decoder as ~ P(Z) P ( 0) We express the latent space dynamics in terms of Structural Causal Model (SCM) PPearl 2009) on Z. An SCM is defined by a set of so-called N1. ..., ..., with distribution P(N) that factorises, and set of structural assignments f1. .... fn of the form:
F
F
F
F
F
noise variables
R
Z: = (PA,, for i 1, .....
(1)
F
F2
F3
F4
Fs
where PA, refers to the set of direct causes of the variable Z₁- The set of directed interactions between causal variables iden- tifies graph, which is usually assumed to be acyclic (i.e.
Figure 1: Diagram illustrating In- lerventional consistency: change in one of the factors participating in P(Z) (i.. FA) affects only one of (2) the factors in P(Z') (i.e. Eg). The response R maps Z to Z".
DAG). This formulation naturally entails a distribution P(Z) and corresponding causal factorisation:
P(Z)= = II P(Z).
For thorough introduction SCM we refer the reader to Pe-
ters et al. (217)). We can now apply results from causality to the artificial generative process. In particular, we turn the ICM principle (Peters et al. 2017), which states that the modules partic- ipating in the generative process are independent and can thus be separately manipulated. Let us refer to the composition R the latent space Z C R= back itself, ie.. R: Z Z. Let R(Z') [Z) be the posterior distribution entailed by the map, with Z, Zz being distinct random variables. Furthermore, consider the response aggregate posterior distribution and any factorisation thereof:
as the response map, as done in Leeb et al. (202T), mapping
P(Z') / P(Z)R(Z(Z)dZ- = II P(Z]ZZ
The implications of the ICM principle are trivial applied to the prior P(Z): modifying PAZ) will not affect P(Zz|PAA for # However the consequences get far from trivial if we require the response map to preserve the validity of the principle. Let P(Z) denote the intervention distribution:
= P II PAZ)
(3)
Then we say that the ICM principle is preserved by the response map if:
p(Z') / -12)
II ig
P(Z
(4)
where the indices m and & not necessarily equivalent, although we will from here on assume their equivalence Intuitively, R preserves the ICM principle if any localised change in P(Z) produces corresponding localised change in P(Z'). The idea is illustrated in Figure [ Crucially, this property mirrors the definition of disentanglement in the context of the artificial generative process: each factor in the representation (P (2)))) is sensitive to changes single generative factor AAm)). while being relatively invariant to changes elsewhere in the generative process. This
Allowing * & essentially takes into account the possibility of permutation of the original space dimen- sions in the representation, which is the norm in disentanglement metrics. By assuming = k we get rid of degree of freedom in the problem.
2

---

Under review as a conference paper at ICLR 2021
2
2
2
2
2
2
2
2
2
2-
2
2
2
2
2 2 2 2 2
2
2
2
2
3
3
2
2
2
2
2
2
2
2
2 1 2 2
2
2 2 7
2 2
2 2
2
2
2 2
2 2
D
1
2 2
2
2
2
2
6
2
2
2
2
2
2
2
2
b b
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
A
A
4 4
(a) AE
(b) XCAF-I
Figure 2: (Top) Traversals for autoencoder models on MNIST. (Bottom) Response error measured on each of the six latent dimensions when traversing along the last dimension (= 5). Each line (in different colors) represents a sample drawn from the aggregate posterior. The horizontal axis tracks the dimension being traversed. The vertical axis records the difference between response and original sample along the corresponding dimension. For unregularized models, the traversals are performed in a range whose extremities are identified by the aggregate marginal posterior QZZ). Observe that the model equipped with consistency training exhibits significantly lower error upon intervention than a standard autoencoder.
analogy suggests that the condition expressed in Equation has to be satisfied by representation that disentangles the true generative process. We call this condition interventional consistency.
Interventional consistency can formulated in terms of invariance and equivariance the response map. If the autoencoder satisfies interventional consistency, then P(Z is invariant the ac- tion of the group of interventions I. localised P(Zm (Zjmm) with A m. Moreover, assuming we are able to perform an intervention on the response variables I' equivalent to the one performed on the prior I, the response map equivariant to the intervention, meaning that the order which the response and the intervention are applied is not important. From this symmetry viewpoint, inter- ventional consistency belongs to causal representations: 'the right causal order is the one invariant to the right kind of interventions' (Herb Simon, cf. |Hoover (2008)).
Importantly. a representation that satisfies the interventional consistency condition does not neces- sarily solve the identifiability problem (Gresele et al., 2021), as any bijective transformation of the true causal factors would equally satisty the condition for the correspondingly transformed inter- ventions. Instead, it is manipulable by construction. Without this feature, there is guarantee the decoder can extrapolate to latent samples not seen during training. As an example, consider the common diagnostic tool of latent traversals: it consists of atomic interventions applied the differ- ent dimensions the representation. Often good representation is associated with semantically meaningful traversals in the image space. However, the observed result is only half of the story: if the encoder fails to interpret the generated samples the representation cannot be called disentangled. To illustrate this example, in Figure 2 we show the error recorded the response traversals standard autoencoder.
2.1 CONSISTENCY TRAINING
is possible to use the invariance or equivariance formulation to incorporate interventional consis- tency into the solution space of an autoencoding problem. Let LINV (R) and LFQV (It) be real valued functions measuring the violation of the invariance and equivariance property by the response map R, respectively. Then the space of interventionally consistent autoencoders is defined as:
(FF, D) LINV(F D) or LEov D) 0).
3

---

Under review as a conference paper at ICLR 2021
This hard constraint is replaced with a differentiable regularization by augmenting the optimization objective with LINV or LEOV. Next, we propose evaluation metrics that make the consistency penal- ties easier interpret. Algorithm I and 2 in the Appendix demonstrate how to compute LINv and LEOv. Importantly, LINv is independent of the network architecture of choice, therefore it can be applied to any existing autoencoder method with latent space N. Meanwhile, LEOV only requires an explicit distinction between noises N and causes Z (of the artificial generative process) outlined in the previous section, which, in our experiments, is implemented by the explicit causal latent block (see 4
Note some additional mild approximations are necessary make the penalty terms tractable. Firstly, we assume that the response aggregate posterior P(Z') factorises like P(Z). i.e. that the statistical dependencies in the prior are preserved by the response map. When using an autoencoder with ex- plicit structural mappings this assumption is equivalent to assuming that the response aggregate noise posterior P(N') factorises. Secondly, we employ Monte Carlo sampling to estimate the consistency by sampling hard interventions on the noise variable distribution of the form: P(Nm) + ó(v), with v from the aggregate marginal posterior Q( Ex Q( (X). Finally, in order assess equivariance we treat /' = 7, which relies on the equivalence between Z and z'. This assumption is satisfied as long as the reconstructions (i.e. the autoencoder fidelity is high).
sufficiently similar corresponding the training samples
Below, 2 denotes a sample from the prior P(N) and & its intervened version. Moreover we let N* and k be the expectation of the response N and Ñ respectively. Let & be the result of the intervention / applied to N and we denote the corresponding causal variables by 2 and Z
Invariance score. Let U,,, be a measure of the magnitude of the intervention Im and os the standard deviation of each dimension in the response space. Then define the invariance error in response to the intervention Lm recorded on the dimension i as
EN [I[(N' N Im Mell). E Ui
More precisely we define Umm == Ev [I[(N- - NI.. )ellz] Consequently we score the invariance of the i-th dimension with respect interventions the m-th dimension as:
INV i]
E,... -T.m--mm [erm..].
(5)
We approximate all the expectations with their Monte Carlo estimates. According to this formula- tion. a perfectly invariant response map would score on the off-diagonal elements of the matrix. Notice that invariance is not concerned with the diagonal entries of the matrix. However, we can interpret a high invariance score on the i-th diagonal element as a sign of the unresponsiveness of the dimension as it implies interventions on are not registered in the response. We henceforth refer to these unresponsive latent dimensions with the appellative "collapsed", reference to the well-known effect of posterior collapse (rather common in 3VAE models).
Equivariance score. Similarly, we define the equivariance error and equivariance score as
EN [II(
2) mm 2/_ )alla]
Tr... T
EQV[m, (m, i] = 1 1-
[r]m.]]
In the case of equivariance perfect consistency corresponds all the entries being 1. Notice however that high equivariance is also obtained by the trivial solution (i.c. a constant encoding map). This is due to the fact that the intervention values are sampled from the aggregate posterior.
Self-consistency score. Finally, the self-consistency score evaluates the response inner consistency under no intervention. More specifically, we measure the amount of error introduced by the response on average over the prior. In formula:
SCN [ = =1 -1-BBy LE EN [[I N)
Importantly, self consistency is the only score relating the prior and the response space and closely related to the regularization used by AVAES (Cemgil et al. 2020).
4

---

Under review as a conference paper at ICLR 2021
X
N
X
N
x
Figure 4: Diagram comparing the structure of an autoencoder with (right) and without (left) the Explicit Causal Latent Block. Instead of a single space modelling the representation, an XBlock employs a noise or information space N and causal or structure space Z.
2.2 EXPLICIT CAUSAL LATENT BLOCK
Inspired by the distinction between noise and structure in the SCM we insert a similar bias into the latent space of autoencoders by di- rectly joining the representation layer with a structural mixing block as shown in Figure 4 We henceforth will refer to the botteneck layer as noise terms or N, and the units produced by the subsequent mixing layer causes or Z. Each Z, is obtained from its correspond- ing noise term N; and its predecessors Zed through a learned non- linear function fi, much like in SCM (implemented To increase the sparsity of the structure we apply a learnable mask M, E (R(i-1) to the parents such that Z = (M, @ Zei N.). The masks 2016) to induce weight sharpening, and thus act as gating mecha- nisms of the information provided by the predecessors.
1:42
3:83
EAt
2.83
0.04
2:40
6.42
MLP).
5:17
2:38
0..04
0:14
parametrised through the Gumbel trick (Jang al.
0.70
0.70
5:14
2.81
0.31
6:14
(a) AE
Imartencc
We name this extension of the representation layer "explicit causal latent block" (XBlock) and call XNet (or prefix "X") any net- work using this module. Most disentanglement methods attempt to learn representation consisting of causal variables or disentangled "factors" which are statistically independent (Higgins et al., (207a)) However, except in the trivial case, not the Z, that should be treated as statistically independent, but the N₁. Consequently, the explicit separation between noise and structure reconciles with the
1:0
1:32
3:44
6.60
0.50
2.30
5.30
conventional disentanglement perspective while also providing the
model with additional power to learn identifiable relationships tween the latent variables terms. The causal links revealed by M;
(b) XCAE-1
can then be used to help identify how interventions will affect the
resulting samples for downstream tasks such controllable gener-
DA2
241
ation. Moreover, differentiating between noise and structure allows
0
expand the dimensionality of the causal variables Z, without any need to reparametrise the noise distribution. We partition the causes
6:00
6:0
2.11
in units, each one consisting of multiple dimensions and representing a single statistical variable.
330
3 RELATED WORK
0.
0.0
0.0
322
To the best of our knowledge, this is the first work to formally intro- duce the notion of interventional consistency and develop a training scheme based on it. However, related ideas have been discussed in previous works.
(c) XCAE-E
Figure 3: Invariance matri- ces for autoencoders of the Standard (a), 'XC-I' (b) and 'XC-E' (c) variant. We ob- serve pronounced increase in the score due to the con- sistency training. Note that invariance and equivariance regularisse regularisation achieve sim- ilar performance.
Recently, there is particular interest in applying causality to represen- tation learning (Kocaoglu et al. (2017). Suter et al. (2019) Träuble et al. (021). Schölkopt et al. (2021) In Yang et al. (2021 the au- thors introduce a learnable Causal Layer, which essentially describes SCM, in the latent space of a variational autoencoder. Much like our XBlock, the causal layer in Yang et al. (202I) transforms inde- pendent noise variables into causal ones, thus separating the informa-
5

---

Under review as a conference paper at ICLR 2021
tion from the structure. The structural assignments of the Causal Layer integrate physical limitations to the system's is dynamics that are not present in the XBlock (i.e. additive noise assumption, mild non-linearity). Moreover, Yang et al.] adopt perspective radically different from this work: their goal is to solve causal disentanglement, and they rely on weak supervision to do so.
The introduction of symmetries in the representation layer has been examined from multiple per- spectives (Cohen & (201)) Finzi et al. (2020), |iizii et al. (2021). Weiler et al. (2018), 31, [en- [on al. (2020), ?). In Keller & Welling (2021) the authors use topographic capsules parametrised through variational inference to obtain equivariance to geometric transformations in the input. Like Yang al. (202T) the work uses weak supervision (in the form of ordered input sequences) to disentangle salient characteristics in the data. Interestingly. in [Keller& & Welling (2021) disentangle- ment is obtained as by-product of equivariance, echoing the definition of disentanglement given in
Higgins et al. (2018).
Perhaps the discussion most relevant to the present work presented by Leeb et al. (2021) and Besserve et al. (2018). Leeb et al. develop framework to assess the characteristics of the response map of variational autoencoders, thus laying the groundwork score proposed in Section 2.1 is closely related to the Latent Response Matrix presented in Leeb cet al. (02T). Unlike this paper, in Leeb et al. additionally introduce new tools to evaluate rep- resentations in regards to disentanglement based on the response map. In Besserve al. (0018) interventions on the deep layers of a generative model are employed to discover independent mod- ules participating in the representation. Similarly to the present work, [Besserve al., (2018) adopts a perspective completely agnostic of the true generative process instead focusing on the artificial one. However both Leeb et al.) and Besserve et al. are only concerned with analyzing and evaluating generative processes, while we develop training methods and metrics to directly optimize for the desired structure.
the present study. The invariance
EXPERIMENTS
In the following experiments we demonstrate the relevance of interventional consistency and the strength of the proposed approach. More specifically, we are able to learn representations that are markedly more robust to interventions and can correctly handle manipulations. We quantitatively and qualitatively verify that consistency training is effective in improving the model's 's interventional consistency without harming the model's generative performance. Moreover we produce evidence on the benefit of differentiating between information and structure in the latent space, by analysing the effects of the XBlock on the learned representation.
4.1 EXPERIMENTAL DETAILS
In our experiments we employ three baselines with fundamental differences in their representa- tion layer, namely: convolutional autoencoders (AE) (Bourlard & Kamp (1888), Hinton & Zemel (994)), ) variational autoencoders (BVAE) (Higgins al. 2017b), and structural autoencoders (SAE) Leeb et al. 2020). For each of the baselines two vanants are considered: 'X' and 'XC', where the former extends the original model by integrating the explicit causal latent block and the lat- ter augments the 'X' version through consistency training. The 'XC' variant in turn comes in two modalities employing the invariance (XCTT) and equivariance ('XCE") regularisation respectively. Additionally, we include implementation of the AVAE model to compare the two consistency training schemes.
The experiments are performed on MNIST (eeCnn & Cortes 2010) and CelebA (Liu et al. 2015)
datasets. These datasets have been chosen for being closer to weal-world scenarios than artificially manufactured alternatives whilst being well known and historically recognised in the literature.
The neural architectures considered vary only in size, keeping a coherent overall skeleton across experiments. Specifically, we consider 3 different size standards for MNIST ("S", 'Standard', 'S4') and another for CelebA ('v32', 'v324"). The differences are reported in Table 3 in the appendix, together with the details on the skeleton and training hyperparamters.

---

Under review as a conference paper at ICLR 2021
5 RESULTS AND DISCUSSION
(b) Summary of the invariance score all experiments (MNIST and CelebA). For each model, all its variants are evaluated against the invariance score. The matrix is re- duced to single number through averag- ing. The variational baseline scores system- atically higher than the alternatives. This re- sult is partly due to the presence collapsed dimensions in the representation, which in- crease the model's score due to their abso- lute unresponsiveness. Importantly, tice that the 'XC' variants tends to perform better than the others.
(a) FID scores for the models on MNIST (lower better). We evaluate the FID on the reconstructions (left), prior samples (middle) and traversals (right). Perhaps unsurprisingly, the variational baseline scores markedly higher than the alternative. Crucially, the 'X' and "CC' versions' scores are comparable to the Stan- dard models.
Consistency training improves the model's interventional consistency without harming the re- construction quality. From 6nd can be seen across datasets that the 'X' and 'XC' variants do not interfere with the quality of the reconstructed or prior samples. To quantitatively support this claim we include the corresponding FID scores (Heusel et al. 2017) for all the exper- iments in Figure Sa Additional samples are shown in Section 1 One principle against which to evaluate the quality of the prior samples is their intrinsic diversity. Real world datasets such as rich in low probability details that are thus hard to capture in the represen-
MNIST and CelebA tation. In this regard the autoencoder outperforms the respective VAE models (seel AA.4.I.. The prior samples for deterministic autoencoders
obtained through hybrid sampling, a technique proposed in (Leeb et al. (2020), [Besserve et al.] (2018) which allows to generate previously unseen combi- nations of latent factors by randomly aggregating different posterior samples. For more details on hybrid sampling we refer the reader to Leeb et al.
Figure 2 shows the standard latent traversal plot alongside the traversals response error. The error is measured as the signed difference between the intervened prior noise vector and its expected re- sponse as defined in Section 2.1 i.c. N - Nith. This figure best depicts the discrepancy between generation and inference which lies at the heart of this work and motivates the consistency perspec- tive. By only analysing the traversal plots one might conclude that the learned representation can correctly resolve the alteration introduced by the traversal, and thus that it possesses the generali- sation and modularity suggested by the quality of the generated samples. In short, consistency is assumed to hold. However, the errors recorded on the response show that this assumption false: the intervention applied is not preserved by the encoder, which presumably projects the sample into the observational distribution manifold (Radhakrishnan et al,, 2018). In summary Figure 2 reveals that (i) the current autoencoders (i.e. without consistency requirements) are indeed not consistent under interventions and that (ii) consistency training can improve the model's consistency. Signifi- cantly, consistency training only acts the response error indirectly since the regularisation terms do not include take into account the prior samples. The increase in consistency with consistency training can also be assessed in Figure 3 It must be taken into account that the invariance regu- larisation directly aims at optimising the invariance score, whilst the equivariance score only does so indirectly. Full summaries of the consistency scores for the different training modalities can be found in Section
A quantitative summary of the invariance scores for all experiments is given Figure 5b] We observe that the AVAE baseline has the lowest score on interventional consistency among the variational family, comparably to the 3VAE. On the contrary it is among the highest
7

---

Under review as a conference paper at ICLR 2021
Figure 6: Traversals for the autoencoder model in 3 variants, namely: Standard (left), 'X' (center) and "XC-I' (right). The different variants are comparable in terms generation quality. However, the Xnets show higher degree of modularity between the dimensions. We hypothesise that the Xblock favours independence. We include the full traversal plots in the Additional material (18)
6
6
2
9
2
8
6
2
/
9
2
8
6
6
1
6
2
2
8
6
1
9
5
2
5
4
4 3
2
1
9
5
2
5
4
4
2
8
9
5
2
4
2
8 7
3
8
2
4
5
0
7
3 8
4
5
O
7
8
2
4
3
5 0
1
4 4
2
9
1 7 6
9
& 4 /
8 4
9
5
9
2
5
7
4
4
9
3
3
5 3
0
c 7 0
? 4
6
4
9
8
4
5
4
2
5
9
0
2
7
4
1 0
9
9 5
5
4
2
(a)
(b) XAE
(C)
XCAE-I
Figure 7: Input reconstructions (first row) and prior samples (second row) for the autoencoder model 3 variants, namely: Standard, 'X' and 'XC-I. The prior samples are obtained through hybrid sampling from the aggregate posterior distribution.
performing models when evaluated on self-consistencty (see Figure 28 This discrepancy confirms that consistency, as defined Cemgil et (2020) does not imply interventional consistency.
The explicit causal latent block provides sparsity and self-consistency. A mask element with low value prevents the information from passing from the parent to the child node. Consequently, the noise variables are independent, ing statistical relations between latent units. This is not possible in the current standard of a single space carrying both the information and the statistical structure. A sparse configuration is desirable for interpretability, since it simplifies the dynamics of the generation process. Moreover simpler models more likely be correct, according to the widely accepted principle of parsimony also known as Occam's razor. Section A.4.4 in the appendix displays the mask values for the learned XBlock of different models.
can read off the adjacency matrix the structure of the exist-
Table reports the sparsity levels for different latent sizes. The results show that it is possible to obtain sparsity for every model without the need of any additional regularisation. The introduction of learnable masks endowed with weight sharpening has proved to be sufficient to induce sparsity
LATENT SIZE (N)
SPARSITY (%)
SAMPLING MODE
AE
XAE
XCAE
6
41.66
Posterior
0.96
0.96
0.95
12
45.83
Hybrid 0.75
0.81
0.88
32
48.44
Uniform 0.66
0.68
0.80
Table 1: Sparsity levels per varying latent size. The right column records the percent- age of edges in the Xblock over the of possible connections based on the learned mask.
Table Average self consistency under posterior, hybrid and uniform sampling for AE (left), XAE (middle) and XCAE (right). The average taken over the latent dimension for the S MNIST mod- els.
8

---

Under review as a conference paper at ICLR 2021
in the latent block structure. The number of latent dimension has systematically turn out to be the only indicator for the sparsity level achieved. Moreover, the structure of dependencies in the model varied only slightly with respect to the model family and exhibited greater sensitivity to the latent unit size (c.4..
Table records the self consisteney of autoencoders under three different prior sampling procedures: posterior, hybrid and uniform sampling, respectively from top bottom. 'Pos- terior' refers to drawing samples from the aggregate posterior
Pate
Q(N) = (
z)dz, 'hybrid' applies to its hy- bridised version, and finally in the 'uniform' case we consider
a uniform distribution whose extremes on each dimension are derived from Q(N). We would normally expect a decreasing self-consistency going from the first to the last row: in fact, hybrid sampling breaks any existing statistical correlation be- tween the noise variables, and uniform sampling additionally compels a continuous support in the latent data manifold to be preserved by the response map. In Table 2 the expected trend is approximately confirmed by the AE model. We no- tice that the 'X' version improves the self consistency score under all sampling procedures. proving to be less sensitive to the distribution shifts introduced by hybridisation and conti- nuity. Consistency training further increases the score under all sampling modes. This behaviour suggests the hypothesis that the explicit causal latent block in fact favours indepen- dence between the noise dimensions.
Figure 8: MMD between noises ag- gregate posterior and hybrid sam- pling distribution, measured using an Inverse Multiquadratic kernel (IMQ). The distributions are ob- tained by recording the MMD met- ric over all S models configurations (including different random seeds).
In order to give an objective ground to this hypothesis the additional material (Figure g we look at an empirical measure of independence between the noise variables for all models. More specifically, we compute the Maximum Mcan Discrepancy (MMD) between the aggregate posterior distribution over the noise variables and its hybridised version. By construction hybridisation destroys any existing statistical relationship between the latent dimensions. For both the AE and VAE model families the X' version's metric distribution appears significantly shifted to the left, thus indicating a higher degree of independence between the noise variables.
6 CONCLUSION
In this work we focus on the modularity and self-consistency of a representation, rather than address- ing the identifiability problem. More specifically, have formulated and studied the interventional consistency property of autoencoders. The metrics and regularization objectives we develop result in representations that remain consistent under interventions thus enabling meaningful manipulation. We propose a novel architecture that can be applied to any existing autoencoder, inspired by causal representation learning, to explicitly model the dependencies between latent variables. The architec- ture has empirically shown to improve the model's 's self- celf-conssttency and independence in the latent space. The separation the information and structure of the representation therein connects the dis- entanglement perspective to the more structured causal framework, thereby also opening the door to new analysis tools to enrich the representation's interpretability and controllability by exposing the learned mechanisms.
REPRODUCIBILITY STATEMENT
The used datasets publicly available. Implementation details for all the experiments are in- cluded in the Additional material (A.3). A description of the hyperparameters and network architec- option in the author guide https: 2021/22222000 2202///tthhae /AuthhorCiii for sharing the code anonymously, in a first step we will make comment directly the reviewers and chairs with the link to an anonymous repository once the discussion forums open and in the second step make the full reposi- tory public once the paper is accepted.
tures used is included in Appendix [A.3] As indicated
7/iclr. wcc/onnfeeenceeececcccccc

---

Under review as a conference paper at ICLR 2021
REFERENCES
Philip Bachman, Devon Hjelm, and William Buchwalter. Learning representations by maximizing mutual information across views. arXiv preprint arXiv:1906. SXXV::9966.00100, 2019.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning A review and new perspectives. IEEE transactions pattern analysis and machine intelligence, 35(8):1798-1822 2013.
Gregory Benton, Mare Finzi, Pavel Izmailov, and Andrew Gordon Wilson. Learning invariances in neural networks. arXiv preprini arXiv:2010. 1882, 2020.
Michel Besserve, Arash Mehrjou, Rémy Sun, and Bernhard Schölkopf. Counterfactuals uncover the modular structure of deep generative models. arXiv preprint arXiv:1812.03253, 2018.
Herve Bourlard and Y Kamp. Auto-association by multilayer perceptrons and singular value de- composition. Biological cybernetics, 59:291-4, 02 1988. doi: 10.1007/BBF00332915
A Taylan Cemgil, Sumedh Ghaisas, Krishnamurthy Dvijotham, Sven Gowal, and Pushmeet Kohli. Autoencoding variational autoencoder. arXiv preprint arXiv:2012.03715, 2020.
Taco Cohen and Max Welling. Group equivariant convolutional networks. In International confer- ence on machine learning, pp. 2990-2999. PMLR, 2016.
Mare Finzi, Samuel Stanton, Pavel Izmailov, and Andrew Gordon Wilson. Generalizing convolu- tional neural networks for equivariance to lie groups on arbitrary continuous data. In International Conference on Machine Learning, pp. 3165-3176. PMLR, 2020.
Mare Finzi, Max Welling, and Andrew Gordon Wilson. practical method for constructing equiv- ariant multilayer perceptrons for arbitrary matrix groups. arXiv preprint arXiv:2104. 04.0949,999 2021.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp. 249-256 JMLR Workshop and Conference Proceedings, 2010.
Luigi Gresele, Julius von Kügelgen, Vincent Stimper, Bernhard Schölkopf, and Michel Besserve. Independent mechanism analysis, new concept? arXiv preprint arXiv:2106.05200, 2021.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.
Gans trained by a two time-scale update rule converge local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. VAE: Learning basic visual concepts with constrained variational framework. ICLR, 2(5):6, 2017a.
Irina Higgins, Loie Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew M. Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with a constrained variational framework. In ICLR, 2017b.
Irina Higgins, David Amos, David Pfau, Sebastien Racaniere, Loic Matthey. Danilo Rezende, and Alexander Lerchner. Towards a definition of disentangled representations. arXiv preprint arXiv: 10X2.08//20022300 2018.
Geoffrey E Hinton and Richard S Zemel. Autoencoders, minimum description length, and helmholtz free energy. Advances in neural information processing systems, 6:3-10, 1994.
Kevin Hoover. Causality in Economics and Econometrics, pp. 1-13, 01 2008 doi: 10.1057/
978-1-349-95121-5 2227-1.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameteriization with gumbel-softmax. arXiv preprint 161 /.0/ 144, 2016.
Anderson Keller and Max Welling. Topographic vaes learn equivariant capsules. arXiv preprint arXiv:2 100.:2200901199. 2021.
10

---

Under review as a conference paper at ICLR 2021
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6988, 2014.
Murat Kocaoglu, Christopher Snyder, Alexandros G Dimakis, and Sriram Vishwanath. Causal-
gan: Learning causal implicit generative models with adversarial training. arXiv preprint arXiv:1709.02023 2017.
Yann LeCun and Corinna Cortes. MNIST handwritten digit database. 2010. URL http: /tap:///aan.
lecun. coccun.commeeee
Felix Leeb, Yashas Annadani, Stefan Bauer, and Bernhard Schölkopf. Structure by architecture: Disentangled representations without regularization. arXiv e-prints, pp. arXiv-2006, 2020.
Felix Leeb, Stefan Bauer, and Bernhard Schölkopf. Interventional assays for the latent space of autoencoders. arXiv preprint arXiv:2106. 16091, 2021.
Ziwei Liu, Ping Luo. Xiaogang Wang. and Xiaoou Tang. Deep learning face attributes in the wild.
In Proceedings of the IEEE international conference on computer vision, pp. 3730-3738, 2015.
Francesco Locatello, Stefan Bauer, Mario Lucie, Gunnar Raetsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentangled representations. In international conference on machine learning, pp. 4114 4114-4124. PMLR, 2019.
Francesco Locatello, Ben Poole, Gunnar Rätsch, Bernhard Schölkopf, Olivier Bachem, and Michael Tschannen. Weakly--uuprrissed disentanglement without compromises. In International Confer- ence on Machine Learning. pp. 6348-6335. PMLR, 2020.
Misra D Mish. A self regularized non-monotonic activation function.
arXiv preprint
arXiv: 08681, 2020.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
Pearl. Causality: Models, Reasoning, and Inference. Cambridge University Press, New York, NY, 2nd edition, 2009.
J. Peters, D. Janzing, and B. Schölkopf. Elements of Causal Inference Foundations and Learning Algorithms. MIT Press, Cambridge, MA, USA, 2017.
Adityanarayanan Radhakrishnan, Karren Yang, Mikhail Belkin, and Caroline Uhler. Memorization in overparameterized autoencoders. arXiv preprint arXiv:1810.1033: 2018.
Karl Ridgeway and Michael C. Mozer. Learning deep disentangled embeddings with the f-statistic loss. arXiv preprint arXiv: / 1882.05312. 2018.
Bernhard Schölkopf, Francesco Locatello, Stefan Bauer, Nan Rosemary Ke, Nal Kalchbrenner, Anirudh Goyal, and Yoshua Bengio. Toward causal representation learning. Proceedings of the IEEE, 109(5))612 2344 2021.
Raphael Suter, Djordje Miladinovic, Bernhard Schölkopf, and Stefan Bauer. Robustly disentangled causal mechanisms: Validating deep representations for interventional robustness. In Interna- tional Conference on Machine Learning, pp. 6056-6065. PMLR, 2019.
Frederik Träuble, Elliot Creager, Niki Kilbertus, Francesco Locatello, Andrea Dittadi, Anirudh Goyal, Bernhard Schölkopf, and Stefan Bauer. On disentangled representations learned from correlated data. In International Conference on Machine Learning, pp. 10401-10412. PMLR, 2021.
Maurice Weiler, Mario Geiger, Max Welling, Wouter Boomsma, and Taco Cohen. 3d steerable cnns: Learning rotationally equivariant features in volumetric data. arXiv preprint arXiv:1807.02547. 2018.
Mengyue Yang, Furui Liu, Zhitang Chen, Xinwei Shen, Jianye Hao, and Jun Wang. Causalvae: disentangled representation learning via neural structural causal models. In Proceedings of the IFFF/CVF Conference Computer Vision and Pattern Recognition, pp. 9593-96022 2021.

---

Under review as a conference paper at ICLR 2021
N
R
N
r'
N
N
Figure 9: Diagram illustrating interventional consistency.
A APPENDIX
A.1 NOTATION
In this section report guide to the notation used in the work. Throughout this paper use N and Z for stochastic variables in the latent space. N' and Z' are used for stochastic variables in the response space. X is be used for input data and X' for the input space. (E, D) denote the encoder and decoder mapping, respectively. R = e D defines the response. N denotes a sample from the
prior P(N) and A its intervened version. Moreover we let N and N be the expectation of the response to N and Ñr*, respectively. For Dirac posterior (deterministic map) we use its center as expectation. Finally, Aik marks the result of the intervention applied to N and we denote the corresponding causal variables by 2 and 7.
A.2 CONSISTENCY TRAINING
In this Appendix we take a closer look at the connection between interventional consistency and the pointwise estimate of invariance and equivariance. Interventional consistency can be formalised with the two following definitions.
Definition 1 (Interventions Invariant Response). Given an autoencoder (E, D), and causal fac- torisation of the prior over the latent space P(Z) ПI PPZ.(PA). the response = E D invariant to interventions on Z if for every intervention I in the of interventions Im localised on the factor indexed by m the following conditions are verified:
Z)d7 P(Z') R(Z))(('' Z)d7 = II 7(7
(6)
P(Z') / I o P(Z)) = P(Zm Zj fm)
II P(Z Z441
Definition (Interventions Equivariant Response). Given an autoencoder (E, D), and a causal factorisation of the prior over the latent space P(Z) =I, P(Z.) the response R= E o D is equivariant to interventions on Z if, for every intervention in the set of interventions Im localised on the factor indexed by m. there exist equivalent intervention I' acting in the Z' space, such that the following condition is verified:
[1oP(ZZn FFA...) II P(ZAPA)RZZZZZZ P(Z,, 1mm/m
fm) II P(Z)Z)44))
Equivariance implics invariance, as it provides a strictly stronger condition on the response map. The caveat that evaluate equivariance must additionally deduce 7' from I, which requires
12

---

Under review as a conference paper at ICLR 2021
knowledge of the relation between Z and Z'. Notice that interventional consistency is effectively property the response map and is thus oblivious to the prior Z.
Due to the deterministic nature of the structural assignments definitions and 2 can be reformulated without any loss of generality in terms of the noise variables. Let N and N be the noise variables before and after the response respectively, with
P(N') = / P(N)R(N'NNNN
(9)
P(N') / P(N))(NIN)NN= = 8(NN v) II P(N))RNNINNNN intrr
(10)
In the invariance case we require P(N'); to match P(N'), 4 for all + m. the equivariance case we additionally require P(N')m = l' P(N").. = P(N") = 8(v). In formulas, the solution
space has to satisfy one of the following conditions:
d(P(N'), P(N'): 0 vi A (invariance)
(11)
d(P(N");, I P(N)): = vi e [1, (equivariance),
(12)
where I o P(N), = P(N'), for + m and d a generic distance in the space of distributions on N. We can approximate P(N") and P(N') by sampling from the prior (i.e. P(N) and P(N) re- spectively) and subsequently applying the response map each sample. Using pointwise estimate of the above terms can viewed as adopting a distance that decompose into sum of the intra- sample distances. This choice rules solutions for which the response posterior suffers shift in a non non-intervened dimension when an intervention is applied. We now turn to a full characterisation of the pointwise estimates of invariance and equivariance.
Pointwise invariance. We measure the Euclidean distance between the responses expectations, i.c. || - ///122 for i 1, ... and k 1, ., with T being the total number of samples. By focusing on the noise terms we can leverage the noise independence assumption to significantly simplify the equations. Notice that we couldn't apply the same direct measurement with the causal variables 7 and 7 because of the existing connections between causal variables, i.e. when modifyyng Nm can expect Nimm to stay constant but cannot say the same for Z'um if is descendant of m. However, invariance can be checked distributionally for Z, by looking at the conditionals P(Z
Pointwise equivariance. To compute pointwise equivariance we can directly work in the causal variables space measuring the Euclidean distance -| 1/ The argument for evaluating equiv- ariance the causal variables level is purely practical: in this way the structural assignments receive a stronger training signal, which is beneficial especially early on in training when the set of connec- tions being first outlined.
Algorithms
and show how to compute pointwise invariance and equivariance, respectively.
13

---

Under review as a conference paper at ICLR 2021
Algorithm 2 Algorithm computing the equivari- ance regularisation term.
for in [1, n do
Algorithm 1 Algorithm computing the in- variance regularisation term.
for Im in Im C I'm do
Sample N ..... NT. P(N)
for m in [1,n] do for In in Im Im do
Copy N f Nk
Sample N ..... N', S ~ P(N)
Intervene N <- Vlm
Copy N +- Nk.e for k = ,,,.? T
Copy Not + N
Intervene NK. t 1l
Intervene Nth <- Vm 2.2t + MN,,, PAA) 2 - f.(N¹ i, PAT)
n.T Ex Eijn 11
N
end end
Flm + 1
n.T ΣE Exjm 11 2
CINV
[mml Exmeim elm
end
end
LEgv
Tm Eimem" VIm
The regularisation terms LIvY and LEgv are included in the standard loss weighted by λnv and λBgV. respectively. our experiments we use λINV λBσy 4. We consider both stochastic and deterministic encoders, while treating the decoder as deterministic (a Gaussian noise model used, with the mean parametrised by D). If the encoder is stochastic the response R(N'N) defines distribution over the N' space: in the case of the VAE this distribution is known to be Gaussian, thus fully described by its mean and variance. Experimentally, including information on the spread of the distribution in the regularisation term has not benefited training, often resulting in numerical instabilities. This outcome has been observed for parametric distances as well as well- known divergence measures (such the Kullback-Leibler).
A.3 EXPERIMENTAL DETAILS
In this section we provide the details regarding experimental settings.
All experiments are performed on GPU core NVIDIA A100 40GB and based on the Pytorch (Paszke et al., (017) framework. The models are trained using the stochastic iterative optimiser Adam (Kingma Ba 2014), with default parameters. The learning rate is set to 0.0001 and halved every 30 epochs. The MNIST and CelebA models are trained for 100k and 200k iterations, respec- tively. We use batch size 128 or all the experiments. Each model is trained with different random seeds (11, 13, 17, 37, 121) and every pseudo-random number generator in pytorch, numpy and python. random is seeded accordingly.
As anticipated in 4.1 we adopt single architectural skeleton for all the models, varying only in size. The skeleton is shown in Figure 10 All the architectures use the Mish (Mish) (020) nonlinearity. for MNIST and 7 for CelebA. The number of filters in the
The number of pooling layers is set convolutional layers is fixed to 32 for MNIST, 100 for CelebA, All the layers, except for the nor- malization ones, are initialised using the method discussed Glorot & Bengio (2010). Differently from what is shown in Figure 10 the 'X' and "XC" variants include an additional layer (forming the explicit causal latent block) applied the latent vector as shown Figure 4 Finally, in Table 4 we report the size of each model in terms of the number of trainable parameters.
A.4 ADDITIONAL RESULTS
A.4.1 COMPARING SAMPLES QUALITY
In this section we report the reconstructions, generated and traversal samples for the main experi- ments performed on MNIST and CelebA.
14

---

Under review as a conference paper at ICLR 2021
Inpur
Courrivet
Upsamplaterr
Oulout
Condlookk
Figure 10: Architectural skeleton used for all models. The input is fed to convolutional network (ConvNer) to obtain its latent representation (in green). This is then fed to a second convolutional network (UpsampleNer), equipped with bilinear upsampling layers, that produces the output image. The ConvNet comprises blocks of convolutional layers (ConyBlock) and regularly spaced pooling layers (red). The precise structure of the ConvNet is determined by parameters specifying the number ConyBlock and pooling layers, respectively. As an example, the ConvNet in this figure has depth 6 and 3 pooling layers. Each ConvBlock is composed of convolutional layer with fixed number of filters (blue), a non-linear activation (green), and batch normalisation (yellow). The the ConvNet is followed by a dense head (purple) which processes the flattened
last ConvBlock version of the ConyBlock output to reduce it to the latent space size. The dense head consists of 3 linear layers interleaved by non-linear activations. Finally, UpsampleNet copies the ConvNet structure in reverse order, and with upsampling layers instead of pooling ones. The output of the UpsampleNet is passed through a Sigmoid activation to project back the input space.
NAME
DEPTH
LATENT SIZE (N)
UNIT SIZE (2,)
DATASET
Standard
MNIST
S
12
6
MNIST
$4
MNIST
v32
16
32
CelebA
v324
CelebA
Table 3: Overview of the architecture variants. The Depth entry refers to the number of convolutional layers employed in the encoder. The decoder has depth equal to that of the encoder.
15

---

Under review as a conference paper at ICLR 2021
SIZE
NAME NUMBER of PARAMETERS
Standard
AE
423,259
Standard
XAE
425,677
AE
309,973
S
XAE 310,984
XAE
313,936
v32
AE
2,840,827
v32
XAE
2,850,299
v324
XAE
2,878,523
Standard
VAE
298,771
Standard
XVAE
301,189
S
VAE
186,025
XVAE
187.036
S4
XVAE
188,836
v32
VAE
2,763,707
v32
XVAE
2,773,179
v324
XVAE
2,795,259
Standard
SAE
442,267
Standard
XSAE
444,685
S
SAE
320,488
S
XSAE
320,488
S4
XSAE
323,440
Table 4: Number of trainable parameters for all the models in the experiments. The 'X'' model reported in this table. The training
variants have the size as the 'X' variants and thus time varies between hours (for the smaller models) to 2 days (for the bigger ones, with consistency training).
16

---

Under review as a conference paper at ICLR 2021
5
2
5
8
1
7
9
9
1
3
7
8
2
0
9
8
5
S
5
Y
5
9
1
/
5
5
4
V
4
6
5
4
5
4
7
2
2
9
2
4
7
4
7
2
8
8
8
5
0
3
5
7
O
9
4
3
0
5
8
0
8
0
/
0
9
0
4
2
6
3
6
6
4 2
0
0
6
3
6
/
1
/
3
9
2
9
4
7
5
3
9
3
3
6
9
5
3
9
0
3
6
2
5
3
5
5
6
5
5
a
2
8
4
3
5
6
2
6
2
5
9
0
2 0 5
5
9
(a)
AE S
(b) VAE S
(c) SAE S
5
6
6
7
4
8 5
1
0
/
6
4
8
7
3
1
7
9 8
7
5
8
0
9
5
7
3
7
8
2
0
8
5
S
1 4 1 /
5
6
0
3
4
6
5 4
5
5
1
5
3
4
6
4
5
5
0
3
4
6
5
6
5
5
4
4
7
2
2
6
5
4
5
4
9
2
5
5
4
4
7
2
8
8
8
5
0
8
2
I
3
8
8
8
180099
2
8
8
00500
9
0
1
0
0
6
O
0
0
9
O
0
9
0
3
6
/
/
3
3 6
2
3
6
5
2
0
3
3
8
4
7
3
5
8
4
7
2
4
8
2
8
7
4
9
5
7
4
4
5
(d) XAES
XVAE S
(f) XSAE
8 5
6
6
5
7
8
6 4
8
5
8
0
/ 4
6 4
a
6
5
4
8
7
6
7
3
1
7
8
2
0
9
6
7
3
8
2
0
9
8
3 0
8
1
9
3
4
4
4
5 S
1
5
0
3
4
6
5
5
5
5
6
0
5
7
1
5 2
&
4
2
2
5
4
8
0
2
6
5
4
5
4
2
2
6
5
4
6
3
3 2
9
2
4
4
G
8
1
8
8
0
2
2
2
5
2
9
6
0
6
3
0
9
3
0
6
/
1
3
4
3
2
9
170
0
9
0
3
6
5
7
2
9
3 2
9
5
3
8
60
8
5
2
8
4
1
7
3
3
8
/
8
7
O
4
7
3
/
9
4
5
8
P
2
2
9
5
8
9
2
0
4
7
9
9
2
9
4
2
4
9
7
4 2
9
5
1
2
9
7
3
5
9
(g)
XCAE-IS
(h) XCVAE-IS S
(i
XCSAE-IS
2
3
2
6
8
5 9
6
8 5
6
6
0
6 4
2 2
6
8
5
9
0
3
3
0
6 7
2
0
2
5
8 0
4
1
8
2
2
5
2
6
5
4
6
4
(
8
2
4
O
6
5
2
8
7
4
P027
7
4
3
8
5
0
2
4
4
0
3
5
5
1
7
9
3
6
L
0
0
6
5
5
7
9
5
7
1
1
2
5
3
9
6
3
6
4
9
4
1
5
3
0
1
3
8
0
5
5
9
9
0
5
5
7
2
5
1
0
5
3
5
6
/
8
5
7
4
/
2
8
5
8
6
8
5
9
5
0
6
6
3
7
0
2
2
4
8
1
6
5
0
6
5
6
3
2
0
g
8
4
3
O
3
(i) XCAE-E S
(k) XCVAE-E S
(I) XCSAE-ES
Figure 11: Comparison of reconstruction quality of models.
17

---

Under review as a conference paper at ICLR 2021
2
$ /
8
6
2
5 2
8
G
V
66
7
250
144
1779
0
5
6467
9 6
0
O
9
9
q
/
/
133667
O
6
6
4 7
4
\
3
5 S
1
112
8 (
f 2 1
5
f
5
5
1
S /
2
5O
5
1
9 V 6
1
09
5 1000
8 2
0
6
2
2
9 J 1177777
/
5
4 9 3 b
2
4
6)74
S
5
4
Y 9
9
a
8888
O
0
4
5
5 0
5
6 2 2
9
2
4 7
49
18N
0
3
3
5
9 A
(a) AE S
(b) VAE S
(c) SAE S
1
0
5 Φ
5 r
6
3
7
9
7
9
6
8 9
8
3
7
0
8
1
3 7
5
2
7
9
O
4
5
0
7
3
9
8
8
5
6
7
8
%
1
0
1 9 8
5
5
D
5
9
5 2
6
9
0
9
6
669
6 F
3
7
1
7
$
2
2
1 4 1
1
6
9
0
6
3
2
1
5 8
4
y
8
8
S
5
6
+
8
9
2
8
0
0007
6
5
<
6
6
8
K
7
2
5
9
5
0
8
2
1
6
3 7
3 4
R
/
2
6
2
2 2
2
5
7
J
<
J
2
7
6
7
67
7
9
9
0
(d) XAE
(e) XVAE S
(f) XSAE $
5
5
7
1
/
D
5
6
8
2
3
0
2
8
8
9
9
6 O
à
69 6
5 0 5
C
9 f 3.
4 7
Q 3
\
6
3
3
6
0
5
4
2 L
7
0
3
/
5
7
9
4
3
3 5 9
6
2
>
3
6
5
9
4
2.
3
5
2
2
2
9
2.
8 2 4
0
9
3
1
6
3
4
2
3
4
2
8
6
8
9
9
R P 0
0
7
5
1
6
6
5 2
$
2
3
2
2
9
A 2
2 2
6
/
5
9
2
5
2 5
6
9
9
7
1
3
8
6
9
1
8
7
7
0
7
9
9
9
0
8
5
5
7
J
3
3
3
9
2
5
9
6
7
6
1
7
3
1
1
8
5
4
2
6
3
2
9
3.
9
9 8
3
8
8 2
7 7
(
6
9
4
5
5
9
0
3
7
3
4
2
6
3
2
5
5
O
9
6
5
4
3
2
6
6
8
5
2
2
5
(g) XCAE-IS
(h) XCVAE-1 S
(i) XCSAE-IS
D
6
5
1
6
9 6
8
3
3
9
3
L
2
2
C
1
4
7
8
2 9
2
4
4
3
2
L
5
7
8
3
5
5
7
5 0
8
6
2
9
6
K
4
5
/
8
1
3
3
7
3
6
9
1
<
2
2
7
9
f
S
Y
7
6
1
4
6
5
0
9
3
7
9
3
2
9 T
7
3
$
0
\
4
0
2
0
6
2 /
9
9
$
1
7
1
6
5
8
8
8
1
0
r
3
0
r
6
g
2
3
N2
V
5
7
A
2
5
5
6
1
4
4
4
2
9
7
8 C
5
0
6
2
5
6
8
2
6
5
7
6
6
7
7
3
6
7
7
$
7
9
6
5
8
4
6
8
f
5
3
8
5
3
5
6
0
2
9
7
4
2
4
2
8
4
3
2
2
2
5
7
(i) XCAE-E
(k) XCVAE-E S
(I) XCSAE-E S
Figure 12: Comparison of S models on generated samples quality.
18

---

Under review as a conference paper at ICLR 2021
2
(a) AE
(b) VAE
(c) SAE S
TTTT T
(d) XAE
(c) XVAE S
(I) XSAES S
3 3
3
(g) XCAE-IS S
(h) XCVAE-IS S
(i) XCSAE-IS
5
C
3
( XCAE-ES
(k) XCVAE-ES
(I) EXCSAE-ES
Figure 13: Comparison of S models on traversals quality.
(a) AE v32
(b) XAE v32
(c) XCAE-1 v32
(d) XCAE-E v32
Figure 14: Comparison of v32 models reconstruction quality.
19

---

Under review as a conference paper at ICLR 2021
(a) AE v32
(b) XAE v32
(c) XCAE-I v32
(d) XCAE-E v32
Figure 15: Comparison of v32 models the generated samples quality.
AE v32
(b) XAA v32
(c) XCAE-I v32
(d) XCAE-E v32
Figure 16: Comparison of v32 models on the interpolation samples quality.
20

---

Under review as a conference paper at ICLR 2021
Fansa
Lanka
(a) AE v32
(b) XAE v32
Figure 17: Comparison of v32 models on traversals quality (1).
21

---

Under review as a conference paper at ICLR 2021
(a) XCAE-I v32
(b) XCAE-E E v32
Figure 18: Comparison of v32 models on traversals quality (2).
22

---

Under review as a conference paper at ICLR 2021
(a) Traversal on 0
(b) Traversal on 1
(c) Traversal on 2
(d) Traversal on 3
(e) Traversal on 4
!I) Traversal on 5
Figure 19: BetaVAE traversal response errors.
A.4.2 TRAVERSAL RESPONSES.
We here report the complete traversal response plots for the S version of the MNIST experiments. Each plot shows the errors registered on all the latent dimensions as the traversal is performed.
VVA family traversal response errors. The models in 3VAE family are exposed to the well- known effect of posterior collapse. In the following plots collapsed dimensions can be distinguished by the particular behaviour of the response error (zero when the dimension is not intervened on and equivalent to the intervention value elsewhere). We also include the traversal response errors measured on the AVAE model. Interstingly, all the variants in the VAE family present two collapsed dimensions, except for the XVAE model, that has 3 collapsed dimensions (50% of the total number in this case) and the AVE model which has none.
AE family traversal response errors. autoencoder family of models. Notice how in this case, as for the variational case, consistency training effectively decreases the enror.
Additionally we show the traversal response errors for the
23

---

Under review as a conference paper at ICLR 2021
(a) Traversal on 0
(b) Traversal on 1
(c) Traversal on 2
(d) Traversal on 3
(c) Traversal on
Traversal
Figure 20: AVAE traversal response emrors.
24

---

Under review as a conference paper at ICLR 2021
2.5.3
3.S.3.
4.A.
(a) Traversal on
(b) Traversal on 1
(c) Traversal on 2
(d) Traversal on 3
(c) Traversal on
Traversal
Figure 21: XVAE traversal response errors.
25

---

Under review as a conference paper at ICLR 2021
(a) Traversal on
(b) Traversal on 1
(c) Traversal on 2
(d) Traversal on 3
(c) Traversal on
Traversal on
Figure 22: XCVAE-I traversal response errors.
26

---

Under review as a conference paper at ICLR 2021
(a) Traversal on 0
(b) Traversal on 1
(c) Traversal on 2
(d) Traversal on 3
(c) Traversal on
Traversal on
Figure 23: AE traversal response errors.
27

---

Under review as a conference paper at ICLR 2021
L.A.A
(a) Traversal on 0
(b) Traversal on 1
(c) Traversal on 2
(d) Traversal on 3
(c) Traversal on
Traversal
Figure 24: XAE traversal response errors.
28

---

Under review as a conference paper at ICLR 2021
(a) Traversal on 0
(b) Traversal on 1
(c) Traversal on 2
(d) Traversal on 3
(c) Traversal on
Traversal
Figure 25: XCAE-I traversal response errors.
29

---

Under review as a conference paper at ICLR 2021
A.4.3 CONSISTENCY METRICS
In this sections we report the results on the new consistency metrics for all the models. We make the following key observations: (i) that consistency training generally strengthens the models' invari- ance and equivariance, (ii) that although equivariance implies invariance, models using invariance regularisation have comparable performances to those using equivariance regularisation.
0.85
0.90
08
invanance Score
08
0.60
Standard training Consistency training
ivariance
XCAE
XCSAE
XCVAE
Invariance score for all experiments compar- ing performance of models with and without con- sistency training.
(b) Invariance score for all the models with con- sistency training comparing invariance-bass and gguivariance-basse regularisation.
0.800
Equransacce soore
0.905
0.900
0.000
0.755
0.800
0.800
0.255
0.825
Consesteny traiing
Standard trainin
0800
equivariance
XCAE
XCSAE
XCVAE
(b) Fquivariance score for all the models with con- sistency training comparing invariance-based and equivariance-base regularisation.
Equivariance comparing performance of models with and without consistency training.
We proceed to present the self-consistency scores for the experiments. Notice that self- consistency is the only metric that, similarly to to/Cemgil et al. (2020), directly compares the response to the prior. Since, the AVAE model objective conceptually analogous to self-consistency. we would expect the AVAE model to score higher than the VAE baseline on this consistency metric. Figure 28 confirms this prediction.
30

---

Under review as a conference paper at ICLR 2021
2.4
044
2XT
1047
6.11
145
0.3
0:50
6.5%
0.47
0.64
15
(a) AE S
(b) AE Standard
0.41
2AM
0:M4
6:36
6:66
0.21
0.22
(c) XAE
(d) XAF Standard
0.29
0.50
0:24
0.92
0:23
0.00
0:41
1::01
6.1
3:14
0.93
1:4G
0.1
6m
8:00
1:0
0.03
0.25
9:14
12
(c) XCAE-IS
(f) XCAF-I Standard
Figure 29; Self-consistency score for each noise variable of the autoencoder family. Results are shown for both the S and Standard model sizes.
1.0
model_famm family
AE AVAE
SAE
0.6
044
15+
0.0
XCAE
XCSAE
XOVAE
Figure 28: Summary of self-consiseency for multiple random seeds and model versions. The score is reduced single number by averaging across the dimensions. Note that for the variational models the score is obtained by sampling from the prior distribution, while for the AE and SAE families posterior sampling used.
A.4.4 XNETS
Masks structure. We look at the structure learned in the explicit causal latent blocks. The mask values in the Xblock describe weighted adjacency matrix, which is displayed by the follow- ing heatmaps. Interestingly, the structure not significantly affected by the training scheme adopted (i.e. consistency regularisation) or the model family. Instead its variability is mostly linked to the causal units size and latent size.
31

---

Under review as a conference paper at ICLR 2021
Causal block adjacency matrix
Causal block adjacency matrix
0.00
0.00
1.00
0.00
0.00
0.00
0.00
1.00
0.00
0.00
0.00
0.00
1.00
1.00
0.00
0.00
0.00
0.00
0.00
1.00
1.00
0.00
0.00
0.00
0.00
1.00
0.73
0.00
0.00
0.00
0.00
1.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.90
0.00
0.00
0.00
(a) XAE S
(b) XAE S4
Causal block adjacency matrix
Causal block adjacency matrix
0.00
0.00
0.00
0.00
1.00
0.43
0.00
0.00
0.00
0.00
1.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
(c)
XCAE-IS
(d) XCAE S4
Causal block adjscency matrix
Causal block adjacency matrix
0.00
0.00
0.00
1.00
1.00
0.00
0.00
0.00
0.00
0.00
1.00
1.00
0.00
0.00
0.00
0.00
1.00
0.09
0.00
0.00
0.00
0.00
1.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
(c) XCAE-ES
(f) 4S
32

---

Under review as a conference paper at ICLR 2021
adjacency
0.00
0.00
1.00
0.02
0.00
1.00
1.00
0.00
0.00
0.00
1.00
0.00
0.00
1.00
0.00
1.00
1.00
0:99
1.00
0.00
0.00
0.00
1.00
1.00
0.00
1.00
0.00
0.00
0.00
1.00
1.00
0.00
0.00
0.90
0.00
1.00
0.91
1.00
1:00
1.50
1.00
1.00
1.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
1.00
1.00
1.00
1.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
1.00
0.90
1.00
0.00
1.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
1.00
0.00
0.00
1.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
1.00
0.00
0.00
0.00
0.00
0.50
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
1.00
0.00
0.00
0.90
0.00
0.00
0.00
0.00
0.00
1.00
1.09
0.00
6.00
0.00
0.00
0.00
0.00
10
11
(a) XAE standard
Causal block adjacency matrix
0.08
0.00
0.00
1.00
0.00
3.00
0.08
0.00
1:00
0.00
1.00
1.00
0.00
0.00
0.00
1.00
0.00
0.00
1.00
0.00
1:00
1000
1.00
1.00
0.00
0.00
0.00
1.00
1.00
0.00
1.00
0.00
0.00
0.00
1.00
1.00
0.00
0.00
0.00
0.00
1.00
1.00
1.00
1:00
1:60
1.00
0.85
1.00
0.00
0.00
0.00
0.00
1.00
0.00
1:00
1.00
0.00
0.00
0.00
0.00
0.00
1.00
0.00
0.00
1
11
(b) XCAE-1 Standard
33

---

Under review as a conference paper at ICLR 2021
adjacency
0.00
0.00
0.87
1.00
1.00
0.00
0.00
0.00
1.00
0.00
0.00
1.00
0.00
1.00
1.00
1.00
1.00
0.00
0.00
0.00
1.00
0.98
0.00
1.00
0.00
0.00
0.00
1.00
1.00
0.00
0.00
0.90
0.00
1.00
0.91
1.00
1:00
1.50
1.00
0.99
1.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
1.00
1.00
1.00
1.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
1.00
0.90
1.00
0.00
1.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
1.00
0.00
0.00
1.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
1.00
0.00
0.00
0.00
0.00
0.50
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
1.00
0.00
0.00
0.90
0.00
0.00
0.00
0.00
0.00
1.00
1.09
0.00
0.00
6.00
0.00
0.00
0.00
0.00
0.00
10
11
(a) XCAE-E E Standard
adjacency
adjacency
0.00
0.00
0.00
1.00
0.00
0.00
0.00
0.00
1.00
0.00
0.14
0.00
0.00
0.00
1.00
0.00
0.00
0.00
0.00
0.00
1.00
0.00
0.00
0.00
0.00
0.00
1.00
1.00
0.00
0.00
0.00
0.00
1.00
1.00
0.00
0.00
0.00
0.00
0.00
1.00
0.00
0.00
0.00
0.00
0.00
1.00
0.98
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
(a) XSAE
(b) XCSAE-IS
Causal block adjacency matrix
Causal black adjacency matrix
0.00
0.00
0.00
0.00
0.00
0.00
0.00
1.00
0.89
0.00
0.00
0.00
0.00
1.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
(c) XVAES
(d) XCVAE-1S
XBlock visualisation tools. We introduce a qualitative evaluation tool reliant on interventional consistency that permits to interrogate each representation dimension with specific examples. We
34

---

Under review as a conference paper at ICLR 2021
produce hybrids between two prior samples by separately intervening different dimen- sions. More precisely, given a base N ~ P(N) and an alternative N2 ~ P(N), we obtain n hybrids by applying the following interventions one at time:
N N for i ...,
By decoding the result noise vector we obtain a visualisation of the hybridisation: we can thus visually assess the effect of localised intervention on the representation. Examples are given in Figure 34
Many dimensions do show any change in response to the intervention. A lack of reaction could be due to a general insensitivity of the dimension (e.g. the case of a collapsed unit), or to an inci- dental overlapping of the two samples, or even to an inability of the decoder to generalise outside of support. Moreover for the 'X' and 'XC' model variants we observe that the change produced by the interventions semantically different on distinet latent dimensions. This suggests that the representation has developed certain degree of modularity.
6
O
6
6
6
1
6
2
0
1
6
6
2
O
6
6
6
3
O
2
3
6
6
8
O
8
6
9
O
C
6
5
6
6
9
O
6
b
6
2
0
1
6
5
6
6
8
0
8
6
6
6
0
6
6
6
6
6 2
2
2
2
2 2 6
2
2
2
2
2
2
Figure 35: Double intervention matrix N (left) and Z (right) for XCAE-I S model MNIST. The yellow box contains the base sample, while the two alternatives are displayed inside the blue and red boxes. The hybrid samples are displayed in the matrix according the interventions indices, iee element i, correspond to the hybrid obtained by replacing the i-th and j-th elements of the base with the corresponding components in the first and second alternative respectively. The items on the diagonal correspond to the result of hybridisation only with the first alternative.
We put consistency training to test by simultaneously applying interventions the base sample using two different alternatives. Additionally, we look at the result both when the interventions are applied on the noise and on the causal variables. Applying an intervention on the causes breaks the connection from the parent to the intervened unit, thus altering the statistical relations between causal variables. The results are shown in Figure (35] There is noticeable difference the quality of the hybrid samples between the two kinds of interventions: hybrids on the causal variables more prior samples, sometimes producing meaningless shapes
often do not appear connected to any (e.g. element (4, 1) in the matrix the right). On the other hand, the noise hybrid samples appear convincing and mostly semantically homogencous. Interestingly, as highlighted in orange in Figure 35 the two different interventions can provide two different interpretations of the original ambiguous shape.
35

---

Under review as a conference paper at ICLR 2021
(a) AE v32
(b) XAE v32
(c) XCAE-I v32
(d) XCAE-E v32
Figure 34: Hybridisation on v32 models representation. The left column corresponds to the base sample, the middle column to the presented alternative and the hybridisation result for each dimen- sion is shown on the right.
36


Reviews:
[{"review_id": "zdCVf8cLjja", "reviewer": "Reviewer_zHvy", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n- The proposed concept, the interventional consistency, seems highly novel. It extends the notion of consistency in a meaningful way.\n\nWeaknesses\n- The largest weakness of the paper is the lack of clarity. The paper is very difficult to follow, and I believe the difficulty is attributed to the unpolished exposition. In general, Section 2 assumes the readers have a strong background in causal representation learning to an unnecessary extent. \n  - The ICM principle is introduced without stating its full name (independent causal mechanisms, I guess?). \n  - Since ICM works as a fundamental underlying principle for the work, it deserves a more detailed illustration and justification.\n  - It would be helpful for readers if it is stated that PA states \"parents\". \n  - What is the definition of the intervention distribution? What are the examples? What properties of them are assumed?\n  - Providing a concrete, real-world example of interventional consistency would be helpful.\n  - While the paper says $m$ and $k$ are assumed to be equivalent, Figure 1 depicts the scenario where $m$ and $k$ are different. Additionally, I couldn't come up with a meaningful real-world case where $m\\neq k$.\n- The other parts of the paper also have clarity issues.\n  - Texts in Figure 2, Figure 3, Figure 5, Figure 8 are too small and difficult to read.\n  - The figure numbering is missing for Figure 5.\n  - There is a broken reference link on page 6.", "summary_of_the_paper": "The paper proposes a novel principle for representation learning using an autoencoder, named interventional consistency. The tractable objective functions, namely Invariance Score and Equivariance Score, are formulated, and Explicit Causal Latent Block is proposed to model the causal structure of the latent representation.", "summary_of_the_review": "I vote to reject the paper, due to the lack of clarity. To improve the clarity of the paper, significant rewriting seems to be needed.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "MEYKsj4sff", "reviewer": "Reviewer_hhq1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "- The paper is motivated by the goal of \"causal representation learning\" and explicitly mentions it does not address the important question of identifiability. The most important problem is that there is no justification as to why the authors' notion of interventional consistency has anything to do with what researchers in the past have meant by a \"causal representation\" (e.g. disentangled, modular, robust, ...etc.). \n- I found the paper hard to read even though the ideas proposed are rather simple. The paper is framed within the larger context of causality, but I am not sure this is the best way to present the contribution, which might explain my confusion. \n- The motivation for the \"explicit causal latent block\" is unclear. Why does it make sense to have the output of the encoder be the noise of the SCM over $Z$?\n- Some very important quantity like $\\mathcal{L}\\_\\text{INV}$ and $\\mathcal{L}\\_\\text{EQV}$ (which are the actual regularizer proposed) are defined only in the appendix. Important contributions should be presented in the main text. Also, the explanations for these central quantities lack clarity.\n- It is not clear why learning a SCM in the latent space of MNIST makes sense. For instance, I can't imagine the shape of a digit causing its thickness. \n- Figure 6 shows three latent traversals for a baseline and two variants of their approach and claim that their approach yield more modular traversal. However, it is not clear what is meant here by modularity and I do not see how the latent traversals of their approach are better.\n- The experiments show that indeed the regularizer improves \"interventional consistency\". But, as I said earlier, it is not clear how this relates to \"causal representation learning\" or why it is something beneficial regarding disentanglement, robustness, modularity etc. \n\nMinor:\n\n- The notation is sometimes imprecise: for example, in the equation between (2) and (3), it is not clear what is meant by $Z'\\_{j \\not= i}$. Is it all components of $Z'$ that are not $i$? If so the decomposition is wrong. Is it $Z'\\_{j<i}$ for some ordering of the nodes? If so what is the ordering? This point matters because (4) goes on specifying what is meant for the \"ICM principle to be preserved by the response map\" based on this above equation.\n- The notation is sometimes hard to parse: for instance, Section 2.1 has a few double \"hat\" like this: $\\bar{\\tilde{Z}}$. \n-  Table 1 and 2 have no error bars, making it hard to asses whether the gaps between different methods are significant or not. \n", "summary_of_the_paper": "This work proposes the notion of \"interventional consistency\" as a beneficial property learned representations should have and introduces a regularization term to enforce it in autoencoders. Moreover, the authors introduce an \"explicit latent causal block\" that allows learning a structural causal model (SCM) over the latent factors. The experimental section argues that both the regularization and the \"causal layer\" improve interventional consistency. ", "summary_of_the_review": "Given the points raised above, I cannot recommend acceptance. My main concerns are with (i) the lack of motivation for both the \"interventional consistency\" and the \"explicit causal latent block\", as well as (ii) the (sometimes significant) lack of clarity. The paper did not make a convincing case that \"interventional consistency\" is a valuable tool for representation learning.   ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "BMWvcXtlDpQ", "reviewer": "Reviewer_9iqe", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n- The idea is well-motivated and seems to be theoretically justified.\n- The empirical results are promising and support the main claim of the paper.\n- The limitation of the proposed method is also discussed (in the paragraph right before Section 2.1).\n\nWeaknesses\n- Presentation of the paper can be improved\n  - ICM is not defined when it's firstly used in page 2.\n  - There is a broken citation in the 3rd line of the 3rd paragraph of Section 3\n  - The first two figures in Section 5 is not numbered.\n    - The order of all methods in the two figures is weird/inconsistent. You should keep a consistent order for all baseline + variants.\n    - I also suggest better a better way to label them: You can colour each baseline model with a colour and fill the variants with (no filling, dots, dashed lines).\n\nQuestions\n- For Figure 2, how do I know the traversals are supposed to give low errors? i cannot tell if the traversal is along any disentangled dimension from the images.", "summary_of_the_paper": "The paper proposes to use interventional consistency to regularise representation learning in VAEs.\nThe idea is well-motivated by the ICM principle and theoretically justified.\nThe paper suggests to use interventional consistency for both training and evaluation of the representation learnt by VAEs.\nResults show that the proposed idea can give more modular and interpretable representation.", "summary_of_the_review": "The paper proposes a well-motivated and (seemingly) theoretically justified way to regularise representation learning in VAEs.\nThe results also support the main claim of the paper.\nTherefore I believe this paper is beneficial to the community.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "ANLSfcZR3NC", "reviewer": "Reviewer_6BaE", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Frankly, I am not quite convinced why the interventional consistency is desired even after reading the draft. Some assumptions seem too strong to me and I feel the main objective of the paper is not conveyed well.\n\nSome typos and issues listed here:\n\n1) Unexplained notations: \n\n      a) section 2, composition R = E\\circle D. here neither E nor D is defined. \n\n      b)  Figure 4 comes before the figure 3\n\n      c) the third paragraph of section 3, related work: there is a citation error with question mark\n\n\n2) Lack of clarity:\n\na) The major algorithms 1) and 2) are not described in the main paper.  We have to reach out to the appendix for the algorithm description. \n\nb) The optimization objective L_{INV} and L_{EQV} are not specified in the main paper either.\n\nc) The response map is mainly introduced by citing a related work. However, given its importance to this paper it is better explained and self-contained.\n\nd) There is not much explanation on the latent traversals.\n\n\n3) Assumptions:\n\nThere is an assumption that the statistical dependencies in the prior are preserved by the response map.  This assumption looks to me very strong. Not sure if such an assumption makes sense for real data.\n", "summary_of_the_paper": "The authors argue that the interventional consistency is a desired property for the representation of auto-encoders and proposed a new metric on it. The authors further develop a new training procedure for auto-encoder that has extra regularization terms to enforce the interventional consistency.", "summary_of_the_review": "I feel the draft is not in a status for publication yet. The authors may need to polish it to have a clear story line. Also it needs to be self-contained.\n\n\n########Update after the authors' rebuttal###########\n\nI would like to thank the authors' explanations on the notations as well as the assumption concerns. Overall I feel this draft has potential to be a good one. As stated in the authors feedback, in its current status more work is needed to make it easy to read. (For example, the algorithm descriptions are still hidden in the appendix.) As such I am keeping my ratings unchanged.  ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper introduces the notion of interventional consistency of a representation learned using autoencoders, which is claimed to be a desirable property for disentanglement. The reviewers agree that the contributions are novel and relevant, but they also found the paper hard to follow due to a lack of clarity and motivation. Further, they considered the underlying assumptions very strong and possibly hard to find practical instances where they may hold (e.g., the assumption that statistical dependencies in the prior are preserved by the response map). The reviewers also noted that some real-world examples showing the interventional consistency would be helpful. 

After all, the paper contains interesting ideas and we would like to encourage the authors to pursue this line of work. Still, the paper in its current form is not ready for publication. We encourage the authors to address the reviewers' comments explicitly in a future version of the manuscript.
Decision: Reject