Year: 2021

Abstract:
Autoencoders have played a crucial role in the field of representation learning since its inception, proving to be a flexible learning scheme able to accommodate various notions of optimality of the representation. The now established idea of disentanglement and the recently popular perspective of causality in representation learning identify modularity and robustness to be essential characteristics of the optimal representation. In this work, we show that the current conceptual tools available to assess the quality of the representation against these criteria (e.g. latent traversals or disentanglement metrics) are inadequate. In this regard, we introduce the notion of interventional consistency of a representation and argue that it is a desirable property of any disentangled representation. We develop a general training scheme for autoencoders that takes into account interventional consistency in the optimality condition. We present empirical evidence toward the validity of the approach on three different autoencoders, namely standard autoencoders (AE), variational autoencoders (VAE) and structural autoencoders (SAE). Another key finding in this work is that differentiating between information and structure in the latent space of autoencoders can increase the modularity and interpretability of the resulting representation.

heading: 1 INTRODUCTION
text: Representation learning is the problem of finding a low dimensional description of the data. The characteristics of a ’good representation’ have long since been a matter of debate, often depending on the context in which the representation has to be employed. Multiple works (cf. Bengio et al. (2013), Schölkopf et al. (2021)) identify modularity, robustness to distribution shifts and interpretability as crucial features of a ’good representation’, motivating the quest toward disentanglement (Bachman et al. (2019), Locatello et al. (2020), Ridgeway & Mozer (2018)) and causal representations (Suter et al. (2019), Träuble et al. (2021)). The insight shared by both these fields is that complex world phenomena arise from the ’rich interaction of many sources’, and thus enable compact descriptions in terms of the basic components participating in their generative process. More formally, the hypothesis is that there exist a set of semantically meaningful variables S and an arbitrarily non-linear function G such that X = G(S1, ..., Sn), where X are the observations.
Autoencoders have played a crucial role in the field of representation learning since its inception. Their success is largely due to their simplicity and surprising effectiveness. Mathematically autoencoders can be represented as the tuple (E : Rd → Rn, D : Rn → Rd). Capacity constraints (e.g. n d) force the latent space to prioritize certain information in the input, thus yielding a useful representation. Additional requirements may be imposed either structurally or through regularisation. Intuitively, consistency means that an input generated by the decoder can be mapped back to the point in the latent space that produced it. The central idea of Cemgil et al. (2020) is making the encoder and the decoder consistent both on the training data P (X) and on the auxiliary observations generated by the decoder P (X ′).
Our contribution. Unsupervised causal representation learning is largely unsolved today. In a recent paper Locatello et al. (2019) prove that ’the unsupervised learning of disentangled representations is fundamentally impossible without inductive biases on both the models and the data’. Motivated by this result, in this work we, investigate the structure and in particular the self-consistency
of the learned autoencoder, rather than focusing on the representation’s relation to the ”groundtruth” factors. In this context, we formulate an inductive bias for autoencoders, that we call interventional consistency, and we link it to the solution space of the disentanglement problem. Additionally, we introduce a new architectural module for the latent space of autoencoders which unifies the disentanglement objective and the, more structured, causal perspective.
2 METHOD
Consider the ’artificial’ generative process implemented by the decoder as X ∼ P (Z)P (X|Z; θ). We express the latent space dynamics in terms of a Structural Causal Model (SCM) (Pearl, 2009) on Z. An SCM is defined by a set of so-called noise variables N = N1, ..., Nn, with a distribution P (N) that factorises, and a set of structural assignments f1, ..., fn of the form:
Zi := fi(PAi, Ni) for i = 1, ..., n, (1)
where PAi refers to the set of direct causes of the variable Zi. The set of directed interactions between causal variables identifies a graph, which is usually assumed to be acyclic (i.e. a DAG). This formulation naturally entails a distribution P (Z) and a corresponding causal factorisation:
P (Z) = ∏ i P (Zi|PAi). (2)
For a thorough introduction to SCM we refer the reader to Peters et al. (2017). We can now apply results from causality to the artificial generative process. In particular, we turn to the ICM principle (Peters et al., 2017), which states that the modules participating in the generative process are independent and can thus be separately manipulated. Let us refer to the composition R = E ◦ D as the response map, as done in Leeb et al. (2021), mapping the latent space Z ⊆ Rn back to itself, i.e. R : Z → Z . Let R(Z ′|Z) be the posterior distribution entailed by the map, with Z,Z ′ being distinct random variables. Furthermore, consider the response aggregate posterior distribution and any factorisation thereof:
P (Z ′) = ∫ P (Z)R(Z ′|Z)dZ = ∏ i P (Z ′i|Z ′j 6=i).
The implications of the ICM principle are trivial if applied to the prior P (Z): modifying P (Zi|PAi) will not affect P (Zj |PAj) for j 6= i. However the consequences get far from trivial if we require the response map to preserve the validity of the principle. Let P̃ (Z) denote the intervention distribution:
P̃ (Z) = P̃ (Zm|P̃Am) ∏ i 6=m P (Zi|PAi). (3)
Then we say that the ICM principle is preserved by the response map if:
P̃ (Z ′) = ∫ P̃ (Z)R(Z ′|Z)dZ = P̃ (Z ′k|Zj 6=k) ∏ i 6=k P (Z ′i|Z ′j 6=i), (4)
where the indicesm and k are not necessarily equivalent, although we will from here on assume their equivalence.1 Intuitively, R preserves the ICM principle if any localised change in P (Z) produces a corresponding localised change in P (Z ′). The idea is illustrated in Figure 1. Crucially, this property mirrors the definition of disentanglement in the context of the artificial generative process: each factor in the representation (P (Z ′k|Z ′j 6=k)) is sensitive to changes in a single generative factor (P (Zm|PAm)), while being relatively invariant to changes elsewhere in the generative process. This
1Allowing m 6= k essentially takes into account the possibility of permutation of the original space dimensions in the representation, which is the norm in disentanglement metrics. By assuming m = k we get rid of one degree of freedom in the problem.
analogy suggests that the condition expressed in Equation 4 has to be satisfied by a representation that disentangles the true generative process. We call this condition interventional consistency.
Interventional consistency can be formulated in terms of invariance and equivariance of the response map. If the autoencoder satisfies interventional consistency, then P (Z ′i|Z ′j<i) is invariant to the action of the group of interventions Im localised on P (Zm|Zj<m) with i 6= m. Moreover, assuming we are able to perform an intervention on the response variables I ′ equivalent to the one performed on the prior I , the response map is equivariant to the intervention, meaning that the order in which the response and the intervention are applied is not important. From this symmetry viewpoint, interventional consistency belongs to causal representations: ’the right causal order is the one invariant to the right kind of interventions’ (Herb Simon, cf. Hoover (2008)).
Importantly, a representation that satisfies the interventional consistency condition does not necessarily solve the identifiability problem (Gresele et al., 2021), as any bijective transformation of the true causal factors would equally satisfy the condition for the correspondingly transformed interventions. Instead, it is manipulable by construction. Without this feature, there is no guarantee the decoder can extrapolate to latent samples not seen during training. As an example, consider the common diagnostic tool of latent traversals: it consists of atomic interventions applied to the different dimensions of the representation. Often a good representation is associated with semantically meaningful traversals in the image space. However, the observed result is only half of the story: if the encoder fails to interpret the generated samples the representation cannot be called disentangled. To illustrate this example, in Figure 2 we show the error recorded on the response to traversals on a standard autoencoder.

heading: 2.1 CONSISTENCY TRAINING
text: It is possible to use the invariance or equivariance formulation to incorporate interventional consistency into the solution space of an autoencoding problem. Let LINV(R) and LEQV(R) be real valued functions measuring the violation of the invariance and equivariance property by the response map R, respectively. Then the space of interventionally consistent autoencoders is defined as:
{(E,D) : LINV(E ◦D) = 0 or LEQV(E ◦D) = 0}.
This hard constraint is replaced with a differentiable regularization by augmenting the optimization objective with LINV or LEQV. Next, we propose evaluation metrics that make the consistency penalties easier to interpret. Algorithm 1 and 2 in the Appendix demonstrate how to compute LINV and LEQV. Importantly, LINV is independent of the network architecture of choice, therefore it can be applied to any existing autoencoder method with latent space N . Meanwhile, LEQV only requires an explicit distinction between noises N and causes Z (of the artificial generative process) outlined in the previous section, which, in our experiments, is implemented by the explicit causal latent block (see 4).
Note some additional mild approximations are necessary to make the penalty terms tractable. Firstly, we assume that the response aggregate posterior P (Z ′) factorises like P (Z), i.e. that the statistical dependencies in the prior are preserved by the response map. When using an autoencoder with explicit structural mappings this assumption is equivalent to assuming that the response aggregate noise posterior P (N ′) factorises. Secondly, we employ Monte Carlo sampling to estimate the consistency by sampling hard interventions on the noise variable distribution of the form: P̃ (Nm) ← δ(v), with v from the aggregate marginal posterior Q(Nm) = EXQ(Nm|X). Finally, in order to assess equivariance we treat I ′ = I , which relies on the equivalence between Z and Z ′. This assumption is satisfied as long as the reconstructions are sufficiently similar to corresponding the training samples (i.e. the autoencoder fidelity is high).
Below, Nk denotes a sample from the prior P (N) and Ñk its intervened version. Moreover we let N̄ ′k and ¯̃Nk′ be the expectation of the response to Nk and Ñk, respectively. Let ˆ̄Nk′ be the result of the intervention I applied to N̄ ′k and we denote the corresponding causal variables by ˆ̄Z ′k and ¯̃Zk .
Invariance score. Let UIm be a measure of the magnitude of the intervention Im and σi the standard deviation of each dimension in the response space. Then we define the invariance error in response to the intervention Im recorded on the dimension i as
eIm,i = 1 UImσi EN [ ||(N̄ ′ − ¯̃N ′Im)i||2 ] .
More precisely we define UIm := EN [ ||(N − ÑIm)i||2 ] . Consequently we score the invariance of the i-th dimension with respect to interventions on the m-th dimension as:
INV[m, i] = 1− EIm∼Im [ eIm,i ] . (5)
We approximate all the expectations with their Monte Carlo estimates. According to this formulation, a perfectly invariant response map would score 1 on the off-diagonal elements of the matrix. Notice that invariance is not concerned with the diagonal entries of the matrix. However, we can interpret a high invariance score on the i-th diagonal element as a sign of the unresponsiveness of the dimension as it implies interventions on i are not registered in the response. We henceforth refer to these unresponsive latent dimensions with the appellative ”collapsed”, in reference to the well-known effect of posterior collapse (rather common in βVAE models).
Equivariance score. Similarly, we define the equivariance error and equivariance score as :
rIm,i = 1 U EN [ ||( ˆ̄Z ′Im − ¯̃Z ′Im)i||2 ]
EQV[m, i] = 1− EIm∼Im [ rIm,i ] .
In the case of equivariance perfect consistency corresponds to all the entries being 1. Notice however that high equivariance is also obtained by the trivial solution (i.e. a constant encoding map). This is due to the fact that the intervention values are sampled from the aggregate posterior.
Self-consistency score. Finally, the self-consistency score evaluates the response inner consistency under no intervention. More specifically, we measure the amount of error introduced by the response on average over the prior. In formula:
SCN[i] = 1− 1 σi
EN [ ||(N̄ ′ −N)i||2 ] Importantly, self-consistency is the only score relating the prior and the response space and is closely related to the regularization used by AVAEs (Cemgil et al., 2020).
2.2 EXPLICIT CAUSAL LATENT BLOCK
Inspired by the distinction between noise and structure in the SCM we insert a similar bias into the latent space of autoencoders by directly joining the representation layer with a structural mixing block as shown in Figure 4. We henceforth will refer to the botteneck layer as noise terms or N , and to the units produced by the subsequent mixing layer as causes orZ. EachZi is obtained from its corresponding noise term Ni and its predecessors Z<i through a learned nonlinear function fi, much like in an SCM (implemented as an MLP). To increase the sparsity of the structure we apply a learnable mask Mi ∈ R(i−1) to the parents , such that Zi = fi(Mi Z<i−1, Ni). The masks are parametrised through the Gumbel trick (Jang et al., 2016) to induce weight sharpening, and thus act as a gating mechanisms of the information provided by the predecessors.
We name this extension of the representation layer ”explicit causal latent block” (XBlock) and we call XNet (or prefix ”X”) any network using this module. Most disentanglement methods attempt to learn a representation consisting of causal variables or disentangled ”factors” which are statistically independent (Higgins et al., 2017a). However, except in the trivial case, it is not the Zi that should be treated as statistically independent, but the Ni. Consequently, the explicit separation between noise and structure reconciles with the conventional disentanglement perspective while also providing the model with additional power to learn identifiable relationships between the latent variables terms. The causal links revealed by Mi can then be used to help identify how interventions will affect the resulting samples for downstream tasks such as controllable generation. Moreover, differentiating between noise and structure allows to expand the dimensionality of the causal variables Z, without any need to reparametrise the noise distribution. We partition the causes in units, each one consisting of multiple dimensions and representing a single statistical variable.

heading: 3 RELATED WORK
text: To the best of our knowledge, this is the first work to formally introduce the notion of interventional consistency and develop a training scheme based on it. However, related ideas have been discussed in previous works.
Recently, there is particular interest in applying causality to representation learning (Kocaoglu et al. (2017), Suter et al. (2019), Träuble et al. (2021), Schölkopf et al. (2021) ). In Yang et al. (2021) the authors introduce a learnable Causal Layer, which essentially describes a SCM, in the latent space of a variational autoencoder. Much like our XBlock, the causal layer in Yang et al. (2021) transforms independent noise variables into causal ones, thus separating the informa-
tion from the structure. The structural assignments of the Causal Layer integrate physical limitations to the system’s dynamics that are not present in the XBlock (i.e. additive noise assumption, mild non-linearity). Moreover, Yang et al. adopt a perspective radically different from this work: their goal is to solve causal disentanglement, and they rely on weak supervision to do so.
The introduction of symmetries in the representation layer has been examined from multiple perspectives (Cohen & Welling (2016), Finzi et al. (2020), Finzi et al. (2021), Weiler et al. (2018), Benton et al. (2020), ?). In Keller & Welling (2021) the authors use topographic capsules parametrised through variational inference to obtain equivariance to geometric transformations in the input. Like Yang et al. (2021), the work uses weak supervision (in the form of ordered input sequences) to disentangle salient characteristics in the data. Interestingly, in Keller & Welling (2021) disentanglement is obtained as a by-product of equivariance, echoing the definition of disentanglement given in Higgins et al. (2018).
Perhaps the discussion most relevant to the present work is presented by Leeb et al. (2021) and Besserve et al. (2018). Leeb et al. develop a framework to assess the characteristics of the response map of variational autoencoders, thus laying the groundwork of the present study. The invariance score proposed in Section 2.1 is closely related to the Latent Response Matrix presented in Leeb et al. (2021). Unlike this paper, in Leeb et al. additionally introduce new tools to evaluate representations in regards to disentanglement based on the response map. In Besserve et al. (2018), interventions on the deep layers of a generative model are employed to discover independent modules participating in the representation. Similarly to the present work, Besserve et al. (2018) adopts a perspective completely agnostic of the true generative process instead focusing on the artificial one. However both Leeb et al. and Besserve et al. are only concerned with analyzing and evaluating generative processes, while we develop training methods and metrics to directly optimize for the desired structure.

heading: 4 EXPERIMENTS
text: In the following experiments we demonstrate the relevance of interventional consistency and the strength of the proposed approach. More specifically, we are able to learn representations that are markedly more robust to interventions and can correctly handle manipulations. We quantitatively and qualitatively verify that consistency training is effective in improving the model’s interventional consistency without harming the model’s generative performance. Moreover we produce evidence on the benefit of differentiating between information and structure in the latent space, by analysing the effects of the XBlock on the learned representation.

heading: 4.1 EXPERIMENTAL DETAILS
text: In our experiments we employ three baselines with fundamental differences in their representation layer, namely: convolutional autoencoders (AE) (Bourlard & Kamp (1988), Hinton & Zemel (1994)), variational autoencoders (βVAE) (Higgins et al., 2017b), and structural autoencoders (SAE) (Leeb et al., 2020). For each of the baselines two variants are considered: ’X’ and ’XC’, where the former extends the original model by integrating the explicit causal latent block and the latter augments the ’X’ version through consistency training. The ’XC’ variant in turn comes in two modalities employing the invariance (’XCI’) and equivariance (’XCE’) regularisation respectively. Additionally, we include an implementation of the AVAE model to compare the two consistency training schemes.
The experiments are performed on MNIST (LeCun & Cortes, 2010) and CelebA (Liu et al., 2015) datasets. These datasets have been chosen for being closer to real-world scenarios than artificially manufactured alternatives whilst being well known and historically recognised in the literature.
The neural architectures considered vary only in size, keeping a coherent overall skeleton across experiments. Specifically, we consider 3 different size standards for MNIST (’S’, ’Standard’, ’S4’) and another 2 for CelebA (’v32’, ’v324’). The differences are reported in Table 3 in the appendix, together with the details on the skeleton and training hyperparamters.

heading: 5 RESULTS AND DISCUSSION
text: (a) FID scores for all the models on MNIST (lower is better). We evaluate the FID on the reconstructions (left), prior samples (middle) and traversals (right). Perhaps unsurprisingly, the variational baseline scores markedly higher than the alternative. Crucially, the ’X’ and ’XC’ versions’ scores are comparable to the Standard models. (b) Summary of the invariance score on all experiments (MNIST and CelebA). For each model, all its variants are evaluated against the invariance score. The matrix is reduced to a single number through averaging. The variational baseline scores systematically higher than the alternatives. This result is partly due to the presence of collapsed dimensions in the representation, which increase the model’s score due to their absolute unresponsiveness. Importantly, we notice that the ’XC’ variants tends to perform better than the others.
Consistency training improves the model’s interventional consistency without harming the reconstruction quality. From Figure 7 and 6 it can be seen across all datasets that the ’X’ and ’XC’ variants do not interfere with the quality of the reconstructed or prior samples. To quantitatively support this claim we include the corresponding FID scores (Heusel et al., 2017) for all the experiments in Figure 5a. Additional samples are shown in Section A.4.1. One principle against which to evaluate the quality of the prior samples is their intrinsic diversity. Real world datasets such as MNIST and CelebA are rich in low probability details that are thus hard to capture in the representation. In this regard the autoencoder outperforms the respective VAE models (see A.4.1). The prior samples for deterministic autoencoders are obtained through hybrid sampling, a technique proposed in (Leeb et al. (2020), Besserve et al. (2018)) which allows to generate previously unseen combinations of latent factors by randomly aggregating different posterior samples. For more details on hybrid sampling we refer the reader to Leeb et al..
Figure 2 shows the standard latent traversal plot alongside the traversals response error. The error is measured as the signed difference between the intervened prior noise vector and its expected response as defined in Section 2.1, i.e. Ñki − Ñ ′ki . This figure best depicts the discrepancy between generation and inference which lies at the heart of this work and motivates the consistency perspective. By only analysing the traversal plots one might conclude that the learned representation can correctly resolve the alteration introduced by the traversal, and thus that it possesses the generalisation and modularity suggested by the quality of the generated samples. In short, consistency is assumed to hold. However, the errors recorded on the response show that this assumption is false: the intervention applied is not preserved by the encoder, which presumably projects the sample into the observational distribution manifold (Radhakrishnan et al., 2018). In summary Figure 2 reveals that (i) the current autoencoders (i.e. without consistency requirements) are indeed not consistent under interventions and that (ii) consistency training can improve the model’s consistency. Significantly, consistency training only acts on the response error indirectly since the regularisation terms do not include take into account the prior samples. The increase in consistency with consistency training can also be assessed in Figure 3. It must be taken into account that the invariance regularisation directly aims at optimising the invariance score, whilst the equivariance score only does so indirectly. Full summaries of the consistency scores for the different training modalities can be found in Section A.4.3. A quantitative summary of the invariance scores for all experiments is given in Figure 5b. We observe that the AVAE baseline has the lowest score on interventional consistency among the variational family, comparably to the βVAE. On the contrary it is among the highest
performing models when evaluated on self-consistencty (see Figure 28). This discrepancy confirms that consistency, as defined in Cemgil et al. (2020) does not imply interventional consistency.
The explicit causal latent block provides sparsity and self-consistency. A mask element with a low value prevents the information from passing from the parent to the child node. Consequently, if the noise variables are independent, one can read off the adjacency matrix the structure of the existing statistical relations between latent units. This is not possible in the current standard of a single space carrying both the information and the statistical structure. A sparse configuration is desirable for interpretability, since it simplifies the dynamics of the generation process. Moreover simpler models are more likely to be correct, according to the widely accepted principle of parsimony also known as Occam’s razor. Section A.4.4 in the appendix displays the mask values for the learned XBlock of different models.
Table 1 reports the sparsity levels for different latent sizes. The results show that it is possible to obtain sparsity for every model without the need of any additional regularisation. The introduction of learnable masks endowed with weight sharpening has proved to be sufficient to induce sparsity
in the latent block structure. The number of latent dimension has systematically turn out to be the only indicator for the sparsity level achieved. Moreover, the structure of dependencies in the model varied only slightly with respect to the model family and exhibited greater sensitivity to the latent unit size (cf A.4.4).
Table 2 records the self consistency of autoencoders under three different prior sampling procedures: posterior, hybrid and uniform sampling, respectively from top to bottom. ’Posterior’ refers to drawing samples from the aggregate posterior Q(N) = ∫ Q(N |X)P (X = x)dx, ’hybrid’ applies to its hybridised version, and finally in the ’uniform’ case we consider a uniform distribution whose extremes on each dimension are derived from Q(N). We would normally expect a decreasing self-consistency going from the first to the last row: in fact, hybrid sampling breaks any existing statistical correlation between the noise variables, and uniform sampling additionally compels a continuous support in the latent data manifold to be preserved by the response map. In Table 2 the expected trend is approximately confirmed by the AE model. We notice that the ’X’ version improves the self consistency score under all sampling procedures, proving to be less sensitive to the distribution shifts introduced by hybridisation and continuity. Consistency training further increases the score under all sampling modes. This behaviour suggests the hypothesis that the explicit causal latent block in fact favours independence between the noise dimensions.
In order to give an objective ground to this hypothesis in the additional material (Figure 8) we look at an empirical measure of independence between the noise variables for all models. More specifically, we compute the Maximum Mean Discrepancy (MMD) between the aggregate posterior distribution over the noise variables and its hybridised version. By construction hybridisation destroys any existing statistical relationship between the latent dimensions. For both the AE and VAE model families the ’X’ version’s metric distribution appears significantly shifted to the left, thus indicating a higher degree of independence between the noise variables.

heading: 6 CONCLUSION
text: In this work we focus on the modularity and self-consistency of a representation, rather than addressing the identifiability problem. More specifically, we have formulated and studied the interventional consistency property of autoencoders. The metrics and regularization objectives we develop result in representations that remain consistent under interventions thus enabling meaningful manipulation. We propose a novel architecture that can be applied to any existing autoencoder, inspired by causal representation learning, to explicitly model the dependencies between latent variables. The architecture has empirically shown to improve the model’s self-consistency and independence in the latent space. The separation of the information and structure of the representation therein connects the disentanglement perspective to the more structured causal framework, thereby also opening the door to new analysis tools to enrich the representation’s interpretability and controllability by exposing the learned mechanisms.

heading: REPRODUCIBILITY STATEMENT
text: The used datasets are publicly available. Implementation details for all the experiments are included in the Additional material (A.3). A description of the hyperparameters and network architectures used is included in Appendix A.3. As indicated as an option in the author guide https: //iclr.cc/Conferences/2021/AuthorGuide for sharing the code anonymously, in a first step we will make a comment directly to the reviewers and area chairs with the link to an anonymous repository once the discussion forums open and in the second step make the full repository public once the paper is accepted.

heading: A APPENDIX

heading: A.1 NOTATION
text: In this section we report a guide to the notation used in the work. Throughout this paper we use N and Z for stochastic variables in the latent space. N ′ and Z ′ are used for stochastic variables in the response space. X is be used for input data and X for the input space. (E,D) denote the encoder and decoder mapping, respectively. R = E ◦D defines the response. Nk denotes a sample from the prior P (N) and Ñk its intervened version. Moreover we let N̄ ′k and ¯̃N ′k be the expectation of the
response to Nk and Ñk, respectively. For a Dirac posterior (deterministic map) we use its center as expectation. Finally, ˆ̄Nk′ marks the result of the intervention I applied to N̄ ′k and we denote the corresponding causal variables by ˆ̄Z ′k and ¯̃Zk .

heading: A.2 CONSISTENCY TRAINING
text: In this Appendix we take a closer look at the connection between interventional consistency and the pointwise estimate of invariance and equivariance. Interventional consistency can be formalised with the two following definitions.
Definition 1 (Interventions Invariant Response). Given an autoencoder (E,D), and a causal factorisation of the prior over the latent space P (Z) = ∏ i P (Zi|PAi) the response R = E ◦ D is invariant to interventions on Z if for every intervention I in the set of interventions Im localised on the factor indexed by m the following conditions are verified:
P (Z ′) = ∫ P (Z)R(Z ′|Z)dZ = ∏ i P (Z ′i|Z ′j 6=i) (6)
P̃ (Z ′) = ∫ [I ◦ P (Z)]R(Z ′|Z)dZ = P̃ (Z ′m|Z̃ ′j 6=m) ∏ i6=m P (Z ′i|Z ′j 6=i) (7)
Definition 2 (Interventions Equivariant Response). Given an autoencoder (E,D), and a causal factorisation of the prior over the latent space P (Z) = ∏ i P (Zi|PAi) the response R = E ◦D is equivariant to interventions on Z if, for every intervention I in the set of interventions Im localised on the factor indexed by m, there exist an equivalent intervention I ′ acting in the Z ′ space, such that the following condition is verified:∫
[I ◦ P (Zm|ZP̃Am)] ∏ i6=m P (Zi|PAi)R(Z ′|Z)dZ = I ′ ◦ P (Z ′m|Z ′j 6=m) ∏ i 6=m P (Z ′i|Z ′j 6=i) (8)
Equivariance implies invariance, as it provides a strictly stronger condition on the response map. The caveat is that to evaluate equivariance one must additionally deduce I ′ from I , which requires
knowledge of the relation between Z and Z ′. Notice that interventional consistency is effectively a property of the response map and is thus oblivious to the prior Z.
Due to the deterministic nature of the structural assignments definitions 1 and 2 can be reformulated without any loss of generality in terms of the noise variables. Let N and N ′ be the noise variables before and after the response respectively, with
P (N ′) = ∫ P (N)R(N ′|N)dN (9)
P̃ (N ′) = ∫ P̃ (N)R(N ′|N)dN = ∫ δ(Nm − v) ∏ i 6=m P (Ni)R(N ′|N)dN (10)
In the invariance case we require P (N ′)i to match P̃ (N ′)i for all i 6= m. In the equivariance case we additionally require P̃ (N ′)m = I ′ ◦ P (N ′)m = I ◦ P (N ′)m = δ(v). In formulas, the solution space has to satisfy one of the following conditions:
d(P̃ (N ′)i, P (N ′)i) = 0 ∀i 6= m (invariance) (11)
d(P̃ (N ′)i, I ◦ P (N ′)i) = 0 ∀i ∈ [1, n] (equivariance), (12)
where I ◦ P (N ′)i = P (N ′)i for i 6= m and d is a generic distance in the space of distributions on N . We can approximate P (N ′) and P̃ (N ′) by sampling from the prior (i.e. P (N) and P̃ (N) respectively) and subsequently applying the response map to each sample. Using a pointwise estimate of the above terms can be viewed as adopting a distance that to decompose into the sum of the intrasample distances. This choice rules out solutions for which the response posterior suffers a shift in a non-intervened dimension when an intervention is applied. We now turn to a full characterisation of the pointwise estimates of invariance and equivariance.
Pointwise invariance. We measure the Euclidean distance between the responses expectations, i.e. ||N̄ ′ki −
¯̃Nki ||2 for i = 1, ..., n and k = 1, ..., T , with T being the total number of samples. By focusing on the noise terms we can leverage the noise independence assumption to significantly simplify the equations. Notice that we couldn’t apply the same direct measurement with the causal variables Z̄ ′i
k and ¯̃Z ′ki because of the existing connections between causal variables, i.e. when modifying Nm we can expect N̄ ′i 6=m to stay constant but we cannot say the same for Z̄ ′ i 6=m if i is a descendant of m. However, invariance can be checked distributionally for Z, by looking at the conditionals P (Z ′i|Z ′i 6=m).
Pointwise equivariance. To compute pointwise equivariance we can directly work in the causal variables space measuring the Euclidean distance || ˆ̄Z ′ki −
¯̃Zki ||2. The argument for evaluating equivariance at the causal variables level is purely practical: in this way the structural assignments receive a stronger training signal, which is beneficial especially early on in training when the set of connections is being first outlined.
Algorithms 1 and 2 show how to compute pointwise invariance and equivariance, respectively.
Algorithm 1 Algorithm computing the invariance regularisation term. for m in [1, n] do
for Im in Īm ⊆ Īm do Sample N1, ..., NT ∼ P (N) Copy Ñk ← Nk for k = 1, ..., T Intervene Nkm ← vIm eIm ← 1n·T ∑ k ∑ i 6=m ||N̄ ′ki −
¯̃N ′ki ||2 end
end LINV = 1|Īm| ∑ Im∈Īm eIm
Algorithm 2 Algorithm computing the equivariance regularisation term. for m in [1, n] do
for Im in Īm ⊆ Īm do Sample N1, ..., NT ∼ P (N) Copy Ñk ← Nk Intervene Nkm ← vIm Copy ˆ̄N ′k ← N̄ ′k
Intervene ˆ̄N ′km ← vIm ˆ̄Z ′ki ← fi( ˆ̄Nki , PAki ) ¯̃Z ′ k i ← fi( ¯̃N ′ k i , PA k i )
rIm ← 1n·T ∑ k ∑ i 6=m || ˆ̄Z ′ki − ¯̃Z ′ k
i || end
end LEQV = 1|Īm| ∑ Im∈Īm rIm
The regularisation terms LINV and LEQV are included in the standard loss weighted by λINV and λEQV, respectively. In our experiments we use λINV = λEQV = 4. We consider both stochastic and deterministic encoders, while treating the decoder as deterministic (a Gaussian noise model is used, with the mean parametrised by D). If the encoder is stochastic the response R(N ′|N) defines a distribution over the N ′ space: in the case of the VAE this distribution is known to be Gaussian, thus fully described by its mean and variance. Experimentally, including information on the spread of the distribution in the regularisation term has not benefited training, often resulting in numerical instabilities. This outcome has been observed for parametric distances as well as wellknown divergence measures (such as the Kullback–Leibler).

heading: A.3 EXPERIMENTAL DETAILS
text: In this section we provide the details regarding experimental settings.
All experiments are performed on 1 GPU core NVIDIA A100 40GB and based on the Pytorch (Paszke et al., 2017) framework. The models are trained using the stochastic iterative optimiser Adam (Kingma & Ba, 2014), with default parameters. The learning rate is set to 0.0001 and halved every 30 epochs. The MNIST and CelebA models are trained for 100k and 200k iterations, respectively. We use a batch size of 128 or all the experiments. Each model is trained with 5 different random seeds (11, 13, 17, 37, 121) and every pseudo-random number generator in pytorch, numpy and python.random is seeded accordingly.
As anticipated in 4.1 we adopt a single architectural skeleton for all the models, varying only in size. The skeleton is shown in Figure 10. All the architectures use the Mish (Mish, 2020) nonlinearity. The number of pooling layers is set to 3 for MNIST and 7 for CelebA. The number of filters in the convolutional layers is fixed to 32 for MNIST, 100 for CelebA. All the layers, except for the normalization ones, are initialised using the method discussed in Glorot & Bengio (2010). Differently from what is shown in Figure 10 the ’X’ and ’XC’ variants include an additional layer (forming the explicit causal latent block) applied to the latent vector as shown in Figure 4. Finally, in Table 4 we report the size of each model in terms of the number of trainable parameters.

heading: A.4 ADDITIONAL RESULTS

heading: A.4.1 COMPARING SAMPLES QUALITY
text: In this section we report the reconstructions, generated and traversal samples for the main experiments performed on MNIST and CelebA.

heading: SIZE NAME NUMBER of

heading: A.4.2 TRAVERSAL RESPONSES.
text: We here report the complete traversal response plots for the S version of the MNIST experiments. Each plot shows the errors registered on all the latent dimensions as the traversal is performed.
βVAE family traversal response errors. The models in βVAE family are exposed to the wellknown effect of posterior collapse. In the following plots collapsed dimensions can be distinguished by the particular behaviour of the response error (zero when the dimension is not intervened on and equivalent to the intervention value elsewhere). We also include the traversal response errors measured on the AVAE model. Interstingly, all the variants in the VAE family present two collapsed dimensions, except for the XVAE model, that has 3 collapsed dimensions (50% of the total number in this case) and the AVE model which has none.
AE family traversal response errors. Additionally we show the traversal response errors for the autoencoder family of models. Notice how in this case, as for the variational case, consistency training effectively decreases the error.

heading: A.4.3 CONSISTENCY METRICS
text: In this sections we report the results on the new consistency metrics for all the models. We make the following key observations: (i) that consistency training generally strengthens the models’ invariance and equivariance, (ii) that although equivariance implies invariance, models using invariance regularisation have comparable performances to those using equivariance regularisation.
(a) Invariance score for all experiments comparing performance of models with and without consistency training. (b) Invariance score for all the models with consistency training comparing invariance-based and equivariance-based regularisation.
(a) Equivariance score comparing performance of models with and without consistency training. (b) Equivariance score for all the models with consistency training comparing invariance-based and equivariance-based regularisation.
We proceed to present the self-consistency scores for all the experiments. Notice that selfconsistency is the only metric that, similarly to Cemgil et al. (2020), directly compares the response to the prior. Since, the AVAE model objective is conceptually analogous to self-consistency, we would expect the AVAE model to score higher than the VAE baseline on this consistency metric. Figure 28 confirms this prediction.
A.4.4 XNETS
Masks structure. We can look at the structure learned in the explicit causal latent blocks. The mask values in the Xblock describe a weighted adjacency matrix, which is displayed by the following heatmaps. Interestingly, the structure is not significantly affected by the training scheme adopted (i.e. consistency regularisation) or the model family. Instead its variability is mostly linked to the causal units size and latent size.
(a) XAE S (b) XAE S4
(c) XCAE-I S (d) XCAE-I S4
(e) XCAE-E S (f) XCAE-E 4S
(a) XAE standard
(b) XCAE-I Standard
(a) XCAE-E Standard
(a) XSAE S (b) XCSAE-I S
(c) XVAE S (d) XCVAE-I S
XBlock visualisation tools. We introduce a qualitative evaluation tool reliant on interventional consistency that permits to interrogate each representation dimension with specific examples. We
produce hybrids between two or more prior samples by separately intervening on different dimensions. More precisely, given a base N1 ∼ P (N) and an alternative N2 ∼ P (N), we obtain n hybrids by applying the following interventions one at a time:
N1i ← N2i for i = 1, ..., n
By decoding the result noise vector we obtain a visualisation of the hybridisation: we can thus visually assess the effect of a localised intervention on the representation. Examples are given in Figure 34.
Many dimensions do not show any change in response to the intervention. A lack of reaction could be due to a general insensitivity of the dimension (e.g. the case of a collapsed unit), or to an incidental overlapping of the two samples, or even to an inability of the decoder to generalise outside of support. Moreover for the ’X’ and ’XC’ model variants we observe that the change produced by the interventions is semantically different on distinct latent dimensions. This suggests that the representation has developed a certain degree of modularity.
We put consistency training to test by simultaneously applying two interventions on the base sample using two different alternatives. Additionally, we look at the result both when the interventions are applied on the noise and on the causal variables. Applying an intervention on the causes breaks the connection from the parent to the intervened unit, thus altering the statistical relations between causal variables. The results are shown in Figure 35. There is a noticeable difference in the quality of the hybrid samples between the two kinds of interventions: hybrids on the causal variables more often do not appear connected to any of the prior samples, sometimes producing meaningless shapes (e.g. element (4, 1) in the matrix on the right). On the other hand, the noise hybrid samples appear convincing and mostly semantically homogeneous. Interestingly, as highlighted in orange in Figure 35, the two different interventions can provide two different interpretations of the original ambiguous shape.

Review summaries:
The paper proposes to use interventional consistency to regularise representation learning in VAEs. The idea is well-motivated by the ICM principle and theoretically justified. The paper suggests to use interventional consistency for both training and evaluation of the representation learnt by VAEs. Results show that the proposed idea can give more modular and interpretable representation.

This work proposes the notion of "interventional consistency" as a beneficial property learned representations should have and introduces a regularization term to enforce it in autoencoders. Moreover, the authors introduce an "explicit latent causal block" that allows learning a structural causal model (SCM) over the latent factors. The experimental section argues that both the regularization and the "causal layer" improve interventional consistency.