On the interventional consistency of autoencoders

1 INTRODUCTION . Representation learning is the problem of finding a low dimensional description of the data . The characteristics of a ’ good representation ’ have long since been a matter of debate , often depending on the context in which the representation has to be employed . Multiple works ( cf . Bengio et al . ( 2013 ) , Schölkopf et al . ( 2021 ) ) identify modularity , robustness to distribution shifts and interpretability as crucial features of a ’ good representation ’ , motivating the quest toward disentanglement ( Bachman et al . ( 2019 ) , Locatello et al . ( 2020 ) , Ridgeway & Mozer ( 2018 ) ) and causal representations ( Suter et al . ( 2019 ) , Träuble et al . ( 2021 ) ) . The insight shared by both these fields is that complex world phenomena arise from the ’ rich interaction of many sources ’ , and thus enable compact descriptions in terms of the basic components participating in their generative process . More formally , the hypothesis is that there exist a set of semantically meaningful variables S and an arbitrarily non-linear function G such that X = G ( S1 , ... , Sn ) , where X are the observations . Autoencoders have played a crucial role in the field of representation learning since its inception . Their success is largely due to their simplicity and surprising effectiveness . Mathematically autoencoders can be represented as the tuple ( E : Rd → Rn , D : Rn → Rd ) . Capacity constraints ( e.g . n d ) force the latent space to prioritize certain information in the input , thus yielding a useful representation . Additional requirements may be imposed either structurally or through regularisation . Intuitively , consistency means that an input generated by the decoder can be mapped back to the point in the latent space that produced it . The central idea of Cemgil et al . ( 2020 ) is making the encoder and the decoder consistent both on the training data P ( X ) and on the auxiliary observations generated by the decoder P ( X ′ ) . Our contribution . Unsupervised causal representation learning is largely unsolved today . In a recent paper Locatello et al . ( 2019 ) prove that ’ the unsupervised learning of disentangled representations is fundamentally impossible without inductive biases on both the models and the data ’ . Motivated by this result , in this work we , investigate the structure and in particular the self-consistency of the learned autoencoder , rather than focusing on the representation ’ s relation to the ” groundtruth ” factors . In this context , we formulate an inductive bias for autoencoders , that we call interventional consistency , and we link it to the solution space of the disentanglement problem . Additionally , we introduce a new architectural module for the latent space of autoencoders which unifies the disentanglement objective and the , more structured , causal perspective . 2 METHOD Consider the ’ artificial ’ generative process implemented by the decoder as X ∼ P ( Z ) P ( X|Z ; θ ) . We express the latent space dynamics in terms of a Structural Causal Model ( SCM ) ( Pearl , 2009 ) on Z . An SCM is defined by a set of so-called noise variables N = N1 , ... , Nn , with a distribution P ( N ) that factorises , and a set of structural assignments f1 , ... , fn of the form : Zi : = fi ( PAi , Ni ) for i = 1 , ... , n , ( 1 ) where PAi refers to the set of direct causes of the variable Zi . The set of directed interactions between causal variables identifies a graph , which is usually assumed to be acyclic ( i.e . a DAG ) . This formulation naturally entails a distribution P ( Z ) and a corresponding causal factorisation : P ( Z ) = ∏ i P ( Zi|PAi ) . ( 2 ) For a thorough introduction to SCM we refer the reader to Peters et al . ( 2017 ) . We can now apply results from causality to the artificial generative process . In particular , we turn to the ICM principle ( Peters et al. , 2017 ) , which states that the modules participating in the generative process are independent and can thus be separately manipulated . Let us refer to the composition R = E ◦ D as the response map , as done in Leeb et al . ( 2021 ) , mapping the latent space Z ⊆ Rn back to itself , i.e . R : Z → Z . Let R ( Z ′|Z ) be the posterior distribution entailed by the map , with Z , Z ′ being distinct random variables . Furthermore , consider the response aggregate posterior distribution and any factorisation thereof : P ( Z ′ ) = ∫ P ( Z ) R ( Z ′|Z ) dZ = ∏ i P ( Z ′i|Z ′j 6=i ) . The implications of the ICM principle are trivial if applied to the prior P ( Z ) : modifying P ( Zi|PAi ) will not affect P ( Zj |PAj ) for j 6= i . However the consequences get far from trivial if we require the response map to preserve the validity of the principle . Let P̃ ( Z ) denote the intervention distribution : P̃ ( Z ) = P̃ ( Zm|P̃Am ) ∏ i 6=m P ( Zi|PAi ) . ( 3 ) Then we say that the ICM principle is preserved by the response map if : P̃ ( Z ′ ) = ∫ P̃ ( Z ) R ( Z ′|Z ) dZ = P̃ ( Z ′k|Zj 6=k ) ∏ i 6=k P ( Z ′i|Z ′j 6=i ) , ( 4 ) where the indicesm and k are not necessarily equivalent , although we will from here on assume their equivalence.1 Intuitively , R preserves the ICM principle if any localised change in P ( Z ) produces a corresponding localised change in P ( Z ′ ) . The idea is illustrated in Figure 1 . Crucially , this property mirrors the definition of disentanglement in the context of the artificial generative process : each factor in the representation ( P ( Z ′k|Z ′j 6=k ) ) is sensitive to changes in a single generative factor ( P ( Zm|PAm ) ) , while being relatively invariant to changes elsewhere in the generative process . This 1Allowing m 6= k essentially takes into account the possibility of permutation of the original space dimensions in the representation , which is the norm in disentanglement metrics . By assuming m = k we get rid of one degree of freedom in the problem . analogy suggests that the condition expressed in Equation 4 has to be satisfied by a representation that disentangles the true generative process . We call this condition interventional consistency . Interventional consistency can be formulated in terms of invariance and equivariance of the response map . If the autoencoder satisfies interventional consistency , then P ( Z ′i|Z ′j < i ) is invariant to the action of the group of interventions Im localised on P ( Zm|Zj < m ) with i 6= m. Moreover , assuming we are able to perform an intervention on the response variables I ′ equivalent to the one performed on the prior I , the response map is equivariant to the intervention , meaning that the order in which the response and the intervention are applied is not important . From this symmetry viewpoint , interventional consistency belongs to causal representations : ’ the right causal order is the one invariant to the right kind of interventions ’ ( Herb Simon , cf . Hoover ( 2008 ) ) . Importantly , a representation that satisfies the interventional consistency condition does not necessarily solve the identifiability problem ( Gresele et al. , 2021 ) , as any bijective transformation of the true causal factors would equally satisfy the condition for the correspondingly transformed interventions . Instead , it is manipulable by construction . Without this feature , there is no guarantee the decoder can extrapolate to latent samples not seen during training . As an example , consider the common diagnostic tool of latent traversals : it consists of atomic interventions applied to the different dimensions of the representation . Often a good representation is associated with semantically meaningful traversals in the image space . However , the observed result is only half of the story : if the encoder fails to interpret the generated samples the representation can not be called disentangled . To illustrate this example , in Figure 2 we show the error recorded on the response to traversals on a standard autoencoder . 2.1 CONSISTENCY TRAINING . It is possible to use the invariance or equivariance formulation to incorporate interventional consistency into the solution space of an autoencoding problem . Let LINV ( R ) and LEQV ( R ) be real valued functions measuring the violation of the invariance and equivariance property by the response map R , respectively . Then the space of interventionally consistent autoencoders is defined as : { ( E , D ) : LINV ( E ◦D ) = 0 or LEQV ( E ◦D ) = 0 } . This hard constraint is replaced with a differentiable regularization by augmenting the optimization objective with LINV or LEQV . Next , we propose evaluation metrics that make the consistency penalties easier to interpret . Algorithm 1 and 2 in the Appendix demonstrate how to compute LINV and LEQV . Importantly , LINV is independent of the network architecture of choice , therefore it can be applied to any existing autoencoder method with latent space N . Meanwhile , LEQV only requires an explicit distinction between noises N and causes Z ( of the artificial generative process ) outlined in the previous section , which , in our experiments , is implemented by the explicit causal latent block ( see 4 ) . Note some additional mild approximations are necessary to make the penalty terms tractable . Firstly , we assume that the response aggregate posterior P ( Z ′ ) factorises like P ( Z ) , i.e . that the statistical dependencies in the prior are preserved by the response map . When using an autoencoder with explicit structural mappings this assumption is equivalent to assuming that the response aggregate noise posterior P ( N ′ ) factorises . Secondly , we employ Monte Carlo sampling to estimate the consistency by sampling hard interventions on the noise variable distribution of the form : P̃ ( Nm ) ← δ ( v ) , with v from the aggregate marginal posterior Q ( Nm ) = EXQ ( Nm|X ) . Finally , in order to assess equivariance we treat I ′ = I , which relies on the equivalence between Z and Z ′ . This assumption is satisfied as long as the reconstructions are sufficiently similar to corresponding the training samples ( i.e . the autoencoder fidelity is high ) . Below , Nk denotes a sample from the prior P ( N ) and Ñk its intervened version . Moreover we let N̄ ′k and ¯̃Nk′ be the expectation of the response to Nk and Ñk , respectively . Let ˆ̄Nk′ be the result of the intervention I applied to N̄ ′k and we denote the corresponding causal variables by ˆ̄Z ′k and ¯̃Zk . Invariance score . Let UIm be a measure of the magnitude of the intervention Im and σi the standard deviation of each dimension in the response space . Then we define the invariance error in response to the intervention Im recorded on the dimension i as eIm , i = 1 UImσi EN [ || ( N̄ ′ − ¯̃N ′Im ) i||2 ] . More precisely we define UIm : = EN [ || ( N − ÑIm ) i||2 ] . Consequently we score the invariance of the i-th dimension with respect to interventions on the m-th dimension as : INV [ m , i ] = 1− EIm∼Im [ eIm , i ] . ( 5 ) We approximate all the expectations with their Monte Carlo estimates . According to this formulation , a perfectly invariant response map would score 1 on the off-diagonal elements of the matrix . Notice that invariance is not concerned with the diagonal entries of the matrix . However , we can interpret a high invariance score on the i-th diagonal element as a sign of the unresponsiveness of the dimension as it implies interventions on i are not registered in the response . We henceforth refer to these unresponsive latent dimensions with the appellative ” collapsed ” , in reference to the well-known effect of posterior collapse ( rather common in βVAE models ) . Equivariance score . Similarly , we define the equivariance error and equivariance score as : rIm , i = 1 U EN [ || ( ˆ̄Z ′Im − ¯̃Z ′Im ) i||2 ] EQV [ m , i ] = 1− EIm∼Im [ rIm , i ] . In the case of equivariance perfect consistency corresponds to all the entries being 1 . Notice however that high equivariance is also obtained by the trivial solution ( i.e . a constant encoding map ) . This is due to the fact that the intervention values are sampled from the aggregate posterior . Self-consistency score . Finally , the self-consistency score evaluates the response inner consistency under no intervention . More specifically , we measure the amount of error introduced by the response on average over the prior . In formula : SCN [ i ] = 1− 1 σi EN [ || ( N̄ ′ −N ) i||2 ] Importantly , self-consistency is the only score relating the prior and the response space and is closely related to the regularization used by AVAEs ( Cemgil et al. , 2020 ) . 2.2 EXPLICIT CAUSAL LATENT BLOCK Inspired by the distinction between noise and structure in the SCM we insert a similar bias into the latent space of autoencoders by directly joining the representation layer with a structural mixing block as shown in Figure 4 . We henceforth will refer to the botteneck layer as noise terms or N , and to the units produced by the subsequent mixing layer as causes orZ . EachZi is obtained from its corresponding noise term Ni and its predecessors Z < i through a learned nonlinear function fi , much like in an SCM ( implemented as an MLP ) . To increase the sparsity of the structure we apply a learnable mask Mi ∈ R ( i−1 ) to the parents , such that Zi = fi ( Mi Z < i−1 , Ni ) . The masks are parametrised through the Gumbel trick ( Jang et al. , 2016 ) to induce weight sharpening , and thus act as a gating mechanisms of the information provided by the predecessors . We name this extension of the representation layer ” explicit causal latent block ” ( XBlock ) and we call XNet ( or prefix ” X ” ) any network using this module . Most disentanglement methods attempt to learn a representation consisting of causal variables or disentangled ” factors ” which are statistically independent ( Higgins et al. , 2017a ) . However , except in the trivial case , it is not the Zi that should be treated as statistically independent , but the Ni . Consequently , the explicit separation between noise and structure reconciles with the conventional disentanglement perspective while also providing the model with additional power to learn identifiable relationships between the latent variables terms . The causal links revealed by Mi can then be used to help identify how interventions will affect the resulting samples for downstream tasks such as controllable generation . Moreover , differentiating between noise and structure allows to expand the dimensionality of the causal variables Z , without any need to reparametrise the noise distribution . We partition the causes in units , each one consisting of multiple dimensions and representing a single statistical variable .

Summary: This work proposes the notion of "interventional consistency" as a beneficial property learned representations should have and introduces a regularization term to enforce it in autoencoders. Moreover, the authors introduce an "explicit latent causal block" that allows learning a structural causal model (SCM) over the latent factors. The experimental section argues that both the regularization and the "causal layer" improve interventional consistency.