On the interventional consistency of autoencoders

1 INTRODUCTION . Representation learning is the problem of finding a low dimensional description of the data . The characteristics of a ’ good representation ’ have long since been a matter of debate , often depending on the context in which the representation has to be employed . Multiple works ( cf . Bengio et al . ( 2013 ) , Schölkopf et al . ( 2021 ) ) identify modularity , robustness to distribution shifts and interpretability as crucial features of a ’ good representation ’ , motivating the quest toward disentanglement ( Bachman et al . ( 2019 ) , Locatello et al . ( 2020 ) , Ridgeway & Mozer ( 2018 ) ) and causal representations ( Suter et al . ( 2019 ) , Träuble et al . ( 2021 ) ) . The insight shared by both these fields is that complex world phenomena arise from the ’ rich interaction of many sources ’ , and thus enable compact descriptions in terms of the basic components participating in their generative process . More formally , the hypothesis is that there exist a set of semantically meaningful variables S and an arbitrarily non-linear function G such that X = G ( S1 , ... , Sn ) , where X are the observations . Autoencoders have played a crucial role in the field of representation learning since its inception . Their success is largely due to their simplicity and surprising effectiveness . Mathematically autoencoders can be represented as the tuple ( E : Rd → Rn , D : Rn → Rd ) . Capacity constraints ( e.g . n d ) force the latent space to prioritize certain information in the input , thus yielding a useful representation . Additional requirements may be imposed either structurally or through regularisation . Intuitively , consistency means that an input generated by the decoder can be mapped back to the point in the latent space that produced it . The central idea of Cemgil et al . ( 2020 ) is making the encoder and the decoder consistent both on the training data P ( X ) and on the auxiliary observations generated by the decoder P ( X ′ ) . Our contribution . Unsupervised causal representation learning is largely unsolved today . In a recent paper Locatello et al . ( 2019 ) prove that ’ the unsupervised learning of disentangled representations is fundamentally impossible without inductive biases on both the models and the data ’ . Motivated by this result , in this work we , investigate the structure and in particular the self-consistency of the learned autoencoder , rather than focusing on the representation ’ s relation to the ” groundtruth ” factors . In this context , we formulate an inductive bias for autoencoders , that we call interventional consistency , and we link it to the solution space of the disentanglement problem . Additionally , we introduce a new architectural module for the latent space of autoencoders which unifies the disentanglement objective and the , more structured , causal perspective . 2 METHOD Consider the ’ artificial ’ generative process implemented by the decoder as X ∼ P ( Z ) P ( X|Z ; θ ) . We express the latent space dynamics in terms of a Structural Causal Model ( SCM ) ( Pearl , 2009 ) on Z . An SCM is defined by a set of so-called noise variables N = N1 , ... , Nn , with a distribution P ( N ) that factorises , and a set of structural assignments f1 , ... , fn of the form : Zi : = fi ( PAi , Ni ) for i = 1 , ... , n , ( 1 ) where PAi refers to the set of direct causes of the variable Zi . The set of directed interactions between causal variables identifies a graph , which is usually assumed to be acyclic ( i.e . a DAG ) . This formulation naturally entails a distribution P ( Z ) and a corresponding causal factorisation : P ( Z ) = ∏ i P ( Zi|PAi ) . ( 2 ) For a thorough introduction to SCM we refer the reader to Peters et al . ( 2017 ) . We can now apply results from causality to the artificial generative process . In particular , we turn to the ICM principle ( Peters et al. , 2017 ) , which states that the modules participating in the generative process are independent and can thus be separately manipulated . Let us refer to the composition R = E ◦ D as the response map , as done in Leeb et al . ( 2021 ) , mapping the latent space Z ⊆ Rn back to itself , i.e . R : Z → Z . Let R ( Z ′|Z ) be the posterior distribution entailed by the map , with Z , Z ′ being distinct random variables . Furthermore , consider the response aggregate posterior distribution and any factorisation thereof : P ( Z ′ ) = ∫ P ( Z ) R ( Z ′|Z ) dZ = ∏ i P ( Z ′i|Z ′j 6=i ) . The implications of the ICM principle are trivial if applied to the prior P ( Z ) : modifying P ( Zi|PAi ) will not affect P ( Zj |PAj ) for j 6= i . However the consequences get far from trivial if we require the response map to preserve the validity of the principle . Let P̃ ( Z ) denote the intervention distribution : P̃ ( Z ) = P̃ ( Zm|P̃Am ) ∏ i 6=m P ( Zi|PAi ) . ( 3 ) Then we say that the ICM principle is preserved by the response map if : P̃ ( Z ′ ) = ∫ P̃ ( Z ) R ( Z ′|Z ) dZ = P̃ ( Z ′k|Zj 6=k ) ∏ i 6=k P ( Z ′i|Z ′j 6=i ) , ( 4 ) where the indicesm and k are not necessarily equivalent , although we will from here on assume their equivalence.1 Intuitively , R preserves the ICM principle if any localised change in P ( Z ) produces a corresponding localised change in P ( Z ′ ) . The idea is illustrated in Figure 1 . Crucially , this property mirrors the definition of disentanglement in the context of the artificial generative process : each factor in the representation ( P ( Z ′k|Z ′j 6=k ) ) is sensitive to changes in a single generative factor ( P ( Zm|PAm ) ) , while being relatively invariant to changes elsewhere in the generative process . This 1Allowing m 6= k essentially takes into account the possibility of permutation of the original space dimensions in the representation , which is the norm in disentanglement metrics . By assuming m = k we get rid of one degree of freedom in the problem . analogy suggests that the condition expressed in Equation 4 has to be satisfied by a representation that disentangles the true generative process . We call this condition interventional consistency . Interventional consistency can be formulated in terms of invariance and equivariance of the response map . If the autoencoder satisfies interventional consistency , then P ( Z ′i|Z ′j < i ) is invariant to the action of the group of interventions Im localised on P ( Zm|Zj < m ) with i 6= m. Moreover , assuming we are able to perform an intervention on the response variables I ′ equivalent to the one performed on the prior I , the response map is equivariant to the intervention , meaning that the order in which the response and the intervention are applied is not important . From this symmetry viewpoint , interventional consistency belongs to causal representations : ’ the right causal order is the one invariant to the right kind of interventions ’ ( Herb Simon , cf . Hoover ( 2008 ) ) . Importantly , a representation that satisfies the interventional consistency condition does not necessarily solve the identifiability problem ( Gresele et al. , 2021 ) , as any bijective transformation of the true causal factors would equally satisfy the condition for the correspondingly transformed interventions . Instead , it is manipulable by construction . Without this feature , there is no guarantee the decoder can extrapolate to latent samples not seen during training . As an example , consider the common diagnostic tool of latent traversals : it consists of atomic interventions applied to the different dimensions of the representation . Often a good representation is associated with semantically meaningful traversals in the image space . However , the observed result is only half of the story : if the encoder fails to interpret the generated samples the representation can not be called disentangled . To illustrate this example , in Figure 2 we show the error recorded on the response to traversals on a standard autoencoder . 2.1 CONSISTENCY TRAINING . It is possible to use the invariance or equivariance formulation to incorporate interventional consistency into the solution space of an autoencoding problem . Let LINV ( R ) and LEQV ( R ) be real valued functions measuring the violation of the invariance and equivariance property by the response map R , respectively . Then the space of interventionally consistent autoencoders is defined as : { ( E , D ) : LINV ( E ◦D ) = 0 or LEQV ( E ◦D ) = 0 } . This hard constraint is replaced with a differentiable regularization by augmenting the optimization objective with LINV or LEQV . Next , we propose evaluation metrics that make the consistency penalties easier to interpret . Algorithm 1 and 2 in the Appendix demonstrate how to compute LINV and LEQV . Importantly , LINV is independent of the network architecture of choice , therefore it can be applied to any existing autoencoder method with latent space N . Meanwhile , LEQV only requires an explicit distinction between noises N and causes Z ( of the artificial generative process ) outlined in the previous section , which , in our experiments , is implemented by the explicit causal latent block ( see 4 ) . Note some additional mild approximations are necessary to make the penalty terms tractable . Firstly , we assume that the response aggregate posterior P ( Z ′ ) factorises like P ( Z ) , i.e . that the statistical dependencies in the prior are preserved by the response map . When using an autoencoder with explicit structural mappings this assumption is equivalent to assuming that the response aggregate noise posterior P ( N ′ ) factorises . Secondly , we employ Monte Carlo sampling to estimate the consistency by sampling hard interventions on the noise variable distribution of the form : P̃ ( Nm ) ← δ ( v ) , with v from the aggregate marginal posterior Q ( Nm ) = EXQ ( Nm|X ) . Finally , in order to assess equivariance we treat I ′ = I , which relies on the equivalence between Z and Z ′ . This assumption is satisfied as long as the reconstructions are sufficiently similar to corresponding the training samples ( i.e . the autoencoder fidelity is high ) . Below , Nk denotes a sample from the prior P ( N ) and Ñk its intervened version . Moreover we let N̄ ′k and ¯̃Nk′ be the expectation of the response to Nk and Ñk , respectively . Let ˆ̄Nk′ be the result of the intervention I applied to N̄ ′k and we denote the corresponding causal variables by ˆ̄Z ′k and ¯̃Zk . Invariance score . Let UIm be a measure of the magnitude of the intervention Im and σi the standard deviation of each dimension in the response space . Then we define the invariance error in response to the intervention Im recorded on the dimension i as eIm , i = 1 UImσi EN [ || ( N̄ ′ − ¯̃N ′Im ) i||2 ] . More precisely we define UIm : = EN [ || ( N − ÑIm ) i||2 ] . Consequently we score the invariance of the i-th dimension with respect to interventions on the m-th dimension as : INV [ m , i ] = 1− EIm∼Im [ eIm , i ] . ( 5 ) We approximate all the expectations with their Monte Carlo estimates . According to this formulation , a perfectly invariant response map would score 1 on the off-diagonal elements of the matrix . Notice that invariance is not concerned with the diagonal entries of the matrix . However , we can interpret a high invariance score on the i-th diagonal element as a sign of the unresponsiveness of the dimension as it implies interventions on i are not registered in the response . We henceforth refer to these unresponsive latent dimensions with the appellative ” collapsed ” , in reference to the well-known effect of posterior collapse ( rather common in βVAE models ) . Equivariance score . Similarly , we define the equivariance error and equivariance score as : rIm , i = 1 U EN [ || ( ˆ̄Z ′Im − ¯̃Z ′Im ) i||2 ] EQV [ m , i ] = 1− EIm∼Im [ rIm , i ] . In the case of equivariance perfect consistency corresponds to all the entries being 1 . Notice however that high equivariance is also obtained by the trivial solution ( i.e . a constant encoding map ) . This is due to the fact that the intervention values are sampled from the aggregate posterior . Self-consistency score . Finally , the self-consistency score evaluates the response inner consistency under no intervention . More specifically , we measure the amount of error introduced by the response on average over the prior . In formula : SCN [ i ] = 1− 1 σi EN [ || ( N̄ ′ −N ) i||2 ] Importantly , self-consistency is the only score relating the prior and the response space and is closely related to the regularization used by AVAEs ( Cemgil et al. , 2020 ) . 2.2 EXPLICIT CAUSAL LATENT BLOCK Inspired by the distinction between noise and structure in the SCM we insert a similar bias into the latent space of autoencoders by directly joining the representation layer with a structural mixing block as shown in Figure 4 . We henceforth will refer to the botteneck layer as noise terms or N , and to the units produced by the subsequent mixing layer as causes orZ . EachZi is obtained from its corresponding noise term Ni and its predecessors Z < i through a learned nonlinear function fi , much like in an SCM ( implemented as an MLP ) . To increase the sparsity of the structure we apply a learnable mask Mi ∈ R ( i−1 ) to the parents , such that Zi = fi ( Mi Z < i−1 , Ni ) . The masks are parametrised through the Gumbel trick ( Jang et al. , 2016 ) to induce weight sharpening , and thus act as a gating mechanisms of the information provided by the predecessors . We name this extension of the representation layer ” explicit causal latent block ” ( XBlock ) and we call XNet ( or prefix ” X ” ) any network using this module . Most disentanglement methods attempt to learn a representation consisting of causal variables or disentangled ” factors ” which are statistically independent ( Higgins et al. , 2017a ) . However , except in the trivial case , it is not the Zi that should be treated as statistically independent , but the Ni . Consequently , the explicit separation between noise and structure reconciles with the conventional disentanglement perspective while also providing the model with additional power to learn identifiable relationships between the latent variables terms . The causal links revealed by Mi can then be used to help identify how interventions will affect the resulting samples for downstream tasks such as controllable generation . Moreover , differentiating between noise and structure allows to expand the dimensionality of the causal variables Z , without any need to reparametrise the noise distribution . We partition the causes in units , each one consisting of multiple dimensions and representing a single statistical variable .

Summary: The paper proposes to use interventional consistency to regularise representation learning in VAEs. The idea is well-motivated by the ICM principle and theoretically justified. The paper suggests to use interventional consistency for both training and evaluation of the representation learnt by VAEs. Results show that the proposed idea can give more modular and interpretable representation.