Explicit Information Placement on Latent Variables using Auxiliary Generative Modelling Task

Deep latent variable models , such as variational autoencoders , have been successfully used to disentangle factors of variation in image datasets . The structure of the representations learned by such models is usually observed after training and iteratively refined by tuning the network architecture and loss function . Here we propose a method that can explicitly place information into a specific subset of the latent variables . We demonstrate the use of the method in a task of disentangling global structure from local features in images . One subset of the latent variables is encouraged to represent local features through an auxiliary modelling task . In this auxiliary task , the global structure of an image is destroyed by dividing it into pixel patches which are then randomly shuffled . The full set of latent variables is trained to model the original data , obliging the remainder of the latent representation to model the global structure . We demonstrate that this approach successfully disentangles the latent variables for global structure from local structure by observing the generative samples of SVHN and CIFAR10 . We also clustering the disentangled global structure of SVHN and found that the emerging clusters represent meaningful groups of global structures – including digit identities and the number of digits presence . Finally , we discuss the problem of evaluating the clustering accuracy when ground truth categories are not expressive enough . 1 INTRODUCTION . Prior knowledge can be imposed on a machine learning algorithm in many ways . One of the most important methods is through the construction of the neural network . A notable example is the use of convolutional neural networks ( CNN ) , where CNNs have shown to improve performance of deep neural networks in many image datasets due to its ability to impose priors about spatiality of the data . The choice of loss function can also be seen as an imposition of prior knowledge , e.g . the square distance error is more suited for a continuous dataset than a discrete dataset . In this work , we propose a method that can impose prior knowledge onto the representation of latent variable models . In contrast to previous methods , the imposition of prior knowledge is done through the training of a model rather than the construction of a network or its loss function . This method aims to improve the ability of representation learning algorithms with more explicit control over which information is represented at each latent variable . Many disentangled representation learning methods , such as Variational Auto-Encoder ( VAE ) , implicitly assign the meaning of each latent dimension after training . Since , the representation learned by VAEs is only enforced through the objective function encouraging factorised latent variables . Each latent dimension is only created through the optimisation dynamics which is unknown to the model designer . For example , in the case of the MNIST digit dataset , we have a prior knowledge that the dataset composes of digits with different orientations and thicknesses of strokes . However , it is not possible to explicitly assign the first dimension of the latent variable to represent the orientation while forcing the second dimension to represent the thickness . We can only observe the information placement after a training . If the information can be placed explicitly in the model then different types of prior knowledge can be imposed and , thus improving the model designing process . To this end , we show that explicit information placement can be made possible through the training setup which we call an auxiliary generative modelling task . We demonstrate the ability of this method through the task of explicitly placing global and local information onto different subsets of latent variables . By doing so , global and local information can be disentangled . We contend that disentangling this type of information is appropriate because it is important prior knowledge which is universally applicable to natural images . For example , a perceptual module of a robot would benefit from learning a separation between the texture and identity of an object . We also propose a suite of experiments that aim to demonstrate that global and local information can be explicitly placed on subsets of the latent variables with our method . 1.1 DISENTANGLING LOCAL AND GLOBAL STRUCTURE USING VAE . It is well known that a deep convolutional neural network trained on an image recognition task will learn a hierarchy of features , where the lower convolutional layers correspond to local structure and the deeper layers near the classifier neurons respond to higher level features ( Gatys et al. , 2015 ; Olah et al. , 2018 ) . This disentanglement emerges from the network architecture and the training objective . As the information is pushed through multiple layers of convolutional neurons , it is lossy compressed down to only a class identity in the final output layer . In a latent variable model such as an autoencoder , however , information is processed differently . In contrast to a recognition model , a latent variable model can not throw away most of the information in its input layer . Rather , it must reorganise this information into a compact representation which can be used to reconstruct the original data . One way to control the structure of the representation , for example to promote disentanglement , is to adjust the objective functions which control the level of compression and redundancy in the latent variables ( Higgins et al. , 2017 ; Chen et al. , 2018 ; Kim & Mnih , 2018 ; Esmaeili et al. , 2018 ; Dilokthanakul et al. , 2016 ) . Latent variable models ( LVMs ) assume that the data , x , is generated from a latent variable , z , and a generative process , p ( x|z ) . One way to fit the LVM to the data is to restrict the generative process to a set of parameterised distributions pθ ( x|z ) and search for parameters θ that maximise the loglikelihood of the data , Ep ( x ) [ log p ( x|θ ) ] . A Variational Autoencoder ( VAE ) ( Kingma & Welling , 2014 ; Rezende et al. , 2014 ) is an LVM that parameterises its generative process pθ ( x|z ) and its variational posterior qφ ( z|x ) with deep neural networks , θ , φ . The parameters of the neural networks are trained with stochastic gradient descent to maximise the variational free-energy objective . The variational free-energy objective is a lower bound to the log-likelihood of the data which can be written as L = Ep ( x ) [ Eqφ ( z|x ) [ log pθ ( x|z ) ] −KL ( qφ ( z|x ) ||p ( z ) ) ] . ( 1 ) The VAE is one of the most successful latent variable models , and it has been shown to be effective at modelling image datasets . It has also been shown to be capable of creating latent structure that is compositional and interpretable ( Higgins et al. , 2017 ; Kim & Mnih , 2018 ) . These desirable properties are thought to be the result of the free-energy objective , Eq.1 , which also has the effect of encouraging the latent variables to be factorised . Eq . 1 consists of two terms known as the reconstruction cost and the KL cost . Following Hoffman and Johnson ( Hoffman & Johnson , 2016 ) and Chen et al . ( Chen et al. , 2018 ) , the KL term can be rewritten as Ep ( x ) [ KL ( qφ ( z|x ) ||p ( z ) ) ] = KL ( q ( z , x ) ||q ( z ) p ( x ) ) +KL ( q ( z ) || ∏ j q ( zj ) ) + ∑ j KL ( q ( zj ) ||p ( zj ) ) . ( 2 ) We can see from the second term in Eq.2 that the KL cost tries to reduce the total correlation of the latent variables , thus promoting factorised representation . However , in a complex dataset where images have a variety of textures as well as different global contexts , VAEs struggle to disentangle local structure from global structure and often focus their representations on the local structure which is easier to model . For example , Dilokthanakul et al . ( 2016 ) had shown in an SVHN experiment that multi-modal versions of VAEs tend to cluster local structures into the same mode while ignoring the global context , i.e . disentanglement based on the background colours of the image . Importantly , it is not clear whether the factorised representation objective alone would be enough to disentangle this kind of multi-level structure . Imposing hierarchical structure onto the VAE latent variables , where lower-level latent variables are generated from higher-level latent variables , might be able to disentangle multi-level structure . However , Zhao et al . ( 2017 ) argued that hierarchical VAE failed to use the higher latent variables and avoided this problem by only using one level of latent structure ( flat latent structure ) and carefully creating a neural network architecture that has different levels of computation for each latent dimension . This method successfully disentangles local structure from global structure , but at the cost of needing to carefully tune the architecture , which is the process that is poorly understood . Similarly , Chen et al . ( 2016 ) and Gulrajani et al . ( 2016 ) showed that information placement can be controlled through the capacity of the decoder . They used the pixelCNN decoder with varying sizes of receptive fields to control how much information the latent variables need to model . However , their methods can not represent local structures with latent variables , as the local structures will be captured via the pixelCNN , which is an autoregressive model . 2 METHODS . 2.1 EXPLICIT PLACEMENT OF INFORMATION THROUGH AUXILIARY GENERATIVE MODELLING . In this section , we outline our method , which allows the explicit placement of information in each latent variable . Let ’ s denote the original data as a stochastic variable x and a modified version of the data x̂ . x̂ is created from a transformation of x in such a way that the information of interest is destroyed . This transformation has to be crafted for different types of information , e.g . global information can be destroyed by shuffling patches of the images , or colour information is destroyed by conversion to grey scale . Next , we create two sets of latent variables , zi and zr . We aim to model the information of interest with zi and the remainder with zr . We assume a generative processes as follows . zi , zr ∼ p ( zi , zr ) ( 3 ) x ∼ pθ ( x|zi , zr ) ( 4 ) x̂ ∼ pθ̂ ( x̂|zr ) . ( 5 ) The process of generating x is the usual generative assumption in VAE . In addition to x , x̂ is assumed to be generated from zr which is shared with x . By also learning the generative model pθ̂ ( x̂ ) , we can control the representation of zr by controlling the information contained in x̂ . In this case , x̂ does not hold details about the information of interest and , therefore , forces zr to model the leftover information after transformation of x . Due to the compression objective of VAE , the remaining variable zi is pressured to represent the information of interest in order to reduce redundancy in the latent variables . This is what we refer to as an auxiliary modelling task , which explicitly places information of interest in a subset of the latent variables .

Summary: The paper proposes a method to disentangle latent variables for certain factors of interest in an image by considering the original input image and a transformation of the image where information about the factors of interest is removed. The generative process is then modeled by having two latent variables --  the first responsible for generating the transformed image whereas both latent variables are responsible for generating the original input image. This inductive bias naturally enforces that the second latent variable will not model the information which the first needs to reconstruct the transformed image, due to the VAE objective penalizing redundancy in information present in the latents. The paper demonstrates this in one setting where the transformation is random shuffling of image patches, which should remove the global information of the original input image.