Disentangling Action Sequences: Discovering Correlated Samples

1 INTRODUCTION . The basis of artificial intelligence is to understand and reason about the world based on a limited set of observations . Unsupervised disentanglement learning is highly desirable due to its similarity with the way we as human think . For instance , we can infer the movement of a running ball based on a single glance . This is because the human brain is capable of disentangling the positions from a set of images . It has been suggested that a disentangled representation is helpful for a large variety of downstream tasks ( Schölkopf et al. , 2012 ; Peters et al. , 2017 ) . According to Kim & Mnih ( 2018 ) , a disentangled representation promotes interpretable semantic information . That brings substantial advancement , including but not limited to reducing the performance gap between humans and AI approaches ( Lake et al. , 2017 ; Higgins et al. , 2018 ) . Other instances of disentangled representation include semantic image understanding and generation ( Lample et al. , 2017 ; Zhu et al. , 2018 ; Elgammal et al. , 2017 ) , zero-shot learning ( Zhu et al. , 2019 ) , and reinforcement learning ( Higgins et al. , 2017b ) . Despite the advantageous of the disentangling representation approaches , there are still two issues to be addressed including the abstract notion and the weak explanations . Notion The conception of disentangling factors of variation is first proposed in 2013 . It is claimed in Bengio et al . ( 2013 ) that for observations the considered factors should be explanatory and independent of each other . The explanatory factors are however hard to formalize and measure . An alternative way is to disentangle the ground-truth factors ( Ridgeway , 2016 ; Do & Tran , 2020 ) . However , if we consider the uniqueness of the ground-truth factors , a question which arises here is how to discover it from multiple equivalent representations ? As a proverb “ one can not make bricks without straw ” , Locatello et al . ( 2019 ) prove the impossibility of disentangling factors without the help of inductive biases in the unsupervised setting . Explanation There are mainly two types of explanations for unsupervised disentanglement : information bottleneck , and independence assumption . The ground-truth factors affect the data independently , therefore , the disentangled representations must follow the same structure . The approaches , holding the independence assumption , encourage independence between the latent variables ( Schmidhuber , 1992 ; Chen et al. , 2018 ; Kim & Mnih , 2018 ; Kumar et al. , 2018 ; Lopez et al. , 2018 ) . However , the real-world problems have no strict constraint on the independence assumption , and the factors may be correlative . The other explanation incorporates information theory into disentanglement . Burgess et al . ; Higgins et al . ; Insu Jeon et al . ; Saxe et al . suggest that a limit on the capacity of the latent information channel promotes disentanglement by enforcing the model to acquire the most significant latent representation . They further hypothesize that the information bottleneck enforces the model to find the significant improvement . In this paper , we first demonstrate that instead of the ground-truth factors the disentangling approaches learn actions of translating based on the orientation of the images . We then propose the concept of disentangling actions which discover the commonalities between the images and categorizes them into sequences . We treat disentangling action sequences as a necessary step toward disentangling factors , which can capture the internal relationships between the data , and make it possible to analyze the inductive biases from the data perspective . Furthermore , the results on a toy example show that the significance of actions is positively correlated with the threshold of latent information . Then , we promote that conclusion to complex problems . Our contributions are summarized in the following : • We show that the significance of action is related to the capacity of learned latent information , resulting in the different thresholds of factors . • We propose a novel framework , fractional variational autoencoder ( FVAE ) to extracts explanatory action sequences step-by-step , and at each step , it learns specific actions by blocking others ’ information . We organize the rest of this paper as follows . Sec.2 describes the development of unsupervised disentanglement learning and the proposed methods based on VAEs . In Sec.3 , through an example , we show that the disentangled representations are relative to the data itself and further introduce a novel concept , i.e. , disentangling action sequences . Then , we investigate the inductive biases on the data and find that the significant action has a high threshold of latent information . In Sec.4 , we propose a step-bystep disentangling framework , namely fractional VAE ( FVAE ) , to disentangle action sequences . For the labelled and unlabelled tasks , we respectively introduce two methods to measure their thresholds . We then evaluate FVAE on a labelled dataset ( dSprites , Matthey et al . ( 2017 ) ) and an unlabelled dataset ( 3D Chairs , Aubry et al . ( 2014 ) ) . Finally , we conclude the paper and discuss the future work in Sec.5 2 UNSUPERVISED DISENTANGLEMENT LEARNING . We first introduce the abstract concepts and the basic definitions , followed by the explanations based on information theory and other related works . This article focuses on the explanation of information theory and the proposed models based on VAEs . 2.1 THE CONCEPT . Disentanglement learning is fascinating and challenging because of its intrinsic similarity to human intelligence . As depicted in the seminal paper by Bengio et al. , humans can understand and reason from a complex observation to the explanatory factors . A common modeling assumption of disentanglement learning is that the observed data is generated by a set of ground-truth factors . Usually , the data has a high number of dimensions ; hence it is hard to understand , whereas the factors have a low number of dimensions , thus simpler and easier to be understood . The task of disentanglement learning is to uncover the ground-truth factors . Such factors are invisible to the training process in an unsupervised setting . The invisibility of factors makes it hard to define and measure disentanglement ( Do & Tran , 2020 ) . Furthermore , it is shown in Locatello et al . ( 2019 ) that it is impossible to unsupervised disentangle the underlying factors for the arbitrary generative models without inductive biases . In particular , they suggest that the inductive biases on the models and the data should be exploited . However , they do not provide a formal definition of the inductive bias and such a definition is still unavailable . 2.2 INFORMATION BOTTLENECK . Most of the dominant disentangling approaches are the variants of variational autoencoder ( VAE ) . The variational autoencoder ( VAE ) is a popular generative model , assuming that the latent variables obey a specific prior ( normal distribution in practice ) . The key idea of VAE is maximizing the likelihood objective by the following approximation : L ( θ , φ ; x , z ) =Eqφ ( z|x ) [ logpθ ( x|z ) ] −DKL ( qφ ( z|x ) ||p ( z ) ) , ( 1 ) which is known as the evidence lower bound ( ELBO ) ; where the conditional probabilityP ( x|z ) , Q ( z|x ) are parameterized with deep neural networks . Higgins et al . find that the KL term of VAEs encourages disentanglement and introduce a hyperparameter β in front of the KL term . They propose the β-VAE maximizing the following expression : L ( θ , φ ; x , z ) =Eqφ ( z|x ) [ logpθ ( x|z ) ] −βDKL ( qφ ( z|x ) ||p ( z ) ) . ( 2 ) β controls the pressure for the posterior Qφ ( z|x ) to match the factorized unit Gaussian prior p ( z ) . Higher values of β lead to lower implicit capacity of the latent information and ambiguous reconstructions . Burgess et al . propose the Annealed-VAE that progressively increases the information capacity of the latent code while training : L ( θ , φ ; x , z , C ) =Eqφ ( z|x ) [ logpθ ( x|z ) ] −γ|DKL ( qφ ( z|x ) ||p ( z ) ) −C| ( 3 ) where γ is a large enough constant to constrain the latent information , C is a value gradually increased from zero to a large number to produce high reconstruction quality . As the total information bottleneck gradually increasing , they hypothesize that the model will allocate the capacity of the most improvement of the reconstruction log-likelihood to the encoding axes of the corresponding factor . However , they did not exploit why each factor makes different contributions to the reconstruction log-likelihood . 2.3 OTHER RELATED WORK . The other dominant direction initiates from the prior of factors . They assume that the ground-truth factors are independent of each other , and a series of methods enforce the latent variables have the same structure as the factors . FactorVAE ( Kim & Mnih , 2018 ) applies a discriminator to approximately calculate the total correlation ( TC , Watanabe ( 1960 ) ) ; β-TCVAE ( Chen et al. , 2018 ) promotes the TC penelty by decomposing the KL term ; DIP-VAE ( Kumar et al. , 2018 ) identifies the covariance matrix of q ( z ) . 3 DISENTANGLING ACTION SEQUENCES . For machines and humans , disentangling the underlying factors is a challenging task . For instance , there are more than 1,400 breeds of dogs in the world , and it seems impossible for an ordinary person to distinguish all of them just by looking at their pictures . The challenge in disentangling the underlying factors is mainly due to the complexity of establishing relationships without supervision , where the corresponding models should contain some level of prior knowledge or inductive biases . However , it is possible to determine the differences without having extensive knowledge . For example , one may mistakenly identify the breed of a dog by looking at a picture but is almost impossible to misrecognize a dog as a cat . Therefore , in practice , discovering the differences or similarities is often a much easier task than that of uncovering the underlying factors , and it does not also need much prior knowledge . One may conclude that discovering the commonalities between things is an important step toward disentanglement . 3.1 ACTION SEQUENCES . Action The continuous set of images over a certain direction . The observed data consists of a set of actions such as rotation , translation , and scaling . Such actions are meaningful and underlying . Therefore , the goal of disentanglement learning is to separate the underlying actions of observation . Usually , the continuous actions are infeasible for the machine learning system , and we have to convert them into some discrete and consistent sequences . Given a dataset S , we assume it consists of parameterized action sequences , and each action sequence is a subset of the dataset . We can model a parameterized action sequence as : Si ( mj , j 6=i ) =Tmj , j 6=i ( { t } ) , ( 4 ) where Si∈S denotes an action sequence , i denotes the type of actions , m denotes a parameter vector , T denotes a transformation , t denotes the step of the action . The procedure is similar to the latent traversal in Higgins et al . ( 2017a ) . Differently , action sequences describe both the real observed data and the reconstructed data : For the real observed data , m is the ground-truth factor vector , and T is the generating function ; For the reconstructed data , m is the latent variable vector , andT is a neural network . For clarity , we denote generating actions as sequences of images controlled by the ground-truth factors and action sequences as the approximation of these actions generated by a neural network . However , the neural networks are flexible enough to approximate any actions , and it ’ s tricky to infer and learn the generating actions . The marvelous thing about disentanglement learning is that the models seem to learn these actions unsupervisedly . A popular view of disentanglement believes the ground-truth factors should be separated into independent latent variables Locatello et al . ( 2019 ) ; Chen et al . ( 2018 ) . That means the learned action sequences have to match the generating actions precisely . However , ( Rolinek et al. , 2019 ) shows that the VAEs conduct a PCA-like behavior , and the models prefer to extract the principal component . Hence , the models may select some action sequences having more considerable variations and learn the significant action sequences rather than the generating actions . Using VAEs , minimizing the objective increases the overlap between the posterior distributions across the dataset Burgess et al . ( 2018 ) , and it then leads to the learned action sequences showing some internal relationships . Although ( Locatello et al. , 2019 ) suggest that the inductive biases on the models and the data should be exploited , the current researches focus on the role of the models on disentanglement Burgess et al . ( 2018 ) ; Chen et al . ( 2018 ) ; Rolinek et al . ( 2019 ) .

Summary: In this submission, a common modelling assumption for unsupervised disentanglement is challenged: that the disentangled representation follows the independence structure of the underlying (data generating) factors. Instead, the paper proposes to consider *action sequences* which describe how datapoints are interrelated. The paper provides evidence that the capacity of the latent representation (controlled by Lagrange parameter beta in beta-VAE related models) is related to the significance of particular action sequence for disentanglement. To leverage this insight, the fractional VAE (FVAE) is proposed, consisting of several sub-encoders and different training stages. The disentangling properties of the FVAE is demonstrated on the dSprites and 3D chairs datasets, with the FVAE performing favourably to the beta-VAE w.r.t. the Mutual Information Gap (MIG) disentanglement metric on dSprites.