Towards Nonlinear Disentanglement in Natural Data with Temporal Sparse Coding

1 INTRODUCTION . Natural scene understanding can be achieved by decomposing the signal into its underlying factors of variation . An intuitive approach for this problem assumes that a visual representation of the world can be constructed via a generative process that receives factors as input and produces natural signals as output ( Bengio et al. , 2013 ) . This analogy is justified by the fact that our world is composed of distinct entities that can vary independently , but with regularity imposed by physics . What makes the approach appealing is that it formalizes representation learning by directly comparing representations to underlying ground-truth states , as opposed to the indirect evaluation of benchmarking against heuristic downstream tasks ( e.g . object recognition ) . However , the core issue with this approach is non-identifiability , which means a set of possible solutions may all appear equally valid to the model , while only one identifies the true generative factors . Our work is motivated by the question of whether the statistics of natural data will allow for the formulation of an identifiable model . Our core observation that enables us to make progress in ∗‡Equal contribution . Code : https : //github.com/bethgelab/slow_disentanglement addressing this question is that generative factors of natural data have sparse transitions . To estimate these generative factors , we compute statistics on measured transitions of area and position for object masks from large-scale , natural , unstructured videos . Specifically , we extracted over 300,000 object segmentation mask transitions from YouTube-VOS ( Xu et al. , 2018 ; Yang et al. , 2019 ) and KITTI-MOTS ( Voigtlaender et al. , 2019 ; Geiger et al. , 2012 ; Milan et al. , 2016 ) ( discussed in detail in Appendix D ) . We fit generalized Laplace distributions to the collected data ( Eq . 2 ) , which we indicate with orange lines in Fig . 1 . We see empirically that all marginal distributions of temporal transitions are highly sparse and that there exist complex dependencies between natural factors ( e.g . motion typically affects both position and apparent size ) . In this study , we focus on the sparse marginals , which we believe constitutes an important advance that sets the stage for solving further issues and eventually applying the technology to real-world problems . With this information at hand , we are able to provide a stronger proof for capturing the underlying generative factors of the data up to permutations and sign flips that is not covered by previous work ( Hyvärinen and Morioka , 2016 ; 2017 ; Khemakhem et al. , 2020a ) . Thus , we present the first work , to the best of our knowledge , which proposes a theoretically grounded solution that covers the statistics observed in real videos . Our contributions are : With measurements from unstructured natural video annotations we provide evidence that natural generative factors undergo sparse changes across time . We provide a proof of identifiability that relies on the observed sparse innovations to identify nonlinearly mixed sources up to a permutation and sign-flips , which we then validate with practical estimation methods for empirical comparisons . We leverage the natural scene information to create novel datasets where the latent transitions between frames follow natural statistics . These datasets provide a benchmark to evaluate how well models can uncover the true latent generative factors in the presence of realistic dynamics . We demonstrate improved disentanglement over previous models on existing datasets and our contributed ones with quantitative metrics from both the disentanglement ( Locatello et al. , 2018 ) and the nonlinear ICA community ( Hyvärinen and Morioka , 2016 ) . We show via numerous visualization techniques that the learned representations for competing models have important differences , even when quantitative metrics suggest that they are performing equally well . 2 RELATED WORK – DISENTANGLEMENT AND NONLINEAR ICA . Disentangled representation learning has its roots in blind source separation ( Cardoso , 1989 ; Jutten and Herault , 1991 ) and shares goals with fields such as inverse graphics ( Kulkarni et al. , 2015 ; Yildirim et al. , 2020 ; Barron and Malik , 2012 ) and developing models of invariant neural computation ( Hyvärinen and Hoyer , 2000 ; Wiskott and Sejnowski , 2002 ; Sohl-Dickstein et al. , 2010 ) ( see Bengio et al. , 2013 , for a review ) . A disentangled representation would be valuable for a wide variety of machine learning applications , including sample efficiency for downstream tasks ( Locatello et al. , 2018 ; Gao et al. , 2019 ) , fairness ( Locatello et al. , 2019 ; Creager et al. , 2019 ) and interpretability ( Bengio et al. , 2013 ; Higgins et al. , 2017 ; Adel et al. , 2018 ) . Since there is no agreed upon definition of disentanglement in the literature , we adopt two common measurable criteria : i ) each encoding element represents a single generative factor and ii ) the values of generative factors are trivially decodable from the encoding ( Ridgeway and Mozer , 2018 ; Eastwood and Williams , 2018 ) . Uncovering the underlying factors of variation has been a long-standing goal in independent component analysis ( ICA ) ( Comon , 1994 ; Bell and Sejnowski , 1995 ) , which provides an identifiable solution for disentangling data mixed via an invertible linear generator receiving at most one Gaussian factor as input . Recent unsupervised approaches for nonlinear generators have largely been based on Variational Autoencoders ( VAEs ) ( Kingma and Welling , 2013 ) and have assumed that the data is independent and identically distributed ( i.i.d . ) ( Locatello et al. , 2018 ) , even though nonlinear methods that make this i.i.d . assumption have been proven to be non-identifiable ( Hyvärinen and Pajunen , 1999 ; Locatello et al. , 2018 ) . Nonetheless , the bottom-up approach of starting with a nonlinear generator that produces well-controlled data has led to considerable achievements in understanding nonlinear disentanglement in VAEs ( Higgins et al. , 2017 ; Burgess et al. , 2018 ; Rolinek et al. , 2019 ; Chen et al. , 2018 ) , consolidating ideas from neural computation and machine learning ( Khemakhem et al. , 2020a ) , and seeking a principled definition of disentanglement ( Ridgeway , 2016 ; Higgins et al. , 2018 ; Eastwood and Williams , 2018 ) . Recently , Hyvärinen and colleagues ( Hyvärinen and Morioka , 2016 ; 2017 ; Hyvärinen et al. , 2018 ) showed that a solution to identifiable nonlinear ICA can be found by assuming that generative factors are conditioned on an additional observed variable , such as past states or the time index itself . This contribution was generalized by Khemakhem et al . ( 2020a ) past the nonlinear ICA domain to any consistent parameter estimation method for deep latent-variable models , including the VAE framework . However , the theoretical assumptions underlying this branch of work do not account for the sparse transitions we observe in the statistics of natural scenes , which we discuss in further detail in appendix F.1.1 . Another branch of work requires some form of supervision to demonstrate disentanglement ( Szabó et al. , 2017 ; Shu et al. , 2019 ; Locatello et al. , 2020 ) . We select two of the above approaches , that are both different in their formulation and state-of-the-art in their respective empirical settings , Hyvärinen and Morioka ( 2017 ) and Locatello et al . ( 2020 ) , for our experiments below . The motivation of our method and dataset contributions is to address the limitations of previous approaches and to enable unsupervised disentanglement learning in more naturalistic scenarios.1 The fact that physical processes bind generative factors in temporally adjacent natural video segments has been thoroughly explored for learning in neural networks ( Hinton , 1990 ; Földiák , 1991 ; Mitchison , 1991 ; Wiskott and Sejnowski , 2002 ; Denton and Birodkar , 2017 ) . We propose a method that uses time information in the form of an L1-sparse temporal prior , which is motivated by the natural scene measurements presented above as well as by previous work ( Simoncelli and Olshausen , 2001 ; Olshausen , 2003 ; Hyvärinen et al. , 2003 ; Cadieu and Olshausen , 2012 ) . Such a prior would intuitively allow for sharp changes in some latent factors , while most other factors remain unchanged between adjacent time-points . Almost all similar methods are variants of slow feature analysis ( SFA , Wiskott and Sejnowski , 2002 ) , which measure slowness in terms of the Euclidean ( i.e . L2 , or log Gaussian ) distance between temporally adjacent encodings . Related to our approach , a probabilistic interpretation of SFA has been previously proposed ( Turner and Sahani , 2007 ) , as well as extensions to variational inference ( Grathwohl and Wilson , 2016 ) . Additionally , Hashimoto ( 2003 ) suggested that a sparse ( Cauchy ) slowness prior improves correspondence to biological complex cells over the L2 slowness prior in a two-layer model . However , to the best of our knowledge , an L1 temporal prior has previously only been used in deep auto-encoder frameworks when applied to semi-supervised tasks ( Mobahi et al. , 2009 ; Zou et al. , 2012 ) , and was mentioned in Cadieu and Olshausen ( 2012 ) , who used an L2 prior , but claimed that an L1 prior performed similarly on their task . Similar to Hyvärinen et al . ( Hyvärinen and Morioka , 2016 ; Hyvärinen et al. , 2018 ) , we only assume that the latent factors are temporally dependent , thus avoiding assuming knowledge of the number of factors where the two observations differ ( Shu et al. , 2019 ; Locatello et al. , 2020 ) . Most of the standard datasets for disentanglement ( dSprites ( Matthey et al. , 2017 ) , Cars3D ( Reed et al. , 2015 ) , SmallNORB ( LeCun et al. , 2004 ) , Shapes3D ( Kim and Mnih , 2018 ) , MPI3D ( Gondal et al. , 2019 ) ) have been compiled into a disentanglement library ( DisLib ) by Locatello et al . ( 2018 ) . However , all of the DisLib datasets are limited in that the data generating process is independent and identically distributed ( i.i.d . ) and all generative factors are assumed to be discrete . In a follow-up study , Locatello et al . ( 2020 ) proposed combining pairs of images such that only k factors change , as this matches their modeling assumptions required to prove identifiability . Here , k ∈ U { 1 , D− 1 } and D denotes the number of ground-truth factors , which are then sampled uniformly . We additionally use the measurements from Fig . 1 to construct datasets for evaluating disentanglement that have time transitions which directly correspond to natural dynamics . 1As in slow feature analysis , we consider learning from videos without labels as unsupervised . 3 THEORY . 3.1 GENERATIVE MODEL . We have provided evidence to support the hypothesis that generative factors of natural videos have sparse temporal transitions ( see Fig . 1 ) . To model this process , we assume temporally adjacent input pairs ( xt−1 , xt ) coming from a nonlinear generator that maps factors to images x = g ( z ) , where generative factors are dependent over time : p ( zt , zt−1 ) = p ( zt|zt−1 ) p ( zt−1 ) . ( 1 ) Assume the observed data ( xt , xt−1 ) comes from the following generative process , where different latent factors are assumed to be independent ( cf . Appendix F.2 ) : x = g ( z ) , p ( zt−1 ) = d∏ i=1 p ( zt−1 , i ) , p ( zt|zt−1 ) = d∏ i=1 αλ 2Γ ( 1/α ) exp− ( λ|zt , i − zt−1 , i|α ) , ( 2 ) where λ is the distribution rate , p ( zt−1 ) is a factorized Gaussian prior N ( 0 , I ) ( as in Kingma and Welling , 2013 ) and p ( zt|zt−1 ) is a factorized generalized Laplace distribution ( Subbotin , 1923 ) with shape parameter α , which determines the shape and especially the kurtosis of the function.2 Intuitively , smaller α implies larger kurtosis and sparser temporal transitions of the generative factors ( special cases are Gaussian , α = 2 , and Laplacian , α = 1 ) . Critically , for our proof we assume α < 2 to ensure that temporal transitions are sparse . The novelty of our approach lies in our explicit modeling of sparse transitions that cover the statistics of natural data , which results in a stronger identifiability proof than previously achieved ( see Appendix F.1.1 for a more detailed comparison with Hyvärinen and Morioka , 2017 ; Khemakhem et al. , 2020a ) .

Summary: The paper starts with an observation that temporal transitions in sequences of natural images are sparse, which is supported by data collected from two big datasets (youtube-vots and kitti-mots). This suggests using a sparse prior for temporal transitions of latent variables when modelling naturalistic scenes. The authors then introduce SlowVAE, a model for temporal independent component analysis (ICA), that depends on such a sparsity prior, and prove that latent variables are identifiable under this model up to permutation and sign flips, which, they say, is stronger than any previous result. Additionally, this work introduces a number of datasets of increasing complexity (and similarity to natural datasets) for testing disentanglement as well as it performs a large scale and detailed evaluation of the introduced model.