Disentangling Identifiable Features from Noisy Data with Structured Nonlinear ICA

1 Introduction . A central tenet of unsupervised deep learning is that noisy and high dimensional real world data is generated by a nonlinear transformation of lower dimensional latent factors . Learning such lower dimensional features is valuable as they may allow us to understand complex scientific observations in terms of much simpler , semantically meaningful , representations ( Morioka et al. , 2020 ; Zhou and Wei , 2020 ) . Access to a ground truth generative model and its latent features would also greatly enhance several other downstream tasks such as classification ( Klindt et al. , 2021 ; Banville et al. , 2021 ) , transfer learning ( Khemakhem et al. , 2020b ) , as well as causal inference ( Monti et al. , 2019 ; Wu and Fukumizu , 2020 ) . A recently popular approach to deep representation learning has been to learn disentangled features . Whilst not rigorously defined , the general methodology has been to use deep generative models such as VAEs ( Kingma and Welling , 2014 ; Higgins et al. , 2017 ) to estimate semantically distinct factors of variation that generate and encode the data . A substantial problem with the vast majority of work on disentanglement learning is that the models used are not identifiable – that is , they do not learn the true generative features , even in the limit of infinite data – in fact , this task has been proven ∗hermanni.halva @ helsinki.fi †Equal senior authorship 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . impossible without inductive biases on the generative model ( Hyvärinen and Pajunen , 1999 ; Locatello et al. , 2019 ) . Lack of identifiability plagues deep learning models broadly and has been implicated as one of the reasons for unexpectedly poor behaviour when these models are deployed in real world applications ( D ’ Amour et al. , 2020 ) . Fortunately , in many applications the data have dependency structures , such as temporal dependencies which introduce inductive biases . Recent advances in both identifiability theory and practical algorithms for nonlinear ICA ( Hyvärinen and Morioka , 2016 , 2017 ; Hälvä and Hyvärinen , 2020 ; Morioka et al. , 2021 ; Klindt et al. , 2021 ; Oberhauser and Schell , 2021 ) exploit this and offer a principled approach to disentanglement for such data . Learning statistically independent nonlinear features in such models is well-defined , i.e . those models are identifiable . However , the existing nonlinear ICA models suffer from numerous limitations . First , they only exploit specific types of temporal structures , such as either temporal dependencies or nonstationarity . Second , they often work under the assumption that some ’ auxiliary ’ data about a latent process is observed , such as knowledge of the switching points of a nonstationary process as in Hyvärinen and Morioka ( 2016 ) ; Khemakhem et al . ( 2020a ) . Furthermore , all the nonlinear ICA models cited above , with the exception of Khemakhem et al . ( 2020a ) , assume that the data are fully observed and noise-free , even though observation noise is very common in practice , and even Khemakhem et al . ( 2020a ) assumes the noise distribution to be exactly known . This approach of modelling observation noise explicitly is in stark contrast to the approach taken in papers , such as Locatello et al . ( 2020 ) , who instead consider general stochasticity of their model to be captured by latent variables – this approach would be ill-suited to the type of denoising one would often need in practice . Lastly , the identifiability theorems in previous nonlinear ICA works usually restrict the latent components to a specific class of models such as exponential families ( but see Hyvärinen and Morioka ( 2017 ) ) . In this paper we introduce a new framework for identifiable disentanglement , Structured Nonlinear ICA ( SNICA ) , which removes each of the aforementioned limitations in a single unifying framework . Furthermore , the framework guarantees identifiability of a rich class of nonlinear ICA models that is able to exploit dependency structures of any arbitrary order and thus , for instance , extends to spatially structured data . This is the first major theoretical contribution of our paper . The second important theoretical contribution of our paper proves that models within the SNICA framework are identifiable even in the presence of additive output noise of arbitrary , unknown distribution . We achieve this by extending the theorems by Gassiat et al . ( 2020b , a ) . The subsequent practical implication is that SNICA models can perform dimensionality reduction to identifiable latent components and de-noise observed data . We note that noisy-observation part of the identifiability theory is not even limited to nonlinear ICA but applies to any system observed under noise . Third , we give mild sufficient conditions , relating to the strength and the non-Gaussian nature of the temporal or spatial dependencies , enabling identifiability of nonlinear independent components in this general framework . An important implication is that our theorems can be used , for example , to develop models for disentangling identifiable features from spatial or spatio-temporal data . As an example of the flexibility of the SNICA framework , we present a new nonlinear ICA model called ∆-SNICA . It achieves the following very practical properties which have previously been unattainable in the context of nonlinear ICA : the ability to account for both nonstationarity and autocorrelation in a fully unsupervised setting ; ability perform dimensionality reduction ; model latent states ; and to enable principled estimation and inference by variational maximum-likelihood methods . We demonstrate the practical utility of the model in an application to noisy neuroimaging data that is hypothesized to contain meaningful lower dimensional latent components and complex temporal dynamics . 2 Background . We start by giving some brief background on Nonlinear ICA and identifiability . Consider a model where the distribution of observed data x is given by pX ( x ; θ ) for some parameter vector θ . This model is called identifiable if the following condition is fulfilled : ∀ ( θ , θ′ ) pX ( x ; θ ) = pX ( x ; θ′ ) ⇒ θ = θ′ . ( 1 ) In other words , based on the observed data distribution alone , we can uniquely infer the parameters that generated the data . For models parameterized with some nonparametric function estimator f , such as a deep neural network , we can replace θ with f in the equation above . In practice , identifiability might hold for some parameters , not all ; and parameters might be identifiable up to some more or less trivial indeterminacies , such as scaling . In a typical nonlinear ICA setting we observe some x ∈ RN which has been generated by an invertible nonlinear mixing function f from latent independent components s ∈ RN , with p ( s ) = ∏N i=1 p ( s ( i ) ) , as per : x = f ( s ) , ( 2 ) Identifiability of f would then mean that we can in theory find the true f , and subsequently the true data generating components . Unfortunately , without some additional structure this model is unidentifiable , as shown by Hyvärinen and Pajunen ( 1999 ) : there is an infinite number of possible solutions and these have no trivial relation with each other . To solve this problem , previous work ( Sprekeler et al. , 2014 ; Hyvärinen and Morioka , 2016 , 2017 ) developed models with temporal structure . Such time series models were generalized and expressed in a succinct way by Hyvärinen et al . ( 2019 ) ; Khemakhem et al . ( 2020a ) by assuming the independent components are conditionally independent upon some observed auxiliary variable ut : p ( st|ut ) = ∏N i=1 p ( s ( i ) t |ut ) . In a time series context , the auxiliary variable might be history , e.g . ut = xt−1 , or the index of a time segment to model nonstationarity ( or piece-wise stationarity ) . ( It could also be data from another modality , such as audio data used to condition video data ( Arandjelovic and Zisserman , 2017 ) . ) Notice that the mixing function f in ( 2 ) is assumed bijective and thus identifiable dimension reduction is not possible in most of the models discussed above . The only exceptions , we are aware of , are Khemakhem et al . ( 2020a ) ; Klindt et al . ( 2021 ) who choose f as injective rather than bijective . Further , Khemakhem et al . ( 2020a ) assume additive noise on the observations x = f ( s ) + ε , which allows to estimate posterior of s by an identifiable VAE ( iVAE ) . We will take a similar strategy in what follows . 3 Definition of Structured Nonlinear ICA . In this section , we first present the new framework of Structured Nonlinear ICA ( SNICA ) – a broad class of models for identifiable disentanglement and learning of independent components when data has structural dependencies . Next , we give an example of a particularly useful specific model that fits within our framework , called ∆-SNICA , by using switching linear dynamical latent processes . 3.1 Structured Nonlinear ICA framework . Consider observations ( xt ) t∈T = ( ( x ( 1 ) t , . . . , x ( M ) t ) ) t∈T where T is a discrete indexing set of arbitrary dimension . For discrete time-series models , like previous works , T would be a subset of N. Crucially , however , we allow it to be any arbitrary indexing variable that describes a desired structure . For instance , T could be a subset of N2 for spatial data . We assume the data is generated according the following nonlinear ICA model . First , there exist latent components s ( i ) = ( s ( i ) t ) t∈T for i ∈ { 1 , . . . , N } where for any t , t′ ∈ T , the distributions of ( s ( i ) t ) 16i6N and ( s ( i ) t′ ) 16i6N are the same , which is a weak form of stationarity . Second , we assume that for any m ∈ N∗ and ( t1 , . . . , tm ) ∈ Tm , p ( st1 , . . . , stm ) = ∏N i=1 p ( s ( i ) t1 , . . . , s ( i ) tm ) : that is , the components are unconditionally independent . We further assume that the nonlinear mixing function f : RN → RM with M > N is injective , so there may be more observed variables than components . Finally , denote observational noise by εt ∈ RM and assume that they are i.i.d . for all t ∈ T and independent of the signals s ( i ) . Putting these together , we assume the mixing model where for each t ∈ T , xt = f ( st ) + εt , ( 3 ) where st = ( s ( 1 ) t , . . . , s ( N ) t ) . Importantly , εt can have any arbitrary unknown distribution , even with dependent entries ; in fact , it may even not have finite moments . The main appeal of this framework is that , under the conditions given in next section , we can now guarantee identifiability for a very broad and rich class of models . First , notice that all previous Nonlinear ICA time-series models can be reformulated and often improved upon when viewed through this new unifying framework . In other words , we can create x1 x2 x3 s ( i ) 1 s ( i ) 2 s ( i ) 3 u ( i ) 1 u ( i ) 2 u ( i ) 3 1 6 i 6 N ( a ) HMM modulated components c.f . ( Hälvä and Hyvärinen , 2020 ) models that are very much like those previous works , and capture their dependency profiles , but with the changes that by assuming unconditional independence and output noise we now allow them to perform dimension reduction ( this does also require some additional assumptions needed in our identifiability theorems below ) . To see this , consider the model in Hälvä and Hyvärinen ( 2020 ) which captures nonstationarity in the independent components through a global hidden Markov chain . We can transform this model into the SNICA framework if we instead model each independent component as its own HMM ( Figure 1a ) , with the added benefit that we now have marginally independent components and are able to perform dimensionality reduction into low dimensional latent components . Nonlinear ICA with time-dependencies , such as in an autoregressive model , proposed by Hyvärinen and Morioka ( 2017 ) is also a special case of our framework ( Figure 1b ) , but again with the extension of dimensionality reduction . Furthermore , this framework allows for a plethora of new Nonlinear ICA models to be developed . As described above , these do not have to be limited to time-series but could for instance be a process on a two-dimensional graph with appropriate ( in ) dependencies ( see Figure 1c ) . However , we now proceed to introduce a particularly useful time-series model using our framework .

Summary: The authors present a new disentanglement framework for structured data. They claim that the method generalizes to scenarios with spatial dependencies, independent additive noise, and non-square mixing processes. They demonstrate the utility of the framework by applying an approximate model to toy data as well as MEG data.