Disentangling Identifiable Features from Noisy Data with Structured Nonlinear ICA
Year: 2021

Abstract:
We introduce a new general identifiable framework for principled disentanglement referred to as Structured Nonlinear Independent Component Analysis (SNICA). Our contribution is to extend the identifiability theory of deep generative models for a very broad class of structured models. While previous works have shown identifiability for specific classes of time-series models, our theorems extend this to more general temporal structures as well as to models with more complex structures such as spatial dependencies. In particular, we establish the major result that identifiability for this framework holds even in the presence of noise of unknown distribution. Finally, as an example of our framework’s flexibility, we introduce the first nonlinear ICA model for time-series that combines the following very useful properties: it accounts for both nonstationarity and autocorrelation in a fully unsupervised setting; performs dimensionality reduction; models hidden states; and enables principled estimation and inference by variational maximum-likelihood.

heading: 1 Introduction
text: A central tenet of unsupervised deep learning is that noisy and high dimensional real world data is generated by a nonlinear transformation of lower dimensional latent factors. Learning such lower dimensional features is valuable as they may allow us to understand complex scientific observations in terms of much simpler, semantically meaningful, representations (Morioka et al., 2020; Zhou and Wei, 2020). Access to a ground truth generative model and its latent features would also greatly enhance several other downstream tasks such as classification (Klindt et al., 2021; Banville et al., 2021), transfer learning (Khemakhem et al., 2020b), as well as causal inference (Monti et al., 2019; Wu and Fukumizu, 2020).
A recently popular approach to deep representation learning has been to learn disentangled features. Whilst not rigorously defined, the general methodology has been to use deep generative models such as VAEs (Kingma and Welling, 2014; Higgins et al., 2017) to estimate semantically distinct factors of variation that generate and encode the data. A substantial problem with the vast majority of work on disentanglement learning is that the models used are not identifiable – that is, they do not learn the true generative features, even in the limit of infinite data – in fact, this task has been proven ∗hermanni.halva@helsinki.fi †Equal senior authorship
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
impossible without inductive biases on the generative model (Hyvärinen and Pajunen, 1999; Locatello et al., 2019). Lack of identifiability plagues deep learning models broadly and has been implicated as one of the reasons for unexpectedly poor behaviour when these models are deployed in real world applications (D’Amour et al., 2020). Fortunately, in many applications the data have dependency structures, such as temporal dependencies which introduce inductive biases. Recent advances in both identifiability theory and practical algorithms for nonlinear ICA (Hyvärinen and Morioka, 2016, 2017; Hälvä and Hyvärinen, 2020; Morioka et al., 2021; Klindt et al., 2021; Oberhauser and Schell, 2021) exploit this and offer a principled approach to disentanglement for such data. Learning statistically independent nonlinear features in such models is well-defined, i.e. those models are identifiable.
However, the existing nonlinear ICA models suffer from numerous limitations. First, they only exploit specific types of temporal structures, such as either temporal dependencies or nonstationarity. Second, they often work under the assumption that some ’auxiliary’ data about a latent process is observed, such as knowledge of the switching points of a nonstationary process as in Hyvärinen and Morioka (2016); Khemakhem et al. (2020a) . Furthermore, all the nonlinear ICA models cited above, with the exception of Khemakhem et al. (2020a), assume that the data are fully observed and noise-free, even though observation noise is very common in practice, and even Khemakhem et al. (2020a) assumes the noise distribution to be exactly known. This approach of modelling observation noise explicitly is in stark contrast to the approach taken in papers, such as Locatello et al. (2020), who instead consider general stochasticity of their model to be captured by latent variables – this approach would be ill-suited to the type of denoising one would often need in practice. Lastly, the identifiability theorems in previous nonlinear ICA works usually restrict the latent components to a specific class of models such as exponential families (but see Hyvärinen and Morioka (2017)).
In this paper we introduce a new framework for identifiable disentanglement, Structured Nonlinear ICA (SNICA), which removes each of the aforementioned limitations in a single unifying framework. Furthermore, the framework guarantees identifiability of a rich class of nonlinear ICA models that is able to exploit dependency structures of any arbitrary order and thus, for instance, extends to spatially structured data. This is the first major theoretical contribution of our paper.
The second important theoretical contribution of our paper proves that models within the SNICA framework are identifiable even in the presence of additive output noise of arbitrary, unknown distribution. We achieve this by extending the theorems by Gassiat et al. (2020b,a). The subsequent practical implication is that SNICA models can perform dimensionality reduction to identifiable latent components and de-noise observed data. We note that noisy-observation part of the identifiability theory is not even limited to nonlinear ICA but applies to any system observed under noise.
Third, we give mild sufficient conditions, relating to the strength and the non-Gaussian nature of the temporal or spatial dependencies, enabling identifiability of nonlinear independent components in this general framework. An important implication is that our theorems can be used, for example, to develop models for disentangling identifiable features from spatial or spatio-temporal data.
As an example of the flexibility of the SNICA framework, we present a new nonlinear ICA model called ∆-SNICA . It achieves the following very practical properties which have previously been unattainable in the context of nonlinear ICA: the ability to account for both nonstationarity and autocorrelation in a fully unsupervised setting; ability perform dimensionality reduction; model latent states; and to enable principled estimation and inference by variational maximum-likelihood methods. We demonstrate the practical utility of the model in an application to noisy neuroimaging data that is hypothesized to contain meaningful lower dimensional latent components and complex temporal dynamics.

heading: 2 Background
text: We start by giving some brief background on Nonlinear ICA and identifiability. Consider a model where the distribution of observed data x is given by pX(x;θ) for some parameter vector θ. This model is called identifiable if the following condition is fulfilled:
∀(θ,θ′) pX(x;θ) = pX(x;θ′)⇒ θ = θ′ . (1)
In other words, based on the observed data distribution alone, we can uniquely infer the parameters that generated the data. For models parameterized with some nonparametric function estimator f , such as a deep neural network, we can replace θ with f in the equation above. In practice, identifiability
might hold for some parameters, not all; and parameters might be identifiable up to some more or less trivial indeterminacies, such as scaling.
In a typical nonlinear ICA setting we observe some x ∈ RN which has been generated by an invertible nonlinear mixing function f from latent independent components s ∈ RN , with p(s) = ∏N i=1 p(s
(i)), as per:
x = f(s) , (2)
Identifiability of f would then mean that we can in theory find the true f , and subsequently the true data generating components. Unfortunately, without some additional structure this model is unidentifiable, as shown by Hyvärinen and Pajunen (1999): there is an infinite number of possible solutions and these have no trivial relation with each other. To solve this problem, previous work (Sprekeler et al., 2014; Hyvärinen and Morioka, 2016, 2017) developed models with temporal structure. Such time series models were generalized and expressed in a succinct way by Hyvärinen et al. (2019); Khemakhem et al. (2020a) by assuming the independent components are conditionally independent upon some observed auxiliary variable ut: p(st|ut) = ∏N i=1 p(s (i) t |ut) . In a time series context, the auxiliary variable might be history, e.g. ut = xt−1, or the index of a time segment to model nonstationarity (or piece-wise stationarity). (It could also be data from another modality, such as audio data used to condition video data (Arandjelovic and Zisserman, 2017).)
Notice that the mixing function f in (2) is assumed bijective and thus identifiable dimension reduction is not possible in most of the models discussed above. The only exceptions, we are aware of, are Khemakhem et al. (2020a); Klindt et al. (2021) who choose f as injective rather than bijective. Further, Khemakhem et al. (2020a) assume additive noise on the observations x = f(s) + ε , which allows to estimate posterior of s by an identifiable VAE (iVAE). We will take a similar strategy in what follows.

heading: 3 Definition of Structured Nonlinear ICA
text: In this section, we first present the new framework of Structured Nonlinear ICA (SNICA) – a broad class of models for identifiable disentanglement and learning of independent components when data has structural dependencies. Next, we give an example of a particularly useful specific model that fits within our framework, called ∆-SNICA , by using switching linear dynamical latent processes.

heading: 3.1 Structured Nonlinear ICA framework
text: Consider observations (xt)t∈T = ((x (1) t , . . . , x (M) t ))t∈T where T is a discrete indexing set of arbitrary dimension. For discrete time-series models, like previous works, T would be a subset of N. Crucially, however, we allow it to be any arbitrary indexing variable that describes a desired structure. For instance, T could be a subset of N2 for spatial data. We assume the data is generated according the following nonlinear ICA model. First, there exist latent components s(i) = (s(i)t )t∈T for i ∈ {1, . . . , N} where for any t, t′ ∈ T, the distributions of (s
(i) t )16i6N and (s (i) t′ )16i6N are the same, which is a weak form of stationarity. Second, we assume that for any m ∈ N∗ and (t1, . . . , tm) ∈ Tm, p(st1 , . . . , stm) = ∏N i=1 p(s (i) t1 , . . . , s (i) tm): that is, the components are unconditionally independent. We further assume that the nonlinear mixing function f : RN → RM with M > N is injective, so there may be more observed variables than components. Finally, denote observational noise by εt ∈ RM and assume that they are i.i.d. for all t ∈ T and independent of the signals s(i). Putting these together, we assume the mixing model where for each t ∈ T,
xt = f(st) + εt , (3)
where st = (s (1) t , . . . , s (N) t ). Importantly, εt can have any arbitrary unknown distribution, even with dependent entries; in fact, it may even not have finite moments.
The main appeal of this framework is that, under the conditions given in next section, we can now guarantee identifiability for a very broad and rich class of models.
First, notice that all previous Nonlinear ICA time-series models can be reformulated and often improved upon when viewed through this new unifying framework. In other words, we can create
x1 x2 x3
s (i) 1 s (i) 2 s (i) 3
u (i) 1 u (i) 2 u (i) 3
1 6 i 6 N
(a) HMM modulated components c.f. (Hälvä and Hyvärinen, 2020)
models that are very much like those previous works, and capture their dependency profiles, but with the changes that by assuming unconditional independence and output noise we now allow them to perform dimension reduction (this does also require some additional assumptions needed in our identifiability theorems below). To see this, consider the model in Hälvä and Hyvärinen (2020) which captures nonstationarity in the independent components through a global hidden Markov chain. We can transform this model into the SNICA framework if we instead model each independent component as its own HMM (Figure 1a), with the added benefit that we now have marginally independent components and are able to perform dimensionality reduction into low dimensional latent components. Nonlinear ICA with time-dependencies, such as in an autoregressive model, proposed by Hyvärinen and Morioka (2017) is also a special case of our framework (Figure 1b), but again with the extension of dimensionality reduction. Furthermore, this framework allows for a plethora of new Nonlinear ICA models to be developed. As described above, these do not have to be limited to time-series but could for instance be a process on a two-dimensional graph with appropriate (in)dependencies (see Figure 1c). However, we now proceed to introduce a particularly useful time-series model using our framework.

heading: 3.2 ∆-SNICA : Nonlinear ICA with switching linear dynamical systems
text: While the above framework has great generality, any practical application will need a specific model. Next we propose one which combines the following properties of previous nonlinear ICA models into a single model: ability to account for both nonstationarity and autocorrelation in a fully unsupervised setting, to perform dimensionality reduction and model hidden states. Real world processes, such as video/audio data, financial time-series, and brain signals, exhibit these properties – disentangling latent features in such data would hence be very useful.
Our new model is depicted in Figure 1d. The independent components are generated by a Switching Linear Dynamical System (SLDS) (Ackerson and Fu, 1968; Chang and Athans, 1978; Hamilton, 1990; Ghahramani and Hinton, 2000) with additional latent variables to express rich dynamics. Formally, for each independent component i ∈ {1, . . . , N}, consider the following SLDS over some latent vector y(i)t :
y (i) t = B (i) ut y (i) t−1 + b (i) ut + ε (i) ut , (4)
where ut := u (i) t is a state of a first-order hidden Markov chain (u (i) t )t=1:T . Crucially, we assume that the independent components at each time-point are the first elements y(i)t,1 of y (i) t = (y (i) t,1, . . . , y (i) t,d) T , i.e. s(i)t = y (i) t,1. The rest of the elements in y (i) t are latent variables modelling hidden dynamics. The great utility of using such a higher-dimensional latent variable is that this model allows us, for example, as a special case, to consider higher-order ARMA processes, thus modelling each s(i)t as switching between ARMA processes of an order determined by the dimensionality of yt. We call the ensuing model ∆-SNICA ("Delta-SNICA", with delta as in "dynamic").

heading: 4 Identifiability
text: In this section, we present two very general identifiability theorems for SNICA. We basically decouple the problem into two parts. First, we consider identifying the noise-free distribution of f(st) from noisy data. Theorem 1 states conditions—on tail behaviour, non-degeneracy, and non-Gaussianity— under which it is possible to recover the distribution of a process based on noisy data with unknown noise distribution. Second, we consider demixing of the nonlinearly mixed data. Theorem 2 provides general conditions—on temporal or spatial dependencies, and non-Gaussianity—that allow recovery of the mixing function f when there is no more noise. We then consider application of these theorems to SNICA.

heading: 4.1 Identifiability with unknown noise distribution
text: Consider the model xt = zt + εt , (5)
where (zt)t∈T is a family of random variables in RM such that all zt, t ∈ T, have the same marginal distribution, and (εt)t∈T is a family of independent (over t) and identically distributed random variables, independent of (zt)t∈T. Let P be the common distribution of each εt, for t ∈ T. Let t1 and t2 in T, and consider the following assumptions.
• (A1) [Tail behaviour] For some ρ < 3, there exist A and B such that for all λ ∈ RN , E[exp(〈λ, zt1〉)] 6 A exp(B‖λ‖ρ) .
• (A2) [Non-degeneracy] For any η ∈ CM , E[exp{〈η, zt2〉}| zt1 ] is not the null random variable.
• (A3) [Non-Gaussianity] The following assertion is false: there exist a vector η ∈ RM and independent random variables z̃ and u, such that u is a non dirac Gaussian random variable and 〈η, zt1〉 has the same distribution as z̃ + u.
We defer the detailed discussion on the practical meaning of the assumptions (A1-A3) in the context of SNICA to Section 4.3. We next present Theorem 1 which establishes identifiability under unknown noise (its proof is postponed to Section A.1 in the Supplementary Material):
Theorem 1 Assume that assumptions (A1), (A2) and (A3) hold for some (t1, t2) ∈ T2. Then, up to translation, for all m > 2, for all (t3, . . . , tm) ∈ Tm−2, the application that associates the distribution of (zt1 , . . . , ztm) and P to the distribution of (xt1 , . . . ,xtm) is one-to-one.
Here, up to translation means that adding a constant vector to all εt, and substracting this constant to all zt, t ∈ {t1, . . . , tm}, does not change the distribution of (xt1 , . . . ,xtm). The proof of Theorem 1 extends that of Theorem 1 in (Gassiat et al., 2020b), see also (Gassiat et al., 2020a), which assumed sub-Gaussian noise-free data. Our extension allows the noise-free data to have heavier tails, which is important since (noise-free) data in many real-world applications is super-Gaussian, i.e. heavy-tailed, as is well-known in work on linear ICA (Hyvärinen et al., 2001).
Importantly, there is no assumption on the unknown noise distribution in Theorem 1. In fact, it does not even assume a mixing as in ICA, and thus extends greatly outside of the framework of this paper.

heading: 4.2 Identifiability of the mixing function
text: Based on Theorem 1, it is possible to recover the distribution of the noise-free data in SNICA in (3) by setting zt = f(st). Next, we consider under which conditions the mixing function f is identifiable. Denote by S = S(1) × · · · × S(N) the support of the distribution of all st. We consider the situation where each S(i) ⊂ R, 1 6 i 6 N , is connected, so that each S(i) is an interval. We assume moreover that the injective mixing function f is a C2 diffeomorphism between S and a C2 differentiable manifoldM ⊂ RM . Formally, this means that there exists an atlas {ϕϑ : Uϑ → RN}ϑ∈Θ ofM such that for all ϑ, ϑ′ ∈ Θ, the map ϕϑ ◦ ϕ−1ϑ′ is a C2 map, and f is a bijection RN →M such that for all ϑ ∈ Θ, ϕϑ ◦ f and f−1 ◦ ϕ−1ϑ have continuous second derivatives. The sets Uϑ, ϑ ∈ Θ, cover M and are open inM. The proof of Theorem 2 is postponed to Section A.2 in the Supplementary Material.
Theorem 2 Assume that there exist m > 2 and (t1, . . . , tm) ∈ Tm such that the vector (s
(i) t1 , . . . , s (i) tm) has a density p (i) m which is C2 on (S(i))m. Assume moreover that there exist
(k, l) ∈ {1, . . . ,m}2 with k 6= l such that the following assumptions hold with Q(i)m = log p(i)m .
• (B1) (Uniform (k, l)-dependency). For all i ∈ {1, . . . , N}, the set of zeros of ∂ 2
∂s (i) tk ∂s (i) tl
Q (i) m
is a meagre subset of (S(i))m, i.e. it contains no open subset.
• (B2) (Local (k, l)-non quasi Gaussianity). For any open subset A ⊂ Sm, there exists at most one i ∈ {1, . . . , N} such that there exists a function α : Rm−1 → R and a constant c ∈ R such that for all s ∈ A,
∂2
∂s (i) tk ∂s (i) tl
Q(i)m = c α(s (i) tk , s (i) (−tk,−tl))α(s (i) tl , s (i) (−tk,−tl)) , (6)
where s(i)(−tk,−tl) is (s (i) t1 , . . . , s (i) tm) without the coordinates tk and tl.
Then, f−1 can be recovered up to permutation and coordinate-wise transformations from the distribution of (f(st1), . . . , f(stm)).

heading: 4.3 Applications to SNICA
text: In this section, we provide additional comments on the assumptions (A1-A3) and (B1-B2) and their verification in the context of SNICA.
Assumption (A1) is a condition on the tails of the noise-free data: it allows tails that are somewhat heavier than Gaussian tails. It is in fact equivalent to assuming that for some ρ̃ > 3/2, there exists A′, B′ > 0 such that for all t > 0, P(‖zt1‖ > t) 6 A′ exp(−B′tρ̃).
Assumption (A2) is a non-degeneracy condition likely to be fulfilled for any randomly chosen SNICA model parameters. As an example, consider a model such as Fig. 1c, where there exist hidden variables (ut)t∈T taking values in a finite set {1, . . . ,K} such that the pairs of variables (zt, ut) have the same distribution for all t ∈ T, and such that conditioned on (ut)t∈T, the variables (zt)t∈T are independent and the distribution of zt only depends on ut. (As a special case, this model includes the temporal HMM setting described in Fig. 1a.) Let (t1, t2) ∈ T2. For all u, v ∈ {1, . . . ,K}, let π(u) = put1 (u) be the mass function of ut1 , Q(u, v) = put2 |ut1 (v|u) be the transition matrix from ut1 to ut2 , and γu(z) = pzt1 |ut1 (z|u) be the density of zt1 conditionally to ut1 = u. By assumption, it is also the density of zt2 conditionally to ut2 = u. Theorem 3 provides sufficient conditions for assumption (A2) to hold:
Theorem 3 Assume that Q has full rank, minu π(u) > 0 and the (γu)16u6K are linearly independent, then (A2) is satisfied as soon as the functions (η 7→ ∫ exp(〈η, z〉)γv(z)dz)16v6K do not have simultaneous zeros.
Besides the non-simultaneous zeros assumption, the assumptions of Theorem 3 are reminiscent of those used for the identifiability of non-parametric hidden Markov models, see for instance Gassiat et al. (2016); Lehéricy (2019). The key element is that zt1 and zt2 are not independent. Thus, we see that (A2) holds if the π and the γ are not degenerate (in the precise sense given by Theorem 3), for the latent state models in Figs. 1a,1c.Another situation where (A2) holds is when zt2 is a complete statistic (Lehmann and Casella, 2006) in the statistical model {Pzt2 |zt1 (·|zt1)}zt1 , where Pzt2 |zt1 (·|zt1) is the distribution of zt2 conditionally to zt1 . Consider the two following examples where this holds: 1) When the model {Pzt2 |zt1 (·|zt1)}zt1 is an exponential family. In this situation, complete statistics are known. 2) Autoregressive models with additive innovation of the form zt2 = h(zt1) + vt2 for some bijective function h when the additive noise vt2 is a complete statistic in the statistical model {Pvt2 |zt1 (·|zt1)}zt1 (note that vt2 cannot be independent of zt1 here). The case in Fig. 1b is typically covered by this example.
Assumption (A3) states that no direction of the noise free data has a non Dirac Gaussian variable component. It holds as soon as zt = f(st) and the range of f is such that its orthogonal projection on
any line is not the full line. This assumption holds for instance in the following cases: 1) The range of f is compact, or 2) the range of f is contained in a half-cylinder, that is, there exists a hyperplane such that the range of f is only on one side of this hyperplane and the projection of the range of f on this hyperplane is bounded.
Assumption (B1) and Assumption (B2) are similar to those in (Hyvärinen and Morioka, 2017; Oberhauser and Schell, 2021) in the special case of time-series, i.e. T = N. (B1) then entails that there must be sufficiently strong statistical dependence between nearby time points. (B2) is a condition which excludes Gaussian processes and processes which can be trivially transformed to be Gaussian. (For treatment of the Gaussian case, see Appendix B in Supplementary Material.) We can further provide a simple and equivalent formulation when the independent components s(i) follow independent and stationary HMMs with two hidden states, which is a special case of SNICA. Denote by γ(i)0 and γ (i) 1 the densities of s (i) t conditionally to {u (i) t = 0} and {u (i) t = 1} respectively.
Theorem 4 Assume that the stationary distribution π of the hidden chain is such that 0 < π(0) < 1 and that its transition matrix is invertible. Then (B1) and (B2) are satisfied with m = 2 if and only if on any open interval, γ(i)0 and γ (i) 1 are not proportional.
Thus, a very simple HMM leads to these conditions being verified. Hyvärinen and Morioka (2017) already showed that the conditions (B1) and (B2) also hold in the case of non-Gaussian autoregressive models. Thus, we see that our identifiability theory applies both in the case HMM’s (Fig 1a) and autoregressive models (Fig 1b), the two principal kinds of temporal structure proposed in previous work, while extending them to further cases and combinations such as in Fig 1c,1d.
A simplification of (B1,B2) It is also possible to combine the assumptions (B1) and (B2) in one, while slightly weakening the generality. The key is to notice that (6) in (B2) implies the derivative in (B1) is zero, by setting c = 0. But there is still the difference that (B2) considers all but one index while (B1) considers all indices i. If we simply assume (6) does not hold for any i, we can replace (B1) and (B2) by the new condition:
• (B’) For any open subset A ⊂ Sm and for any i ∈ {1, . . . , N}, a function α : Rm−1 → R and a constant c ∈ R do not exist such that (6) would hold for all s ∈ A.
Note that Hyvärinen and Morioka (2017) defined uniform dependency and (non-)quasi-Gaussianity as two separate properties, but in fact their assumption of non-quasi-Gaussianity was weaker than ours: it did not consider all open subsets separately, which is why this simplification was not possible for them. We believe their definition of non-quasi-Gaussianity was in fact not quite sufficient to prove their theorem, and our stronger version may be needed, in line with Oberhauser and Schell (2021).

heading: 5 Experiments
text: Estimation method One challenge is that it is not practically possible to learn ∆-SNICA by exact maximum-likelihood methods. Instead, we perform learning and inference using Structured VAEs (Johnson et al., 2016) – the current state-of-art in variational inference for structured models. Specifically, this consists of assuming that the latent posterior factorizes as per q(y(1:N)1:T , u
(1:N) 1:T ) =∏N
i=1 q(y (i) 1:T )q(u (i) 1:T ), which allows us to optimize the resulting evidence lower bound (ELBO): log L̂ = Eq [ T∑ t=1 log p(xt | s(1)t , ..., s (N) t ) ] + N∑ i=1 ( −KL [ q(u (i) 1:T ) ∣∣∣∣p(u(i)1:T )]+ H[q(s(i)1:T )]
+ Eq [ log p(s (i) 1 | u (i) 1 ) ] + T∑ t=2 Eq [ log p(s (i) t | s (i) t−1, u (i) t ) ]) . (7)
Since all the distributions are in conjugate exponential families (encoder neural network is used to approximate the natural parameters of the nonlinear likelihood term) efficient message passing can be used for inference, and the mixing function is learned as decoder neural network. Even though this method lacks consistency guarantees (but see Wang and Blei (2018)), we find that our model performs very well. A more detailed treatment of estimation and inference of ∆-SNICA is given in Appendix C. Our code will be openly available at https://github.com/HHalva/snica.

heading: 5.1 Experiments on simulated data
text: The identifiability theorems stated above hold in the limit of infinite data. Additionally, a consistent estimator would be required to learn the ground-truth components. In the real world, we are limited by data and estimation methods and hence it is unclear as to what extent we are actually able to estimate identifiable components – and whether identifiability reflects in better performance in real world tasks. To explore this, we first performed experiments on simulated data. We compared the performance of our model to the current state-of-the-art, IIA-HMM (Morioka et al., 2021), as well as identifiable VAE (iVAE) (Khemakhem et al., 2020a) and standard linear Gaussian state-space model (LGSSM). Since iVAE is not able to handle latent auxiliary variables, we allow it to "cheat" by giving it access to the true data generating latent-state, thereby creating a presumably challenging baseline (denoted iVAE∗ in our figures). LGSSM was included as a naive baseline which is only able to estimate linear mixing function.
Investigating identifiability and consistency We simulated 100K long time-sequences from the ∆- SNICA model and computed the mean absolute correlation coefficient (MCC) between the estimated latent components and ground truth independent components (see Supplementary material for further implementation details). More precisely, to illustrate the dimensionality reduction capabilities we considered two settings where the observed data dimension M , was either 12 or 24 and the number of independent components, N was 3 and 6, respectively. Since IIA-HMM is unable to do dimensionality reduction, we used PCA to get the data dimension to match that of the latent states. We considered four levels of mixing of increasing complexity by randomly initialized MLPs of the following number of layers: 1 (linear ICA), 2, 3, and 5. The results in Figure 2a) illustrate the clearly superior performance of our model. The especially poor performance of IIA-HMM maybe explained by lack of noise model, much simpler latent dynamics, and lost information due to PCA pre-processing. See Appendix D for further discussion and training details.
Application to denoising ∆-SNICA is able to denoise time-series signals by learning the generative model and then performing inference on latent variables. Specifically, SVAE learns the encoder network which is used to perform inference on the posterior of the independent components. We illustrate this using the same settings as above, with the exception that we now use our learned encoder and inference to get the posterior means of the independent components and input these in to the estimated decoder to get predicted noise-free observations, denoted as f̂(st) – we measured the correlation between f̂(st) and the ground-truth f(st). Note that IIA-HMM, is not able to perform this task. The results in Figure 2b) show that the other models, designed to handle denoising, perform well at this task, as would be expected – identifiability of the latent state is not necessary for good denoising performance. For LGSSM, denoising is done with the Kalman Smoother algorithm.

heading: 5.2 Experiments on real MEG data
text: To demonstrate real-data applicability, ∆-SNICA was applied to multivariate time series of electrical activity in the human brain, measured by magnetoencephalography (MEG). Recently, many studies have demonstrated the existence of fast transient networks measured by MEG in the resting state and the dynamic switching between different brain networks (Baker et al., 2014; Vidaurre et al., 2017). Additionally, such MEG data is high-dimensional and very noisy. Thus this data provides an excellent target for ∆-SNICA to disentangle the underlying low-dimensional components.
Data and Preprocessing We considered a resting state MEG sessions from the Cam-CAN dataset. During the resting state recording, subjects sat still with their eyes closed. In the task-session data, the subjects carried out a (passive) audio–visual task including visual stimuli and auditory stimuli. We exclusively used the resting-session data for the training of the network, and task-session data was only used in the evaluation. The modality of the sensory stimulation provided a class label that we used in the evaluation, giving in total two classes. We band-pass filtered the data between 4 Hz and 30 Hz (see Supplementary Material for the details of data and settings).
Methods The resting-state data from all subjects were temporally concatenated and used for training. The number of layers of the decoder and encoder were equal and took values 2, 3, 4. We fixed the number of independent components to 5 so that our result can be fairly compared to those in Morioka et al. (2021). To evaluate the obtained features, we performed classification of the sensory stimulation categories by applying feature extractors trained with (unlabeled) resting-state data to (labeled) task-session data. Classification was performed using a linear support vector machine (SVM) classifier trained on the stimulation modality labels and sliding-window-averaged features obtained for each trial. The performance was evaluated by the generalizability of a classifier across subjects. i.e., one-subject-out cross-validation. For comparison, we evaluated the baseline methods: IIA-HMM and IIA-TCL (Morioka et al., 2021). We also visualized the spatial activity patterns obtained by ∆-SNICA , using the weight vectors from encoder neural network across each layer.
Results Figure 3 a) shows the classification accuracies of the stimulus categories, across different methods and the number of layers for each model. The performances by ∆-SNICA were consistently higher than those by the other (baseline) methods, which indicates the importance of the modeling of the MEG signals by ∆-SNICA . Figure 3 b) shows an example of spatial patterns from the encoder network learned by the ∆-SNICA . We used the visualization method presented in (Hyvärinen and Morioka, 2016). We manually picked one out of the hidden nodes from the third layer in encoder network, and plotted its weighted-averaged sensor signals, We also visualized the most strongly contributing second- and first-layer nodes. We see progressive pooling of L1 units to form left lateral frontal, right lateral frontal and parietal patterns in L2 which are then all pooled together in L3 resulting in a lateral frontoparietal pattern. Most of the spatial patterns in the third layer (not shown) are actually similar to those previously reported using MEG (Brookes et al., 2011). Appendix E provides more detail to the interpretation of the ∆-SNICA results.

heading: 6 Related work
text: Previous works on nonlinear ICA have exploited autocorrelations (Hyvärinen and Morioka, 2017; Oberhauser and Schell, 2021) and nonstationarities (Hyvärinen and Morioka, 2016; Hälvä and Hyvärinen, 2020) for identifiability. The SNICA setting provides a unifying framework which allows for both types of temporal dependencies, and further, extends identifiability to other temporal structures as well as any arbitrary higher order data structures which has not previously been considered in the context of nonlinear ICA. Another major theoretical contribution here is to show that identifiability with noise of unknown, arbitrary distribution, while previous work on noisy nonlinear ICA assumed noise of known distribution and known variance (Khemakhem et al., 2020a).
Importantly, the SNICA framework is fully probabilistic and thus accommodates higher order latent variables, leading to "purely unsupervised" learning. This is in large contrast to previous research which have been developed for the case where we are able to observe some additional auxiliary variable, such as audio signals accompanying video (Hyvärinen et al., 2019; Khemakhem et al., 2020a,b), or heuristically define the auxiliary variable based on time structure (Hyvärinen and Morioka, 2016). In practice this means that we are able to estimate our models using (variational)
MLE, which is more principled than the heuristic self-supervised methods in most earlier papers. The only existing frameworks allowing MLE (Hälvä and Hyvärinen, 2020; Khemakhem et al., 2020a) used model restricted to exponential families, and had either no HMM or a very simple one.
The switching linear dynamical model, ∆-SNICA in Section 3.2, shows the above benefits in the form of a single model. That is, unlike previous nonlinear ICA models, it combines: 1) temporal dependencies and "non-stationarity" (or HMM) in a single model 2) dimensionality reduction within a rigorous maximum likelihood learning and inference framework, and 3) a separate observation equation with general observational noise. This results in a very rich, realistic, and principled model for time series.
Very recently, Morioka et al. (2021) proposed a related model by considering innovations of time series to be nonstationary. However, their model is noise-free, restricted to exponential families of at least order two, and not applicable to the spatial case, thus making our identifiability results significantly stronger. From a more practical viewpoint, their model suffers from the fact that it either does not allow for dimensionality reduction (if an HMM is used) or requires a manual segmentation (if HMM is not used). Nor does it have a clear distinction into a state dynamics equation and a measurement equation which allows for cleaning or denoising of the data.
Limitations Our identifiability theory makes some restrictive assumptions, and it remains to be seen if they could be lifted in future work. In particular, the data is not allowed to have too heavy tails; the noise must be additive, and independent of the signal; and the practical interpretation of some of the assumptions, such as (A3) is difficult. It is also difficult to say whether our assumption of unconditionally independent components is realistic in practice. Regarding practical applications, our specific model only scratches the surface of what is possible in this framework. In particular, we did not develop a model with spatial distributions, nor did we model non-Gaussian observational noise – our main aim was to lay the foundations for the relevant identification theory. Future work should aim to make the estimation more efficient computationally; this is a ubiquitous problem in deep learning, but specific solutions for this concrete problem may be achievable (Gresele et al., 2020).

heading: 7 Conclusion
text: We proposed a new general framework for identifiable disentanglement, based on nonlinear ICA with very general temporal dynamics or spatial structure. Observational noise of arbitrary unknown distribution is further included. We prove identifiability of the models in this framework with high generality and mathematical rigour. For real data analysis, we propose a special case which subsumes the properties of all existing time series models in nonlinear ICA, while generalizing them in many ways (see Section 6 for details). We hope this work will contribute to wide-spread application of identifiable methods for disentanglement in a highly principled, probabilistic framework.

Review summaries:
The authors provide an identifiability result for non-linear ICA for a class of time series models. There is a clear contribution with respect to previous work that was typically restricted to condionnally independent sources, conditionned on a hidden markov chain in finite stare spaces. This new result allows identifying independent sources with a specific form of time dependency.

This paper proposed a new general identifiable nonlinear ICA framework for structured data. They proved identifiability for this general framework. They also proposed a specific application model from the framework and applied it to the simulated and real data. 

The authors present a new disentanglement framework for structured data. They claim that the method generalizes to scenarios with spatial dependencies, independent additive noise, and non-square mixing processes. They demonstrate the utility of the framework by applying an approximate model to toy data as well as MEG data.