Disentangling Identifiable Features from Noisy Data with Structured Nonlinear ICA
Authors: Hermanni Hälvä
Sylvain Le Corff
Luc Lehéricy
Jonathan So
Yongjie Zhu
Elisabeth Gassiat
Aapo Hyvarinen
Venue: NeurIPS 2021 Poster (2021)
Keywords: nonlinear ICA
ICA
identifiable
disentanglement
representation learning
deep generative models
denoising

Abstract: We introduce a new general identifiable framework for principled disentanglement referred to as Structured Nonlinear Independent Component Analysis (SNICA). Our contribution is to extend the identifiability theory of deep generative models for a very broad class of structured models. While previous works have shown identifiability for specific classes of time-series models, our theorems extend this to more general temporal structures as well as to models with more complex  structures such as spatial dependencies. In particular, we establish the major result that identifiability for this framework holds even in the presence of noise of unknown distribution. Finally, as an example of our framework's flexibility, we introduce the first nonlinear ICA model for time-series that combines the following very useful properties: it accounts for both nonstationarity and autocorrelation in a fully unsupervised setting;  performs dimensionality reduction;  models hidden states; and  enables principled estimation and inference by variational maximum-likelihood.

Disentangling Identifiable Features from Noisy Data with Structured Nonlinear ICA
Hermanni Hälvä
Sylvain Le Corff
Lue Lehéricy
Jonathan So
Yongjie Zhu
Elisabeth Gassiat
Aapo Hyvärinen' f
Department of Computer Science, University of Helsinki, Finland
Samovar, Télécom SudParis, département CITI, Institut Polytechnique de Paris, Palaiseau, France
Laboratoire J. A. Dieudonné, Université Côte d' Azur, CNRS, 06100, Nice, France
Department of Engineering, University of Cambridge, UK Université Paris-Saclay, CNRS, Laboratoire de mathématiques d' O'Orayy 91405, Orsay, France
Abstract
We introduce new general identifiable framework for principled disentanglement referred to as Structured Nonlinear Independent Component Analysis (SNICA). Our contribution to extend the identifiability theory of deep generative models for very broad class of structured models. While previous works have shown identifiability for specific classes time-series models, our theorems extend this more general temporal structures as well as to models with more complex structures particular, we establish the major result that identifiability for this framework holds even in the presence of noise of unknown distribution. Finally, as an example of our framework's s flexibility, we introduce the first nonlinear ICA model for time-series that combines the following very useful properties: it accounts for both nonstationarity and autocorrelation a fully unsupervised setting: performs dimensionality reduction; models hidden states; and enables principled estimation and inference by variational maximum likelihood.
such as spatial dependencies.
Introduction
A central tenet of unsupervised deep learning is that noisy and high dimensional real world data is generated by a nonlinear transformation of lower dimensional latent factors. Learning such lower dimensional features is valuable they may allow to understand complex scientific observations in terms of much simpler, semantically meaningful, representations (Morioka et al., 2020; Zhou and Wei, 2020). Access to ground truth generative model and its latent features would also greatly enhance several other downstream tasks such as classification (Klindt et al., 2021: 1; Banville et al., 2021), transfer learning (Khemakhem et al., 2020b), as well as causal inference (Monti al., 2019; Wu and Fukumizu, 2020).
A recently popular approach to deep representation learning has been to learn disentangled features. Whilst not rigorously defined, the general methodology has been to use deep generative models such as VAEs (Kingma and Welling, 2014; Higgins et al., 2017) estimate semantically distinct factors of variation that generate and encode the data. substantial problem with the vast majority of work on disentanglement learning is that the models used are not identifiable that is, they do not lear the true generative features, even in the limit of infinite data in fact, this task has been proven
hermanni. halva helsinki.fi Equal senior authorship
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

impossible without inductive biases on the generative model (Hyvärinen and Pajunen, 1999; Locatello et al., 2019). Lack of identifiability plagues deep learning models broadly and has been implicated as one of the reasons for unexpectedly poor behaviour when these models are deployed in real world applications (D' Amour et al., 2020). Fortunately, many applications the data have dependency structures, such as temporal dependencies which introduce inductive biases. Recent advances in both identifiability theory and practical algorithms for nonlinear ICA (Hyvärinen and Morioka, 2016, 2017; Hälvä and Hyvärinen, 2020; Morioka et al., 2021; Klindt et al., 2021; Oberhauser and Schell, 2021) exploit this and offer a principled approach to disentanglement for such data. Learning statistically independent nonlinear features in such models is well-defined, i.e. those models are identifiable.
However, the existing nonlinear ICA models suffer from numerous limitations. First, they only exploit specific types of temporal structures, such as either temporal dependencies or nonstationarity. Second, they often work under the assumption that some "auxiliary" data about a latent process observed, such as knowledge of the switching points of a nonstationary process as in Hyvärinen and Morioka (2016); Khemakhem et al. (2020a). Furthermore, all the nonlinear ICA models cited above, with the exception of Khemakhem et al. (2020a), assume that the data are fully observed and noise-free, even though observation noise is very common in practice, and even Khemakhem et al. (2020a) assumes the noise distribution to be exactly known. This approach of modelling observation noise explicitly is in stark contrast to the approach taken in papers, such as Locatello et al. (2020), who instead consider general stochasticity of their model to be captured by latent variables this approach would be ill-suited to the type of denoising one would often need in practice. Lastly, the identifiability theorems in previous nonlinear ICA works usually restrict the latent components to specific class of models such as exponential families (but see Hyvärinen and Morioka (2017)).
In this paper we introduce a new framework for identifiable disentanglement, Structured Nonlinear ICA (SNICA), which removes each of the aforementioned limitations in a single unifying framework. Furthermore, the framework guarantees identifiability of a rich class of nonlinear ICA models that is able to exploit dependency structures of any arbitrary order and thus, for instance, extends to spatially structured data. This is the first major theoretical contribution of our paper.
The second important theoretical contribution of our paper proves that models within the SNICA framework are identifiable even in the presence of additive output noise of arbitrary. unknown distribution. We achieve this by extending the theorems by Gassiat et al. (2020b,a). The subsequent practical implication is that SNICA models can perform dimensionality reduction to identifiable latent components and de-noise observed data. We note that woisy-observation part of the identifiability theory is not even limited to nonlinear ICA but applies to any system observed under noise.
Third, we give mild sufficient conditions, relating to the strength and the non-Gaussian nature of the temporal or spatial dependencies, enabling identifiability of nonlinear independent components in this general framework. An important implication is that our theorems can be used, for example, to develop models for disentangling identifiable features from spatial or apatio-temporu data.
As example of the flexibility of the SNICA framework, we present new nonlinear ICA model called A-SNICA. It achieves the following very practical properties which have previously been unattainable in the context of nonlinear ICA: the ability account for both nonstationarity and autocorrelation in a fully unsupervised setting; ability perform dimensionality reduction; model latent states; and enable principled estimation and inference by variational maximum-likelihon methods. We demonstrate the practical utility of the model in an application to noisy neuroimaging data that hypothesized contain meaningful lower dimensional latent components and complex temporal dynamics.
2 Background
We start by giving some brief background on Nonlinear ICA and identifiability. Consider a model where the distribution of observed data is given by px (x; 0) for some parameter vector 0. This model called identifiable if the following condition is fulfilled:
v(0,0)) 0')
px 0) - (x 0') 0'0
(1)
In other words, based on the observed data distribution alone, we can uniquely infer the parameters that generated the data. For models parameterized with some nonparametric function estimator f, such as deep neural network, we can replace 0 with f in the equation above. In practice, identifiability
2

---

might hold for some parameters, not all: and parameters might be identifiable up to some more or less trivial indeterminacies, such scaling.
In typical nonlinear ICA setting we observe some x E TI which has been generated by an invertible nonlinear mixing function f from latent independent components s E IRN, with p(s) = =[[ p(s()), as per:
x = f()),
Identifiability of f would then mean that we can in theory find the true f, and subsequently the true data generating components. Unfortunately, without some additional structure this model is unidentifiable, as shown by Hyvärinen and Pajunen (1999): there an infinite number of possible solutions and these have no trivial relation with each other. To solve this problem, previous work (Sprekeler et al., 2014; Hyvärinen and Morioka, 2016, 2017) developed models with temporal structure. Such time series models were generalized and expressed in succinct way by Hyvärinen et al. (2019); Khemakhem et al. (2020a) by assuming the independent components are conditionally (f)
independent upon some observed auxiliary variable p(s:/u)) = I[ p(s; |ue) In time series
context, the auxiliary variable might be history, e.g. Uy = X;-1, or the index of a time segment to model nonstationarity (or piece-wise stationarity). (It could also be data from another modality, such as audio data used to condition video data (Arandjelovic and Zisserman, 2017).)
Notice that the mixing function in (2) is assumed bijective and thus identifiable dimension reduction not possible most of the models discussed above. The only exceptions, we are aware of, are Khemakhem et al. (2020a): Klindt et al. (2021) who choose f as injective rather than bijective. Further, Khemakhem al. (2020a) assume additive noise on the observations x f(s) + E, which allows to estimate posterior of s by an identifiable VAE (iVAE). We will take a similar strategy in what follows.
Definition of Structured Nonlinear ICA
In this section, we first present the new framework of Structured Nonlinear ICA (SNICA) broad class of models for identifiable disentanglement and learning of independent components when data has structural dependencies. Next, we give an example of particularly useful specific model that fits within our framework, called A-SNICA, by using switching linear dynamical latent processes.
3.1 Structured Nonlinear ICA framework
Consider observations (x: )ee = ((..)) (M) )eer where T a discrete indexing set of arbitrary dimension. For discrete time-series models, like previous works, T would be subset of N. Crucially, however, we allow it to be any arbitrary indexing variable that describes a desired structure. For instance, T could be subset of 22 for spatial data.
We assume the data generated according the following nonlinear ICA model. First, there exist latent components s(t) = (s JieT for i E [1....,N] N where for any t, E T. the distributions of ) and (s )IEEEN are the same, which a weak form of stationarity. Second, we assume that for any m E N' and (t1,..., ,tm) TTM,, p(s),, components are unconditionally independent. We further assume that the nonlinear mixing function f: R. + R with M > N is injective, so there may be more observed variables than components. Finally, denote observational noise by E, E RBM and assume that they are i.i.d. for all E and independent of the signals ( Putting these together, we assume the mixing model where for each t T,
11 i= p(s ..... s(0) ): that is, the
x = f(::) Ex,
S1 (N)) Importantly, e, can have any arbitrary unknown distribution, even with dependent entries; in fact, may even not have finite moments.
where s; = (st
The main appeal of this framework is that, under the conditions given next section, we can now guarantee identifiability for very broad and rich class of models.
First, notice that all previous Nonlinear ICA time-series models can be reformulated and often improved upon when viewed through this new unifying framework. In other words, create
3

---

1610N
IGiON
16igN
(a) HMM
modulated
(b) Temporal dependen- cies c.f. (Hyvärinen and Morioka, 2017)
(c) New: Spatial process graph (with latent states de integrated out)
(d) New: A-SNICA linear switching dy- namics model compo- nents
components c.f. (Hälvä and Hyvärinen, 2020)
Figure I: Graphical models for the SNICA framework
models that very much like those previous works, and capture their dependeney profiles, but with the changes that by assuming unconditional independence and output noise we now allow them perform dimension reduction (this does also require some additional assumptions needed in identifiability theorems below). To see this, consider the model in Hälvä and Hyvärinen (2020) which captures nonstationarity in the independent components through global hidden Markov chain. We can transform this model into the SNICA framework if we instead model each independent component its own HMM (Figure la), with the added benefit that we now have marginally independent components and are able to perform dimensionality reduction into low dimensional an autoregressive model,
latent components. Nonlinear ICA with dime-dependencies, such proposed by Hyvärinen and Morioka (2017) is also special case of our framework (Figure lb), but again with the extension of dimensionality reduction. Furthermore, this framework allows for plethora of new Nonlinear ICA models to be developed. As described above, these do not have to limited time-series but could for instance be a process on two-dimensional graph with appropriate (in) dinddeeenesncss (see Figure Ic). However, we now proceed to introduce a particularly useful time-series model using our framework.
3.2 A-SNICA Nonlinear ICA with switching linear dynamical systems
While the above framework has great generality, any practical application will need a specific model. Next we propose one which combines the following properties of previous nonlinear ICA models into single model: ability to account for both nonstationarity and autocorrelation setting, to perform dimensionality reduction and model hidden states. Real world processes, such as video/audio data, financial time-series, and brain signals, exhibit these properties disentangling latent features in such data would hence be very useful.
fully unsupervised
Our new model is depicted in Figure 1d. The independent components generated by a Switching Linear Dynamical System (SLDS) (Ackerson and Fu, 1968; Chang and Athans, 1978; Hamilton, 1990; Ghahramani and Hinton, 2000) with additional latent variables to express rich dynamics. Formally, for each independent component {1,...,A N). consider the following SLDS over some latent vector y (i).
B
((()
e())
where u, := ur a state a first-order hidden Markov chain (u, ( I Crucially, we assume that (6)
the independent components at each time-point are the first elements V of y (w(.) (ui.....,,
..,a) (.
........... Le. s (i) ! The rest of the elements in y (i) latent variables modelling hidden dynamics. The great utility using such a higher-dimensionall latent variable is that this model allows us, for example, special case, to consider higher-order ARMA processes, thus modelling each s(0) as switching between ARMA processes an order determined by the dimensionality of yt. We call the ensuing model A ANSCICA "olta-SNICAA,, with delta in "dynamic").

---

4 Identifiability
In this section, we present two very general identifiability theorems for SNICA. We basically decouple the problem into two parts. First, we consider identifying the noise-free distribution of f(s;) from noisy data. Theorem states conditions-on tail behaviour, non-degeneracy, and non-Gausiianity- under which it is possible to recover the distribution of a process based on noisy data with unknown noise distribution. Second, we consider demixing of the nonlinearly mixed data. Theorem 2 provides general conditions on temporal or spatial dependencies, and non Gaussianity that allow recovery of the mixing function f when there no more noise. We then consider application of these theorems to SNICA.
4.1 Identifiability with unknown noise distribution
Consider the model
X, Z, + Ee,
(5)
where (Z)) a family of random variables in R such that all z₁, E T, have the same marginal
distribution, and (E))TET is family of independent (over t) and identically distributed random variables, independent of (2, keT) Let P be the common distribution of each E,, for & E T. Let 4 and (2 in T, and consider the following assumptions.
(A1) [Tail behaviour] For some p < 3, there exist A and such that for all E RN,,
Ejexp((X (A, 2,, < Aexp(BI)lI))
(A2) [Non-degeneracy] For any E CM, E/exp (n. 2t z,, is not the null random variable.
(A3) Non-Gaussianity] The following assertion is false: there exist vector 1) E R and independent random variables E and u, such that u is a non dirac Gaussian random variable and (1, zr,) has the same distribution as ã + u.
We defer the detailed discussion on the practical meaning of the assumptions (Al-A3 in the context of SNICA to Section 4.3. We next present Theorem I which establishes identifiability under unknown noise (its proof is postponed to Section A. in the Supplementary Material):
Theorem Assume that assumptions (A7), (A2) and (A3) hold for some (21,12) E T Then, up to translation, for all m 2 2, for all (la...., bm) E Tm-2, the application that associates the distribution of (Zers.... Ztm.) and P to the distribution of (Xt.....,X. Xt... is one-to-one.
Here, up to translation means that adding constant vector to all ét, and substracting this constant to all Z2, E (41,..., J. J. does not change the distribution of (xty).... Xt,, ). The proof of Theorem 1 extends that of Theorem in (Gassiat et al., 2020b), see also (Gassiat et al., 2020a), which assumed sub-Gaussian noise-free data. Our extension allows the moise-free data to have heavier tails, which is important since moose--free) data in many real-world applications is super-Gaussian i.c. heavy- as is well-known in work on linear ICA (Hyvärinen et al., 2001).
Importantly, there is no assumption on the unknown noise distribution in Theorem 1. In fact, it does not even assume mixing in ICA, and thus extends greatly outside of the framework of this paper.
4.2 Identifiability of the mixing function
Based on Theorem 1, possible to recover the distribution of the noise-free data in SNICA in (3) by setting 2, f(::). Next, we consider under which conditions the mixing function f is identifiable. Denote by S S
x S(N) the support of the distribution of all s,. We consider the situation where each (()) C R. 1 < i N, is connected, so that each S(1) is an interval. We assume moreover C2 diffeomorphism between S and C² differentiable manifold M c R Formally, this means that there exists an atlas {4a U + R Jaee of M such that for all 0, 0' E O, the map Pa op. is a C² map, and f is bijection RN + M such that for all / E O, sPa and f-1 o up -1 have continuous second derivatives. The sets Ua. 0 E O cover M and are open in M. The proof of Theorem 2 is postponed to Section A.2 in the Supplementary Material.
that the injective mixing function
5

---

Theorem 2 Assume that there exist m > 2 and (t))...,tmm T such that the vector .......8
(8) (oh) ...., ,im)) has a density pmm which is C² on (S Assume moreover that there exist
(i) (k,1) E (l,...,m) mm} with lo + such that the following assumptions hold with Q log Pm
(BI) (Uniform (k, dependdeen)) For all i (N....,N,, the set of zeros of is a meagre subset of (5())m, iee. it contains no open subset.
o² deig B01
(B2) (Local (k, I)-non quasi Gaussianity). For any open subset A STh, there exists at most one i E .1,.... N such that there exists function o Rm-1 _ R and constant c E IR such that for all E A,
8²
(1) Q = ca(st) : (t)
( Da(sn : 1
(6)
Üs 0s
where (2)
( without the coordinates ty and tq.
(1) is (5
Then, f-1 can be recovered up to permutation and coordinate -iiee transformations from the distribu- tion of (f(st....... ((st..).
4.3 Applications to SNICA
In this section, we provide additional comments on the assumptions (A1-A3) and (B1-B2) and their verification in the context of SNICA.
Assumption (A1) is condition on the tails of the noise-free data: it allows tails that are somewhat heavier than Gaussian tails. It is in fact equivalent to assuming that for some P > 3/2, there exists A' B' 0 such that for all t 0, P( t) A'
Assumption (A2) is a non-degeneracy condition likely to be fulfilled for any randomly chosen SNICA model parameters. As example, consider model such as Fig. Ic, where there exist hidden variables (Ur)eeT taking values in a finite set {l..... .....,,K K) } such that the pairs of variables (Zr, uz) have the same distribution for all T, and such that conditioned on (We)Ee, the variables (Ze)xeT are independent and the distribution of 2, only depends on Uy. (As a special case, this model includes the temporal HMM setting described in Fig. 1a.) Let (#1, t2) E T For all u,
(l..... K}. let T (u) = Pus, (u) be the mass function of ue, Q(u, v) = Iui,, (u(u) be the transition matrix from Me, to ueg- and (z) = Pa,, lov, (z/u) be the density of 2t, conditionally to "4, = u. By assumption, it is also the density of zt, conditionally to Ut, = Theorem 3 provides sufficient conditions for assumption (A2) to hold:
Theorem Assume that Q has full rank, min, x (u) > 0 and the (Tu) are linearty indepen- deni, then (A2) is satisfied as soon as the functions (1) + J exp((1) (2)))))) L do not have simultaneous zeros.
Besides the non-simultaneous zeros assumption, the assumptions of Theorem 3 are reminiscent of those used for the identifiability of non-parametrie hidden Markov models, see for instance Gassiat al. (2016); Lehéricy (2019). The key element is that 2,, and Zı, are not independent. Thus, that (A2) holds if the π and the y are not degenerate (in the precise sense given by Theorem 3), for the latent state models in Figs. la, lc.nohhrr situation where (A2) holds is when 21, is complete statistic (Lehmann and Casella, 2006) in the statistical model (F2.. |z., (-1z,,) Iz., where P2,,, ja., (-12e,) is the distribution of Zc, conditionally to Zz,- Consider the two following examples where this holds: I) When the model (Pr.,Ime, (-Iza,) ) )2., is exponential family. In this situation, complete statistics known. 2) Autoregressive models with additive innovation of the form 21, h(2,, ) Vt, for some bijective function when the additive noise Vt, is complete statistic in the statistical model [IPv.,,te,, ln., (-Iz,,)) 2z,, (note that Vr, cannot be independent of z:, here). The case in Fig, lb typically covered by this example.
Assumption (A3) states that no direction of the noise free data has non Dirae Gaussian variable component. It holds as soon as f(st) and the range of f is such that its orthogonal projection on

---

any line is not the full line. This assumption holds for instance in the following cases: 1) The range of f is compact, or 2) the range of is contained in half-cylinder, that is, there exists hyperplane such that the range of f is only on one side of this hyperplane and the projection of the range of f on this hyperplane is bounded.
Assumption (B1) and Assumption (B2) are similar to those in (Hyvärinen and Morioka, 2017;
Oberhauser and Schell, 2021) in the special case of time-series, i.e. T = N. (B1) then entails that there must be sufficiently strong statistical dependence between nearby time points. (B2) is a condition which excludes Gaussian processes and processes which can be trivially transformed be Gaussian. (For treatment of the Gaussian case, see Appendix B in Supplementary Material.) We can further provide a simple and equivalent formulation when the independent components s(9) follow independent and stationary HMMs with two hidden states, which is special case of SNICA. Denote by Yo
(i) and 7i
(6) the densities of s; conditionally to {uz ()) = 03 and {ul = 1} respectively.
Theorem 4 Assume that the stationary distribution π of the hidden chain is such that 0 < ( < 1 and that its transition matrix is invertible. Then (B1) and (B2) are satisfied with m 2 if and only if on any open interval, %o and 7) are not proportional.
Thus, a very simple HMM leads to these conditions being verified. Hyvärinen and Morioka (2017) already showed that the conditions (B1) and (B2) also hold in the case of non-Gaussian autoregressive models. Thus, we see that our identifiability theory applies both in the case HMM's (Fig la) and autoregressive models (Fig lb), the two principal kinds of temporal structure proposed in previous work, while extending them to further cases and combinations such as in Fig Ic,ld.
A simplification of (B1,B2) It is also possible to combine the assumptions (B1) and (B2) in one, while slightly weakening the generality. The key is to notice that (6) in (B2) implies the derivative in (B1) is zero, by setting с 0. But there is still the difference that (B2) considers all but one index while (BI) I) considers all indices i. If we simply assume (6) does not hold for any i, we can replace (B1) and (B2) by the new condition:
(B') For any open subset A C S and for any E (1,..., N). a function a: m=- I and a constant e E R do not exist such that (6) would hold for all s E A.
Note that Hyvärinen and Morioka (2017) defined uniform dependency and Saann--uasi--aaaan as two separate properties, but in fact their assumption of con--uasi-Gaussiaan ours: it did not consider all open subsets separately, which is why this simplification was not possible for them. We believe their definition of non-quasi- Gaussianity was in fact not quite sufficient to prove their theorem, and our stronger version may be needed, in line with Oberhauser and Schell (2021).
was weaker than
5 Experiments
Estimation method One challenge is that it is not practically possible to learn A-SNICA by exact maximun-likelihood methods. Instead, we perform learning and inference using Structured VABS (Johnson al., 2016) the current state-of-art in variational inference for structured models.
Specifically, this consists of assuming that the latent posterior factorizes as per 4(y (1=N) .uirr ((=N)) =
I g(v12 11r))((())))
(8) which allows us to optimize the resulting evidence lower bound (ELBO):
log 2 = F.g
Σ log ((x, s
(
+
Σ
KL.
q ((u p((d))))))) + H ((5)))))
T
+ E.g
logp(s (1)
E.g log p(s) (8)
1111 u (3)
Since all the distributions are in conjugate exponential families (encoder neural network is used to approximate the natural parameters of the nonlinear likelihood term) efficient message passing can be used for inference, and the mixing function is learned as decoder neural network. Even though this method lacks consistency guarantees (but see Wang and Blei (2018)), we find that our model performs very well. A more detailed treatment of estimation and inference of A-SNICA is given in Appendix C. Our code will be openly available at https://github.co http://o.ooooooaaoooaee
7

---

(a) Identifiability experiment
(b) Denoising experiment
Mean Correlation Coefficient cefffiiennt
1.0
1.00
0.8
0.95
0.6
0.90
0.4
0.2
1050 1len Me322
16-19M (M-4 M-241
0.85
LOSSM
0.0
0.80
L=1
L=2
L=3
L=5
L=1
L=2
L=3
L=5
Number of mixing layers
Number of mixing layers
Figure 2: (a) Mean absolute correlation coefficients between ground-truth independent components and their estimates by A A-SNICA, IIA-HMM, LGSSM and iVAE", with different orders of complexity (number of layers) and two different dimensions of observed (12, 24) and latent (6, 12) data. (b) Mean absolute correlation coefficient between estimated noise free data and ground-truth noise free data for same of models except IIA-HMM. Please note the difference in y-axis scales.
5.1 Experiments on simulated data
The identifiability theorems stated above hold in the limit of infinite data. Additionally, a consistent estimator would required to learn the ground-truth components. the real world, we are limited by data and estimation methods and hence it is unclear as to what extent we are actually able to estimate identifiable components - and whether identifiability reflects in better performance in real world tasks. To explore this, first performed experiments on simulated data. We compared the performance of our model to the current state-of-the-art, IIA-HMM (Morioka et al., 2021), as well as identifiable VAE (iVAE) (Khemakhem al., 2020a) and standard linear Gaussian state-space model (LGSSM). Since iVAE is not able to handle latent auxiliary variables, we allow it to "cheat" by giving it access to the true data generating latent-state, thereby creating presumably challenging baseline (denoted iVAE in our figures). LGSSM was included as a naive baseline which is only able estimate linear mixing function.
Investigating identifiability and consistency We simulated 100K long time-sequences from the A- SNICA model and computed the mean absolute correlation coefficient (MCC) between the estimated latent components and ground truth independent components (see Supplementary material for further implementation details). More precisely, illustrate the dimensionality reduction capabilities we considered two settings where the observed data dimension M, was either 12 or 24 and the number of independent components, N 3 and 6, respectively. Since IIA-HMM is unable to do dimensionality reduction, we used PCA to get the data dimension to match that of the latent states. We considered four levels of mixing of increasing complexity by randomly initialized MLPs of the following number of layers: 1 (linear ICA), 2, 3, and 5. The results in Figure 2a) illustrate the clearly superior performance of our model. The especially poor performance of IIA-HMM maybe explained by lack of noise model, much simpler latent dynamics, and lost information due to PCA pre-pro pre-processing. See Appendix D for further discussion and training details.
Application to denoising A-SNICA is able to denoise time-series signals by learning the generative model and then performing inference on latent variables. Specifically, SVAE learns the encoder network which is used to perform inference on the posterior of the independent components. We illustrate this using the same settings as above, with the exception that we now use our learned encoder and inference to get the posterior means of the independent components and input these in to the estimated decoder to get predicted noise-free observations, denoted f(s.)
measured the correlation between f(s:) and the ground-truth f(s:). Note that IIA-HMM, is not able to perform this task. The results in Figure 2b) show that the other models, designed to handle denoising, perform well at this task, as would be expected identifiability of the latent state is not necessary for good denoising performance. For LGSSM, denoising is done with the Kalman Smoother algorithm.
8

---

5.2 Experiments on real MEG data
To demonstrate real-data applicabilitty A-SNICA was applied to multivariate time series of electrical activity in the human brain, measured by magnetuencephallgraphy (MEG). Recently, many studies have demonstrated the existence of fast transient networks measured by MEG in the resting state and the dynamic switching between different brain networks (Baker et al., 2014; Vidaurre et al., 2017). Additionally, such MEG data is high-ding nigh-dimensional and very noisy. Thus this data provides an excellent target for ANSNIC to disentangle the underlying low-dimensional components.
Data and Preprocessing We considered a resting state MEG sessions from the Cam-CAN dataset. During the resting state recording. subjects sat still with their eyes closed. In the task-session data, the subjects carried out a (passive) vudio-visual task including visual stimuli and auditory stimuli. We exclusively used the resting-sesse session data for the training of the network, and task-sessing data was only used in the evaluation. The modality of the sensory stimulation provided a class label that we used in the evaluation, giving in total two classes. We band- pass filtered the data between 4 Hz and 30 Hz (see Supplementary Material for the details of data and settings).
Methods The resting-state data from all subjects were temporally concatenated and used for training. The number of layers of the decoder and encoder were equal and took values 2, 3, 4. We fixed the number of independent components 5 so that our result can be fairly compared those in Morioka et al. (2021). To evaluate the obtained features, we performed classification of the sensory stimulation categories by applying feature extractors trained with (unlabeled) resting-state data to (labeled) task-session data. Classification was performed using a linear support vector machine (SVM) classifier trained on the stimulation modality labels and arddingwwindwwwverreee obtained for each trial. The performance was evaluated by the generalizability of classifier across subjects. i.e., mue-subject-ot
features
validatoi........ For comparison, we evaluated the baseline methods: IIA-HMM and IIA-TCL (Morioka et al., 2021), We also visualized the spatial activity patterns obtained by A A-SNICA, using the weight vectors from encoder neural network across each layer.
Results Figure 3 a) shows the classification accuracies of the stimulus categories, across different methods and the number of layers for each model. The performances by Á-SNICA were consistently higher than those by the other (baseline) methods, which indicates the importance of the modeling of the MEG signals by A-SNICA. Figure 3 b) shows an example of spatial patterns from the encoder network learned by the A-SNICA. We used the visualization method presented in (Hyvärinen and Morioka, 2016). We manually picked one out of the hidden nodes from the third layer in encoder network, and plotted its weighted-averagee sensor signals, We also visualized the most strongly contributing second- and first-layer nodes. We see progressive pooling of LI units to form left lateral frontal, right lateral frontal and parietal patterns in L.2 which are then all pooled together in L3 resulting in lateral frontoparietal pattern. Most of the spatial patterns in the third layer (not shown) actually similar to those previously reported using MEG (Brookes et al., 2011). Appendix E provides more detail to the interpretation of the ASSNICA results.
6
Related work
Previous works on nonlinear ICA have exploited autocorrelations (Hyvärinen and Morioka, 2017; Oberhauser and Schell, 2021) and nonstationarities (Hyvärinen and Morioka, 2016; Hälvä and Hyvärinen, 2020) for identifiability. The SNICA setting provides a unifying framework which allows for both types of temporal dependencies, and further, extends identifiability to other temporal structures as well as any arbitrary higher order data structures which has not previously been considered in the context of nonlinear ICA. Another major theoretical contribution here is to show that identifiability with noise of unknown, arbitrary distribution, while previous work on noisy nonlinear ICA assumed noise of known distribution and known variance (Khemakhem et al., 2020a).
Importantly, the SNICA framework is fully probabilistic and thus accommodates higher order latent variables, leading to "purely unsupervised" learning. This is in large contrast to previous research which have been developed for the case where we are able to observe some additional auxiliary variable, such as audio signals accompanying video (Hyvärinen et al., 2019; Khemakhem et al., 2020a,b), or heuristically define the auxiliary variable based on time structure (Hyvärinen and Morioka, 2016). In practice this means that we are able to estimate our models using (variational)

---

b
Clssiiitcaion accursyy
L3
69
L2
8
L1
SNICA
[A-HMM
IA-TCL
Figure 3: A-SNICA on MEG data. (a) Classification accuracies of linear SVMs trained with auditory- visual data to predict stimulus category, with feature extractors trained by A-SNICA in advance with resting-state data. Each point represents a testing accuracy on a target subject (chance level: 50%). Horizontal dotted line is PCA-only baseline. (b) Example of spatial patterns of the components learned by A-SNICA (L=3). Each topography corresponds to one spatial pattern. L3: approximate total spatial pattern of one third-layer unit. L2: the patterns of the three second- units maximally contributing to this L3 unit. L1: for each 1.2 unit, the two most strongly contributing first-layer units.
MLE, which is more principled than the heuristic self- supervueee methods in most earlier papers. The only existing frameworks allowing MLE (Hälvä and Hyvärinen, 2020; Khemakhem et al., 2020a) used model restricted to exponential families, and had either no HMM or a very simple one.
The switching linear dynamical model, A-SNICA in Section 3.2, shows the above benefits in the form of a single model. That is, unlike previous nonlinear ICA models, it combines: temporal dependencies and "non-stationarity" (or HMM) in a single model 2) dimensionality reduction within rigorous maximum likelihood learning and inference framework, and 3) a separate observation equation with general observational noise. This results in a very rich, realistic, and principled model for time series.
Very recently, Morioka et al. (2021) proposed a related model by considering innovations of time series to be nonstationary. However, their model is noise-free, restricted to exponential families of at least order two, and not applicable to the spatial case, thus making our identifiability results significantly stronger. From more practical viewpoint, their model suffers from the fact that it either does not allow for dimensionality reduction (if an HMM is used) or requires a manual segmentation (if HMM is not used). Nor does it have clear distinction into state dynamics equation and measurement equation which allows for cleaning or denoising of the data.
Limitations Our identifiability theory makes some restrictive assumptions, and it remains to be seen if they could be lifted in future work. In particular, the data is nor allowed to have too heavy tails; the noise must be additive, and independent of the signal; and the practical interpretation of some of the assumptions, such as (A3) is difficult. It is also difficult to say whether our assumption of unconditionally independent components is realistic in practice. Regarding practical applications, our specific model only scratches the surface of what is possible in this framework. In particular, we did not develop a model with spatial distributions, nor did we model non- non-Gaussian observational noise our main aim was to lay the foundations for the relevant identification theory. Future work should aim make the estimation more efficient computationally; this ubiquitous problem in deep learning. but specific solutions for this concrete problem may be achievable (Gresele et al., 2020).
7 Conclusion
We proposed a new general framework for identifiable disentanglement, based on nonlinear ICA with very general temporal dynamics spatial structure. Observational noise of arbitrary unknown distribution is further included. We prove identifiability of the models in this framework with high generality and mathematical rigour. For real data analysis, the properties of all existing time series models in nonlinear ICA, while generalizing them in many ways (see Section 6 for details). We hope this work will contribute to wide-spread application of identifiable methods for disentanglement in a highly principled, probabilistic framework.
propose special case which subsumes
10


Reviews:
[{"review_id": "qQYt5iJ33uA", "reviewer": "Reviewer_yFEi", "summary": "The authors present a new disentanglement framework for structured data. They claim that the method generalizes to scenarios with spatial dependencies, independent additive noise, and non-square mixing processes. They demonstrate the utility of the framework by applying an approximate model to toy data as well as MEG data.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "*Review summary:*\n\nAs far as I know, SNICA is indeed a novel method for nonlinear ICA. The combined departure from typically requiring the latent and data dimensionality to be equal and the explicit noise term in equation 3 allow for a broader set of application spaces. My primary complaint is that the paper makes a number of strong claims about the novelty and superiority of the framework with insufficient evidence to support them. I would request that the authors put more effort into supporting their assertions, or dial them back, before the paper is ready for publication.\n\n*Strengths:*\n\nOverall the paper is well written. The mathematics in the main manuscript are thorough, complete, and easy enough to follow (I only skimmed the proofs in the appendix). The overall objective to push identifiable nonlinear ICA towards more practical regimes of noisy data and lower-dimensional latent structure is a valuable direction for the community. The application to real (MEG) data is also a plus, as many prior works in this space only test on simulated data.\n\n*Weaknesses:*\n\nI see two primary weaknesses in this paper: 1) numerous sweeping claims are made regarding their superiority over previously published results without sufficient support. 2) The empirical demonstration of the model compares against control models that are not well fit to the tasks. Both of these weaknesses are easily addressable, by 1) either supporting their claims more carefully or dialing them back and 2) choosing a more reasonable control or experimental task.\n\n1) The authors make a large number of bold claims, for example in lines 9-10; 44-47; 52-53; 67; 96-99; 110; 123-125; 139; 141-142; 166; 300-301; 339-341. While I do not know the literature well enough to directly refute all of them, I will note a few examples where they appear to be wrong. I believe that this weakness can be resolved by either softening the claims themselves (which, in my opinion, would not reduce the novelty or significance of the submission) or providing a more detailed review of how previous works compare.\n\n1. a) On lines 44-47 the authors note that “all models cited above, with the exception of Khemakhem et al. (2020a), assume that the data are fully observed and noise-free”. While that might be true, I immediately thought of the work of Locatello et al. {1}, which was not cited by the authors, but is an identifiable disentanglement method and comments on their ability to handle noise: “The generative model does not model additive noise; we assume that the noise is explicitly modeled as a latent variable and its effect is manifested through [the ground truth generator], as is done by [several citations].” (last paragraph of section 3) I admit that this might not necessarily be a contradictory claim, but I nonetheless would appreciate it if the authors clarified how their method of explicitly modeling additive noise is beneficial over the type of setup in {1} and the citations in Locatello et al.’s quoted sentence.\n\n1.b) On lines 96-97 the authors note that “the mixing function f is assumed bijective and thus dimension reduction is not possible in most of the above models. The only exception is Khemakhem et al. (2020a) who…” I find the second part of this quote (“dimension reduction is not possible”) surprising in general, considering that any publication attempting to perform disentanglement on the DisentanglementLib dataset necessarily has to assume dimensionality reduction. I thought maybe the authors intended to say that the theory does not allow dimensionality reduction, even if the practical implementations do. However, Klindt et al., which is cited earlier, presents an identifiable disentanglement framework that only assumes an injective mixing function in their theory and therefore allows for dimensionality reduction.\n\n1.c) The authors claim that “under the conditions given in the next section, we can now guarantee identifiability for a very broad and rich class of models. First, notice that all previous Nonlinear ICA time-series models can be recast and often improved upon when viewed through this new unifying framework.” I do not understand how this could possibly be true given that not all previous Nonlinear ICA work abides by the conditions given in sections 3 and 4. Just as two examples, the requirement for unconditional independence on lines 114-115, and tail behavior in assumption A1 are not ubiquitous in the literature. I could definitely be misunderstanding what the authors intended with their statement, in which case I would be happy with a clarification.\n\nI understand that it is not reasonable for the authors to precisely state how every previous contribution fails to meet their claimed advantages. However, given the examples above, I found myself unconvinced in general. I further found myself wondering if it was necessary to make so many sweeping claims in the first place.\n\n2) The authors set up a simulated example problem in section 5.1 to understand how their model works with a restricted experiment. I agree that this is an important step for understanding how the model behaves. They choose to compare against what they claim to be the state of the art, IIA-HMM. However, they state themselves that “IIA-HMM has a much simpler model of dynamics and no noise model, and likely lost information due to PCA pre-processing.” This leaves me wondering why they felt that this was a fair comparison, given that the problem setup is not at all matched to what IIA-HMM was designed to solve. I would be curious to see how they do for a non-dimensionality-reduced setup, since as far as I know their framework does not require the number of generating latents to be fewer than the data dimensionality. Or, a better solution would be for them to compare their model against an alternative that is appropriately matched to the dimensionality reduction task. Furthermore, the authors do not provide a baseline comparison for their denoising task. This might be because the authors wished to focus on identifiable models, which restricts the otherwise large set of denoising methods. However, again given the work cited in {1}, I am unconvinced that there was not a suitable comparison to be made. Without appropriate comparisons, we are left to rely solely on the scalar MCC metric in an unfamiliar simulated example, which I find insufficient.\n\n*Additional minor complaints:*\n1) The citations need to be revised and edited. Many are listed as arxiv prints that are now published in peer-reviewed venues. A few are missing the venue entirely. Morioka 2020b appears to be listed twice.\n\n2) I noticed typos on line 243 (“a complete statistics”), fig 2 caption (“ground true independent”)\n\n3) I think it would be helpful for practitioners if the authors included some justification for their decision on line 318, or at least a discussion on the tradeoffs for the number of independent components chosen.\n\n4) I would appreciate it if the authors spent more time discussing trade-offs made in their framework. As of now it is limited to the first two sentences of the “Limitations” section starting on line 367. I found this to be rather terse given the space allocated for stating the failures of prior work. For example, if I understand the unconditional independence assumption on lines 114-115 correctly, then it seems highly unlikely to be met in real-world data, including their MEG experiment. The constraint on the distribution tails and 2nd order nature of the generator also seem restricting with respect to real data. Perhaps the authors could note alternative work that does not require such assumptions for identifiability (in exchange for other restrictions), which would assist future researchers seeking a more general solution.\n\n{1} http://proceedings.mlr.press/v119/locatello20a.html", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, in the checklist response", "review_text": "", "extra_scores": {"ethical_concerns": "No", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "bjjQICi4grR", "reviewer": "Reviewer_x1ZH", "summary": "This paper proposed a new general identifiable nonlinear ICA framework for structured data. They proved identifiability for this general framework. They also proposed a specific application model from the framework and applied it to the simulated and real data.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: Yes, the framework they proposed is more general than the previous papers. The related work is properly cited.\n\nQuality: The theory part seems technically sound. I have some questions about the experimental part. \n1. The simulated data is from the true SNICA model. The bad performance of IIA-HMM can be due to the model mismatch or the loss of information from the PCA pre-processing. Have you tried eliminating those effects in the comparison?\n2. How do you select the number of independent components to be 5 in the real data application?\n3. In real data, PCA was performed before applying IIA-HMM to the data, while SNICA directly applying to the data. PCA can loss a lot information. Are the comparison still fair? What's the classification accuracy if directly using the features extracted by a 5d PCA to predict the labels?\n4. What do the inferred discrete states, continuous states, and transition matrices look like in real data? Do they have any interpretable meanings?\n\nClarity: Yes, the paper is well written. But I do want clarification for: in line 343-345, you claimed that SNICA has identifiability with noise of unknown, arbitrary distribution, while (Khemakhem et al., 2020a) assumed noise of known distribution. If one specifies a wrong noise distribution in inference, can SNICA still recover the true latents? According to Theorem 1 in Khemakhem et al., 2020a, they only have assumptions on the characteristic function of the noise distribution. Can you clarify what do you mean by saying noise of known distribution?\n\nSignificance: Yes, the results are important. The framework is general, and many models can be builded upon it.\n\nOverall, I think this paper is theoretically sound, yet the experiment can be further improved. The framework is general, which leaves room for possible future works. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have discussed the limitations in detail in the paper.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "Wpq4etukzaL", "reviewer": "Reviewer_8dTg", "summary": "In this work, the authors present an identifiability framework called Structured Nonlinear Independent Component Analysis (SNICA), which extends previous identifiability theory for time-series models into a much broader class of models.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The manuscript is very well written and structured, and is accessible to readers even if they cannot follow all the technical details. The work aims to contribute to the growing area of identifiability models, a field that is starting to gain traction as a basis for a more rigorous understanding of representation learning with nonlinear parametrised models (such as neural networks).\n\nOne confusion I had when reading the manuscript was a seeming contradiction between the definition of structured nonlinear ICA and the nonstationary models presented afterwards. In particular, line 114 assumes that the latent components are independently distributed over time, i.e. knowing the latents at time t yield no information about the latents at time t + 1. Yet this assumption is then being made in equation (4), and is crucial to cover previous identifiable models such as TCL. I'd appreciate if the authors could clarify this point. I am happy to raise my score if this point can be clarified.\n\nSecond, it would be good to give some more intuition on how the two-step procedure described in section 4 (first remove noise, than identify latents) is converted into a practical algorithm. Clearly, for each individual observable x_t one cannot remove the noise, Theorem 1 just shows that the distribution can be uncovered. Is this a step that is explicitly done during model training, or is implicit?\n\nOne crucial aspect currently missing in the manuscript is a clear exposition for how the training/estimation actually works. There is only a brief paragraph at the beginning of section 5 stating that \"by framing the model within conjugate exponential families we are able to perform learning and inference using Structured VAEs\". Currently, this reviewer feels there is no way a reader would be able to replicate the experiments without the code itself. While I appreciate that the authors promised to release the code, I feel the manuscript should contain a clear exposition for the complete training protocol such that replication would be possible without looking at the code.\n\n## Rebuttal\nThe rebuttal clarified my concerns and questions. I am raising my score by one point.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"ethical_concerns": "None.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "Mah30q9XuGW", "reviewer": "Reviewer_9Cro", "summary": "The authors provide an identifiability result for non-linear ICA for a class of time series models. There is a clear contribution with respect to previous work that was typically restricted to condionnally independent sources, conditionned on a hidden markov chain in finite stare spaces. This new result allows identifying independent sources with a specific form of time dependency.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The result seem to build on a recent identifiability result by Gassiat et al. for Markov chains in general state space, which one could have expected to open the way to new ICA identifiability, in line with what has happened in the past for HMMs with finite state space. However, the authors also provide a slightly more general result in theorem 1 (allowing supergaussian source distributions) before exploiting it for ICA identifiability. The result is restricted to a specific source model (Markov switching linear model) but one could argue that it is a good step forward for addressing more general cases.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations are adequately addressed.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3 hours"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper extends the literature on identifiable nonlinear ICA for time series in ways that all reviewers found satisfactory. There is a clear consensus that the work is novel and interesting, and many of the reviewers were convinced by the authors' rebuttal to increase their score even further. Therefore I recommend acceptance.
Decision: Accept (Poster)