Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: The authors provide an identifiability result for non-linear ICA for a class of time series models. There is a clear contribution with respect to previous work that was typically restricted to condionnally independent sources, conditionned on a hidden markov chain in finite stare spaces. This new result allows identifying independent sources with a specific form of time dependency.
main_review: The result seem to build on a recent identifiability result by Gassiat et al. for Markov chains in general state space, which one could have expected to open the way to new ICA identifiability, in line with what has happened in the past for HMMs with finite state space. However, the authors also provide a slightly more general result in theorem 1 (allowing supergaussian source distributions) before exploiting it for ICA identifiability. The result is restricted to a specific source model (Markov switching linear model) but one could argue that it is a good step forward for addressing more general cases.
limitations_and_societal_impact: Limitations are adequately addressed.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: In this work, the authors present an identifiability framework called Structured Nonlinear Independent Component Analysis (SNICA), which extends previous identifiability theory for time-series models into a much broader class of models.
main_review: The manuscript is very well written and structured, and is accessible to readers even if they cannot follow all the technical details. The work aims to contribute to the growing area of identifiability models, a field that is starting to gain traction as a basis for a more rigorous understanding of representation learning with nonlinear parametrised models (such as neural networks).

One confusion I had when reading the manuscript was a seeming contradiction between the definition of structured nonlinear ICA and the nonstationary models presented afterwards. In particular, line 114 assumes that the latent components are independently distributed over time, i.e. knowing the latents at time t yield no information about the latents at time t + 1. Yet this assumption is then being made in equation (4), and is crucial to cover previous identifiable models such as TCL. I'd appreciate if the authors could clarify this point. I am happy to raise my score if this point can be clarified.

Second, it would be good to give some more intuition on how the two-step procedure described in section 4 (first remove noise, than identify latents) is converted into a practical algorithm. Clearly, for each individual observable x_t one cannot remove the noise, Theorem 1 just shows that the distribution can be uncovered. Is this a step that is explicitly done during model training, or is implicit?

One crucial aspect currently missing in the manuscript is a clear exposition for how the training/estimation actually works. There is only a brief paragraph at the beginning of section 5 stating that "by framing the model within conjugate exponential families we are able to perform learning and inference using Structured VAEs". Currently, this reviewer feels there is no way a reader would be able to replicate the experiments without the code itself. While I appreciate that the authors promised to release the code, I feel the manuscript should contain a clear exposition for the complete training protocol such that replication would be possible without looking at the code.

## Rebuttal
The rebuttal clarified my concerns and questions. I am raising my score by one point.
limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The authors present a new disentanglement framework for structured data. They claim that the method generalizes to scenarios with spatial dependencies, independent additive noise, and non-square mixing processes. They demonstrate the utility of the framework by applying an approximate model to toy data as well as MEG data.
main_review: *Review summary:*

As far as I know, SNICA is indeed a novel method for nonlinear ICA. The combined departure from typically requiring the latent and data dimensionality to be equal and the explicit noise term in equation 3 allow for a broader set of application spaces. My primary complaint is that the paper makes a number of strong claims about the novelty and superiority of the framework with insufficient evidence to support them. I would request that the authors put more effort into supporting their assertions, or dial them back, before the paper is ready for publication.

*Strengths:*

Overall the paper is well written. The mathematics in the main manuscript are thorough, complete, and easy enough to follow (I only skimmed the proofs in the appendix). The overall objective to push identifiable nonlinear ICA towards more practical regimes of noisy data and lower-dimensional latent structure is a valuable direction for the community. The application to real (MEG) data is also a plus, as many prior works in this space only test on simulated data.

*Weaknesses:*

I see two primary weaknesses in this paper: 1) numerous sweeping claims are made regarding their superiority over previously published results without sufficient support. 2) The empirical demonstration of the model compares against control models that are not well fit to the tasks. Both of these weaknesses are easily addressable, by 1) either supporting their claims more carefully or dialing them back and 2) choosing a more reasonable control or experimental task.

1) The authors make a large number of bold claims, for example in lines 9-10; 44-47; 52-53; 67; 96-99; 110; 123-125; 139; 141-142; 166; 300-301; 339-341. While I do not know the literature well enough to directly refute all of them, I will note a few examples where they appear to be wrong. I believe that this weakness can be resolved by either softening the claims themselves (which, in my opinion, would not reduce the novelty or significance of the submission) or providing a more detailed review of how previous works compare.

1. a) On lines 44-47 the authors note that “all models cited above, with the exception of Khemakhem et al. (2020a), assume that the data are fully observed and noise-free”. While that might be true, I immediately thought of the work of Locatello et al. {1}, which was not cited by the authors, but is an identifiable disentanglement method and comments on their ability to handle noise: “The generative model does not model additive noise; we assume that the noise is explicitly modeled as a latent variable and its effect is manifested through [the ground truth generator], as is done by [several citations].” (last paragraph of section 3) I admit that this might not necessarily be a contradictory claim, but I nonetheless would appreciate it if the authors clarified how their method of explicitly modeling additive noise is beneficial over the type of setup in {1} and the citations in Locatello et al.’s quoted sentence.

1.b) On lines 96-97 the authors note that “the mixing function f is assumed bijective and thus dimension reduction is not possible in most of the above models. The only exception is Khemakhem et al. (2020a) who…” I find the second part of this quote (“dimension reduction is not possible”) surprising in general, considering that any publication attempting to perform disentanglement on the DisentanglementLib dataset necessarily has to assume dimensionality reduction. I thought maybe the authors intended to say that the theory does not allow dimensionality reduction, even if the practical implementations do. However, Klindt et al., which is cited earlier, presents an identifiable disentanglement framework that only assumes an injective mixing function in their theory and therefore allows for dimensionality reduction.

1.c) The authors claim that “under the conditions given in the next section, we can now guarantee identifiability for a very broad and rich class of models. First, notice that all previous Nonlinear ICA time-series models can be recast and often improved upon when viewed through this new unifying framework.” I do not understand how this could possibly be true given that not all previous Nonlinear ICA work abides by the conditions given in sections 3 and 4. Just as two examples, the requirement for unconditional independence on lines 114-115, and tail behavior in assumption A1 are not ubiquitous in the literature. I could definitely be misunderstanding what the authors intended with their statement, in which case I would be happy with a clarification.

I understand that it is not reasonable for the authors to precisely state how every previous contribution fails to meet their claimed advantages. However, given the examples above, I found myself unconvinced in general. I further found myself wondering if it was necessary to make so many sweeping claims in the first place.

2) The authors set up a simulated example problem in section 5.1 to understand how their model works with a restricted experiment. I agree that this is an important step for understanding how the model behaves. They choose to compare against what they claim to be the state of the art, IIA-HMM. However, they state themselves that “IIA-HMM has a much simpler model of dynamics and no noise model, and likely lost information due to PCA pre-processing.” This leaves me wondering why they felt that this was a fair comparison, given that the problem setup is not at all matched to what IIA-HMM was designed to solve. I would be curious to see how they do for a non-dimensionality-reduced setup, since as far as I know their framework does not require the number of generating latents to be fewer than the data dimensionality. Or, a better solution would be for them to compare their model against an alternative that is appropriately matched to the dimensionality reduction task. Furthermore, the authors do not provide a baseline comparison for their denoising task. This might be because the authors wished to focus on identifiable models, which restricts the otherwise large set of denoising methods. However, again given the work cited in {1}, I am unconvinced that there was not a suitable comparison to be made. Without appropriate comparisons, we are left to rely solely on the scalar MCC metric in an unfamiliar simulated example, which I find insufficient.

*Additional minor complaints:*
1) The citations need to be revised and edited. Many are listed as arxiv prints that are now published in peer-reviewed venues. A few are missing the venue entirely. Morioka 2020b appears to be listed twice.

2) I noticed typos on line 243 (“a complete statistics”), fig 2 caption (“ground true independent”)

3) I think it would be helpful for practitioners if the authors included some justification for their decision on line 318, or at least a discussion on the tradeoffs for the number of independent components chosen.

4) I would appreciate it if the authors spent more time discussing trade-offs made in their framework. As of now it is limited to the first two sentences of the “Limitations” section starting on line 367. I found this to be rather terse given the space allocated for stating the failures of prior work. For example, if I understand the unconditional independence assumption on lines 114-115 correctly, then it seems highly unlikely to be met in real-world data, including their MEG experiment. The constraint on the distribution tails and 2nd order nature of the generator also seem restricting with respect to real data. Perhaps the authors could note alternative work that does not require such assumptions for identifiability (in exchange for other restrictions), which would assist future researchers seeking a more general solution.

{1} http://proceedings.mlr.press/v119/locatello20a.html
limitations_and_societal_impact: Yes, in the checklist response
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposed a new general identifiable nonlinear ICA framework for structured data. They proved identifiability for this general framework. They also proposed a specific application model from the framework and applied it to the simulated and real data.

main_review: Originality: Yes, the framework they proposed is more general than the previous papers. The related work is properly cited.

Quality: The theory part seems technically sound. I have some questions about the experimental part. 
1. The simulated data is from the true SNICA model. The bad performance of IIA-HMM can be due to the model mismatch or the loss of information from the PCA pre-processing. Have you tried eliminating those effects in the comparison?
2. How do you select the number of independent components to be 5 in the real data application?
3. In real data, PCA was performed before applying IIA-HMM to the data, while SNICA directly applying to the data. PCA can loss a lot information. Are the comparison still fair? What's the classification accuracy if directly using the features extracted by a 5d PCA to predict the labels?
4. What do the inferred discrete states, continuous states, and transition matrices look like in real data? Do they have any interpretable meanings?

Clarity: Yes, the paper is well written. But I do want clarification for: in line 343-345, you claimed that SNICA has identifiability with noise of unknown, arbitrary distribution, while (Khemakhem et al., 2020a) assumed noise of known distribution. If one specifies a wrong noise distribution in inference, can SNICA still recover the true latents? According to Theorem 1 in Khemakhem et al., 2020a, they only have assumptions on the characteristic function of the noise distribution. Can you clarify what do you mean by saying noise of known distribution?

Significance: Yes, the results are important. The framework is general, and many models can be builded upon it.

Overall, I think this paper is theoretically sound, yet the experiment can be further improved. The framework is general, which leaves room for possible future works. 
limitations_and_societal_impact: The authors have discussed the limitations in detail in the paper.
review_title: null

Meta-review: The paper extends the literature on identifiable nonlinear ICA for time series in ways that all reviewers found satisfactory. There is a clear consensus that the work is novel and interesting, and many of the reviewers were convinced by the authors' rebuttal to increase their score even further. Therefore I recommend acceptance.
Decision: Accept (Poster)