Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper presents new identifiability result for nonlinear ICA that gets around the need for auxiliary information by leveraging a sparsity assumption on the mixing function. While sparsity has been explored in the nonlinear ICA community, the existing results have focused on the relationship between source distributions (Lachapelle et al., 2022), rather than on sparsity of the mixing function. On exception is Gresele et al [2021] that did show a different sparsity assumption rules out some of the standard counterexamples, but didn't provide a full identification proof. This paper does that and shows analogous results for the linear Gaussian case which well-known to not be identifiable without additional assumptions (usually non-Gaussian sources). The paper also gives results under an "independent influences" assumption which is more similar to Gresele et al [2021] and show identification results there too. Finally, the theory results are supported with some simple experiments.
strengths_and_weaknesses: Strengths
* Strong identification (up to a component-wise transform & permutation) results in a hard problem. 

Weakness
* I am always very uncomfortable with making assumptions on the mixing function $f: s \rightarrow x$, especially in the image domain. While it is relatively intuitive that there should be some simple dependencies among the source variables (as people, we wouldn't be able to understand the world if this were not the case), it's not obvious that the mixing function should have any sparsity: consider how realistic images are rendered in modern rendering engines. The most realistic looking renderers use ray-tracing which explicitly bounce light around all the objects in the scene in order to correctly model how light reflects in a scene. That seems to imply a dense dependence between sources rather than any kind of sparsity. Of course, it may be the case that structured sparsity is a reasonable approximation of the mixing function, but I would have liked to see some more discussion on when we can expect this to hold. The discussion on page 7 with reference to the independent influences assumption was far better.
* The empirical results have the expected relationships between approaches, but I was surprised that the MCC scores were only $\approx 0.8$. 

I am conflicted on this paper - while ICA is not my research area, I know that this is a very well-studied area and identification in this IID setting is hard to achieve (as the paper points out - most papers rely on auxiliary variables). So if we take this work as a pure piece of theory, the structured sparsity condition is a sufficient condition for identification is a difficult domain, and this is very interesting paper. On the other hand - the practitioner in me can barely parse the structured sparsity assumption - so I'd have no idea whether or not it applied in my domain, so I think a lot more work is needed in expanding on when we expect this assumption to hold or alternatively, are there any testable implications of this assumption? 
questions: 1. Do you know what was preventing MCC scores from getting above 0.9? These problems are relatively straightforward so I would have expected to see MCC score $\rightarrow 1$; it would be good to understand what the limitations of the empirical results are?

Minor
* Condition (iii) of theorem 1 is hard to parse without a lot of back references. Consider rewording as follows, "Denote by F the support of (all possible values of **the Jacobian of $f$**) $J_f (s)$. For all $i \in {1, . . . , n}$,. there exist **|F_{i,:} | sources**...". Similar amendments could be made to the other theorem statements too - the conditions in the theorems are already pretty dense, so it helps to remind the reader what all the variables refer to.
limitations: Yes.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper considers the identifiability of the nonlinear ICA model with unconditional priors:
$x = f(s)$
where $x$ is the observed data and $s$ are the unobserved sources.

They prove that the sources $s$ are identifiable identifiable under two different settings:

1. when a "structural sparsity" assumption holds;

2. when an "independent influence" assumption holds.

They also provide experiments where they estimate the latent sources via regularized objective functions. The regularization terms are inspired by the two settings above.

More detail: 

1. Structural sparsity: this assumption is that for every latent source $s_i$, there is a set of observed variables such that $s_i$ is the only latent source that participates in the generation of all variables in the set. 

In Theorem 1, the authors prove that under structural sparsity, the sources can be identified up to permutation and coordinate-wise transformation.  For this result, the authors also assume invertibility of $f$, that the observed Jacobian $J_f(s)$ spans the support of $J_f(s)$, and the estimated decoder is as sparse as the true decoder.

In Theorem 3, the authors again assume structural sparsity but relax the invertibility of $f$. They prove that if the estimated $\widehat{f}$ is a 'rotated-Gaussian mixture preserving automorphism' (i.e. it has a particular functional form which rotates the true decoder $f$) then the sources are identifiable.


2. Independent influence: this assumption is that the influence of each source on the observed variables is independent of each other i.e. $\partial f / \partial s_i$ is independent of the other sources and their influences in a non-statistical sense. 

In Theorem 4, the authors prove that if independent influence holds (among other assumptions) then the latent sources are identifiable.  




strengths_and_weaknesses: Strengths

- The paper is timely and relevant and seems like an important contribution to the study of identifiability

- The writing is clear

Weaknesses 

- Some definitions are unclear (see questions below).

- The paper is missing a number of references relating to sparsity and identifiability in latent variable models. 

In the linear case, Bing et al. (2020) and Rohe and Zeng (2020) prove that the assumption of sparsity in the mixing function (i.e. loadings matrix) gives identifiable solutions.

In the nonlinear case, Moran et al. (2021) prove that assuming sparsity in the decoder of a deep generative model gives identifiable solutions.

Similarly to the experiments in this paper, Rhodes and Lee (2021) propose a L1 regularization term on the Jacobian of the mixing function (without proving identifiability however). 

- Bing, Xin, et al. "Adaptive estimation in structured factor models with applications to overlapping clustering." The Annals of Statistics 48.4 (2020): 2055-2081.

- Rohe, Karl, and Muzhe Zeng. "Vintage factor analysis with varimax performs statistical inference." arXiv preprint arXiv:2004.05387 (2020).

- Moran, Gemma E., et al. "Identifiable variational autoencoders via sparse decoding." arXiv preprint arXiv:2110.10804 (2021).

- Rhodes, Travers, and Daniel Lee. "Local Disentanglement in Variational Auto-Encoders Using Jacobian $ L_1 $ Regularization." Advances in Neural Information Processing Systems 34 (2021): 22708-22719.

questions: - The assumption of independent influence is that the influence of each source on the observed variables is independent of each other i.e. $\partial f / \partial s_i$ is independent of the other sources and their influences in a non-statistical sense.

Can the authors be more precise about what "in a non statistical sense" means? From the paper, it seems that it means uncorrelatedness of the Jacobian columns? Can the authors clarify?
limitations: The authors do not elaborate on real life situations where their assumptions are likely to hold (except for the animal recording example in the independent influences section).
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper proves identifiability results for Independent Component Analysis. In contrast to other recent works which exploit assumptions on auxiliary variables, this work exploits sparsity assumptions on the mixing process.
strengths_and_weaknesses: Identifiably results for ICA is an important problem as it provides a rigorous foundation for other applied areas such as disentangled feature learning and parts of causality. As such, advances in this area are important and potentially impactful.

The paper is well written and clear. It explains the results in the context of the problem and prior work. I believe it makes a solid contribution and is worth accepting to the conference.

One thing I think could improve the paper would be to elaborate more on the assumtions being made and when they are violated. In particular:
- what would happen if the ground truth data were to violate  the assumptions but the method were run anyway? 
- thinking about the equivalence classes of indeterminable solutions without this assumption, is it possible to relate these equivalence classes to the sparse solutions? E.g. does each equivalence class contain exactly one such sparse solution? Do only some contain one sparse solution (and if so, what is special about these equivalence classes)? 



questions: See above
limitations: See above
review_title: null

Meta-review: metareview: Strong paper with all reviewers arguing for acceptance.

Only minor concerns from the reviewers were on whether the preconditions required for the theorems in the paper were likely to hold in practice. This was discussed thoughtfully by the author response, including a new appendix section.
confidence: Certain
Decision: Accept