Venue: CLeaR 2022 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: Summary: The authors present a novel approach for discovering disentangled representations in data sets with auxiliary variables.
Main review: # Review
The authors present a novel approach to discovering disentangled latent variables from observations and auxiliary data generated by a latent variable model. They present identifiability results for their approach (up to permutation), and present empirical comparisons with related work on synthetic data. Both the motivation and the presentation is very clear and easy to follow. The comparison with previous literature is sufficient (at least at the conceptual level, see below). The findings are novel and of interest to the conference audience, and overall topic of the paper is very timely.

## Questions and suggestions

Below I present some questions and suggestions to the authors:

- Possibly the only obvious shortcoming of the paper is comparison on real data with similar works, especially with iVAE. Do the authors plan to add such results?
- It would be a good idea to mention the dependency of the method on the observed auxiliary variable (e.g. time) is a necessary condition for the application of this approach in the abstract of the paper.
- Given that sparsity is a concept that can be understood differently according to the context (e.g. in compressed sensing vs. scientific computation), I think it would be a good idea to delineate what sparse means in this context early on in the paper.
- Can the regularization on G_a and G_z undermine sufficient (time or action) variability if taken to large levels? 

## Minor points
- In Thm 4, the explanation for the first assumption is given below at least partly. So in the theorem you could refer the author to the lines below, and from there to Def. 9.
- "This can be interpreted as having a sufficiently nonlinear transition model." Pg. 6. Can any linear transition model meet this criterion?
- For an even more lively exposition (e.g. in Pg. 10), demonstrating how differently results obtained by iVAE would be in a similar example (e.g. that in Fig. 1) could be beneficial.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: Summary: Interesting paper achieving disentanglement by sparsity regularisation
Main review: This paper proposes to achieve disentanglement of latent factors by simultaneously learning them with the sparse causal graph that relates them. Building on recent results in the nonlinear ICA literature, permutation-identifiability is proven under a number of assumptions. The theory makes use of the intuition that entangled representations will require a denser adjancency matrix and that inducing sparsity on this matrix will push the latent factors to disentangle.


## Strengths

The paper is written and structured clearly and easy to follow. It provides examples throughout to illustrate the different concepts, which further helps understanding. Additionally, the theory presented is well-motivated throughout.

While the experiments are limited to (rather unrealistic) small-scale simulated data, they show the improved disentanglement performance under the proposed approach, as well as the effect of violated assumptions consolidating the presented theory.


## Weaknesses

While most assumptions are well motivated, for some this motivation is missing and I would be curious to know how the authors expect these assumptions to restrict the applicability of their method to different scenarios. For example:
1) What does it mean in practice that $\mathbf{f}$ is assumed to be a diffeomorphism? 
2) What does assumption 1 in theorem 5 mean in practice?

Additionally, some symbols are not well-defined. Most importantly, $G$ is described to be causal, but no explanation is given for why that would be the case. Additionally, I find the usage of $\mathbf{T}$ slightly confusing. While it generally describes the sufficient statistic, on page 5 it is also described as an element-wise nonlinearity. On the next line, an example for the Gaussian case is given, where $\mathbf{T}(z) := z$, which is not nonlinear. Other symbols for which I would appreciate a clearer definition or separate description are $y$ and $Py$ (Definition 2), $p$ (starting in Theorem 4) and $\tau$ (section 2.4.1). 


### Disclaimer
I did not check the proofs provided in the Appendix in detail.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: Summary: This paper proposes a method for finding the true, latent factors and their causal relations by enforcing a sparsity among mechanisms. The method is build upon works around ICA and includes proofs under which settings it can correctly disentangle the factors. The paper also discusses a VAE-based approach to implement the proposed method.
Main review: The paper is very well written and easy to follow, despite several more complex statements. The initial research statement is well motivated, and the examples given throughout the paper makes it accessible, on a higher level, to a wider audience (i.e. what the paper is about in general, and what it tries to solve). The approach is novel and concerns an important problem. The idea of using the sparsity of mechanisms to find the true mechanisms seems fair, since it should intuitively hold in many real-world situations. The assumptions of the method, such as that the conditional density belongs to an exponential family, the Jacobian varying sufficiently etc., might limit it in practical applications and can make it hard to verify whether they hold in a real-world dataset or not, but are necessary considering the goal of provably finding the mechanisms. Overall, I would consider the paper as a good contribution to the ICA and causality research field.

The only major drawback of this paper is the limited experimental evaluation. The proposed method is evaluated on two synthetic datasets, one with a time-sparsity setting, and another for action-sparsity. While the proposed method outperforms the shown baselines, it is not clear how robust the method is, and whether it would work on more realistic data. For instance, the noise added to the observation $X$ was chosen to have a small standard deviation of 0.01. What if this noise is increased? How does the method handle higher noise levels? Similarly, what if the noise level is increased for the ground truth transition function (covariance is chosen to be $0.0001I$). In terms of the adjacency matrices, more diverse graphs with random connections can be tested.

#### Minor comments
* Page 5, last paragraph: typo 'which we which we'
* Appendix page 43, 4th line in B.1: double punctuation 'with negative slope of 0.2. .'
* Page 7, paragraph _Sparsity_: It is stated that the minimal graph is found that 'properly fit[s] the data'. What does 'properly' mean in this case? Is it that the data is fit up to a small error $\epsilon$ which is supposed to be lower than the regularization?
* Page 45, paragraph _Learned mechanism_: The MLP for the mean layer is said to use 5 layers with each 512 hidden units. This seems quite large compared to the ground truth functions (f being a 3 layer MLP with 20 hidden units, mean functions often sin/cos-based). Is such a large network necessary to learn the sin/cos functions in the synthetic datasets, and get stable results?
* Page 12, Figure 3: The chosen plot style is not very intuitive to understand at first. The y-axis represents a 0-1 scaling of three metrics which have different trends (SHD - lower is better, R/MCC - higher is better). The label 'diag. graph' and 'non-diag. graph' on this axis add to the confusion since it is a label for the row, not the y-axis itself. It also wasn't clear that the multiple points/crosses represent multiple runs, because several overlay each other. I would recommend splitting this plot into multiple ones, which makes it much easier to grasp for the reader.
review_title: null

Meta-review: The reviewers unanimously acknowledge the clarity of the paper and the relevance of the theoretical contribution. One weakness pointed out by reviewers is the limited experimental evaluation. Overall, this constitutes a good theoretical contribution at the intersection of causal representation learning and nonlinear ICA, which we recommend for acceptance.
Decision: Accept (Poster)