Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper propose two new conditions for nonlinear identifiability of temporal processes. The idea is very promising: discovery of conditional independent sources with time-delayed causal relations in between. 
The proof and the code seem good.
main_review: I have some concerns:
1. The proof of the theorems are not clear. E.g., what is the condition u in Eq. (2) and the following many equations? 
2. about the network structure: input is x_0, x_1, ...., x_T, and the latent is z_0, z_1, z_T, and the reconstruct is x_hat_0, ..., x_hat_T. The "time-delayed" is not clear in this structure. If there is time delay, how the x_0 can be reconstructed, and how is the reconstruction loss is calculated?
3. In Fig. 2, it is not clear what's the importance of each part of the model. How does the "C" part connect to the proposed two conditions? As the model seems big, what about the training stability?
4. The hyperparameters of the loss function are not clear. How to select the hyperparameters and their importance. The importance of each term of the loss is not very clear. 
5. The relation of using masked input and the "sparse" is not clear. The mask is for sparse latent process, while it is also stated in the paper the method does not rely on sparsity in latent processes.
6. The structure of using "contrastive learning" and "flow" to ensure the identifiability are already exist in previous work by Khemakhem et al. I am not sure of the comparison with the previous work and the exact novelty. 
summary_of_the_review: The idea is very promising but some technical details are not very clear. Besides the proofs in the appendix, it is recommended to describe more about the architecture and implementation details in the appendix too, if length limited. It'e better also to add some discussion about how the conditions are satisfied in real applications and what to do if the conditions are not satisfied
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This papers proposes new conditions for identification of latent temporal causal processes, for both non parametric and parametric processes. They then propose an architecture emboyding the derived conditions such as to recover the latent temporal processes. The approach is shown to give better performance than compared baselines on synthetic data and to show good performance on non-synthetic datasets.
main_review: I think this paper is very promising and represents a step forward in the causal inference of temporal processes. However, there is a lack of clarity in the derivation of the model architecture that prevent future readers to use this paper at its full potential. I would be happy to increase my score if this issue is resolved.

### Model details

- The model derivation lacks clarity. I would not be able to reproduce the proposed architecture based only on the information given in the paper (appendix included). I think this is quite critical, even if the code is available online. In particular, Figure 2 has tildas on z, which are not to be found in the derivations of section 3 and this confusion seems still present in 3.1.1. I really encourage the authors to invest effort in making this architure more understandable, and especially highlight how each of the assumptions from theorems 1 and 2 are embodied. 

- Again regarding clarity, the $\mathbf{u}$ are never defined nor properly introduced anywhere. This makes it harder to understand assumption 2 of Theorem 1. The IN condition, similarly, lacks a proper introduction (I assume it is the independent noise assumption).

- The sufficient variability condition is still a bit cryptic to me in terms of its implications. I think it would be nice to expand a bit on that as well.

### Experiments

- For the recovery of the causal relations, you compare the causal skeletons of the true and inferred process but it could you give a measure of the discrepancy between the causal graphs ? Such as minimum edit distance (up to a permutation of the true causal graph) ?

- The performance of the approach are very good compared to the baselines on the synthetic data. However, the data is generated as to fit the assumptions of Theorem 1 and 2. I would be good to also report the results of the baselines on the real-world datasets when the conditions are more general. Indeed, in this case, it's not clear if the proposed approach would as neatly outperform the baselines.

- In "Towards generalizing conditions" and in the conclusion, you seem to state that the only limitation of the method is that is assumes constant causal graph with no instantaneous relations. This is without counting on the assumptions laid out in the theorems. In particular, non-stationary noise and sufficient variablity. Of course, you need some conditions to be able to recover something but the impact of violating those conditions would be worth investigating.

summary_of_the_review: Very promising paper but there is some lack of clarity in the descriptions of the architure used, preventing future readers to use this paper at its full potential. I would be happy to increase my score if this issue is resolved. The baselines should also be run on the real-world datasets for comparison with the proposed approach when the data generating process does not exactly matches the assumptions made in this work.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors proposed a method to recover time-delayed latent causal variables and identify their relations from measured temporal data called Latent tEmporally cAusal Processes estimation (LEAP). This method leverages VAEs by enforcing our conditions through proper constraints in the causal process prior for recovering time-delayed latent processes from nonlinear mixtures without sparsity assumptions. The experimental results on synthetic datasets show that the proposed method outperformed baselines that do not leverage history or nonstationarity information. The results on public datasets including KiTTiMask, Mass-Spring System, and CMU MoCap demonstrate that temporally causal latent processes were reliably identiﬁed from observed variables under different dependency structures.

main_review: Please list both the strengths and weaknesses of the paper. When discussing weaknesses, please provide concrete, actionable feedback on the paper.

The specific strength of this paper is described above. Although technical contributions are derived from mixing the previous techniques, the problem setting seems to be completely new (but I am not an expert in this field), and experiments were sufficiently performed. Partly I have some specific comments as follows. 

1. Sections 1 and 2: Parent (node) may be explicitly unclear. Is it the cause of the factor z? 

2. Sections 1: follow follow

3. Definition 1: p_{\epsilon} is not defined in the introduction.

4. Fig 3c: It was unclear to me how to see Fig 3c. Is the negative correlation acceptable even if Figs 2a and 2b do not seem to have negative correlations or values (or ignore the signs?)

5. Fig 5: Some symbols were lost.
summary_of_the_review: The problem setting seems to be completely new, experiments were sufficiently performed, the results show the superiority of the proposed method, whereas the technical contributions are derived from mixing the previous techniques. 
Since I think the strengths in this paper totally outperformed the weaknesses, I provided a higher rating at this stage (but note that I am not an expert in this field).
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper considers the learning of a temporal latent causal process: x_t=g(z_t), where x_t are observations, z_t are the underlying latent variables, and g is unknown. The paper makes two contributions: 1) presenting conditions under which the process can be identified (up to a permutation of the latent variables and their componentwise invertible transformations), and 2) developing a training framework that enforces the assumed conditions. The first contribution is theoretical and the second methodological. In addition, the paper presents empirical evidence for the ability of the method to recover the underlying causal process using synthetic experiments and real-world data.

Two cases are considered: 1) non-parametric, where a non-parametric transition is assumed in the latent space, and 2) parametric, where linear additive transitions are assumed. The main assumption for identifiability is the independent noise (IN) condition, which states that noise for the dimensions of z_t are independent of each other and across time steps t. This assumption is enforced through a contrastive approach during training. In addition, in the non-parametric case the noise is assumed non-stationary, such there exists sufficiently many auxiliary variables that affect the distribution of the non-stationary noise.

main_review: Pros:
- In general, the topic of deriving rigorous conditions for the identifiability of temporal causal LVMs, and developing novel methods based on these conditions, is very interesting and relevant to the ML community. 
-To my knowledge the setup considered in the article is novel (though see the cons below).
-The theoretical derivations seem correct (although I only glanced through the proofs in the supplement).
-The experiments are sufficiently thorough, including synthetic experiments and 3 real-world data sets.

Cons and questions to clarify:
- It seems that the (independence) assumptions, proofs, and the methodological ideas (e.g. contrastive learning to ensure independece) are rather close to previous works (e.g. by Klindt or by Hyvärinen and Morioka). The previous works have been properly cited in the text. Nonetheless, it would be beneficial for me to see a more concrete side-by-side comparison, to see where the novelty is located exactly (e.g. in the Supplement). For example, Eq. (2) and (4) very clearly present the current setup. Would it be possible to compare these with similar concise summaries of the previous setups?
-The method comes with many hyperparameters and it is not clear what their values are (e.g. beta, gamma, sigma in Eq. 11) or how they were selected (e.g. the latent dimension and the time lag in the experiments). Hence, judging how well the method really behaves in practice is difficult. 
-Some assumptions are a bit abstract, for example the "Sufficient variability". How would a user of the method know if his/her data satisfy these assumptions? How do you know these assumptions are satisfied in your real world examples, and what happens if some of the assumptions are violated?
-The clarity of the presentation could be improved. For example, some figures are so small that it is impossible to make anything out of them when printed out (e.g. Fig. 8). Another concrete example: in the preliminaries of the proof for identifiability (section A.1) a sentence: "Since the learned unmixing function..." Why is g hat called the "unmixing function" here (and not elsewhere), what is g* and what is h (the sentence breaks in the middle before explaining h), and why the "we can assume..."? Similar small clarity and grammatical issues make the reading a bit cumbersome at places, although the technically the papers appears correct in general.

Very minor:
- Can you clarify where Eq. (6) comes from?
-In Fig 9, on the x-axis you have z_{t-1}^2 and z_{t-2}^2 twice. Is this a typo?

summary_of_the_review: An interesting paper with relevant results on the identifiability of latent causal processes. I hope the authors could clarify my above concerns in their rebuttal.

POST-REBUTTAL UPDATE:
I found the authors had done a careful job addressing my original concerns. I don't have any remaining significant concerns. I find the article interesting and significant, and I support its acceptance at this stage. I will increase my score from 5 to 8.
review_title: null

Meta-review: This paper proposes two new sets of conditions under which we can identify temporally causal latent processes. In this sense, this work makes valuable contributions to the theories of identifiability in this topic. The authors also propose LEAP, extending the VAE, to estimate temporally causal latent processes.

The reviewers had many constructive comments, and the authors strived to address them. In the end, the reviewers were satisfied with the final version of the paper. 

Given that the theoretical identifiability theorems are major parts of the paper, I encourage the authors to elaborate more on the two sets of assumptions. They should discuss when these assumptions will hold and provide examples in which they will be violated.
Decision: Accept (Poster)