Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: Edit after author rebuttal and author additions:

I have updated my score from a weak reject (3) to a weak accept (6).

Justification:

1. The authors have pointed out that I misunderstood one of their contributions: pointing out that they are demonstrating that the univariate case is an insufficient setting to test causal discovery methods because it can be done without even looking at the conditional distributions (just the marginals). This contribution seems important to orient future work. They have made this more clear in their recent upload and would likely make it even more clear in a camera-ready version.

2. The authors have made their contribution to the multivariate setting more substantial by adding evaluation on the MOUS-MEG real-world dataset and have better positioned their work relative to others by adding comparisions to multivariate extensions of PNL and CGNN.

====================================================================================================

Original Review:

Summary: The authors focus on the problem of inferring whether the causal structure X —> Y or Y —> X. They first consider the case where both X and Y are scalar (univariate) random variables and then consider the case where X and Y are vector-valued (multivariate) random variables. In the scalar case, motivated by the idea that the effect could be less entropic than the cause (due to data processing inequality), they introduce a method based on comparing reconstruction losses of X and Y and show competitive results in Tables 1 and 2. They establish that this method is not sufficient for the multivariate case in Lemma 2 and move to a new method for the multivariate case. They prove identifiability for this new method in for the multivariate case in Section 4.2 and claim state-of-the-art (SOTA) results in Table 3. 

Main contributions:
- Presents a causal discovery technique for the univariate cases that only examines the marginal distributions of X and Y and seems fairly competitive (Tables 1 and 2)
- Extends the post-nonlinear identifiability analysis of Zhang & Hyv¨arinen (2009) from scalars to vectors and proved that their method will actually identify the correct causal direction
- Demonstrates competitive experimental results for both their univariate method
- Claims SOTA results for their multivariate method

Decision: I lean toward rejecting this paper because 1) I have several questions about the univariate case (see below) that would need to be resolved before I lean toward accept, 2) although I am not too familiar with the literature, I believe that this paper may be missing key related work that also uses independence testing for causal discovery (see, e.g., Heinze-Deml et al. (2017)’s Invariant Causal Prediction for Nonlinear Models), and 3) I am not yet convinced that the comparison done in Table 3 is fair and exhaustive.  

Sufficient reason to accept: If the theorems in Section 4.2 are found to checkout, and the SOTA results in Table 3 are found to be fair, exhaustive comparisons to the previous SOTA, their contribution to the multivariate case would seem to be sufficient for acceptance. I believe more discussion between the authors and reviewers is necessary here.

 Questions about univariate case:

 1. The motivation for the first method (entropy decreasing along a Markov chain due the data processing inequality) seems to only be valid when Y := f(X), but not necessarily when Y := f(X) + E. For example, let f be the identity function and E be independent to X. How did you resolve this argument against the intuition? 

2. Also, I thought the data processing inequality relates mutual information between variables, not necessarily their entropies. Can you make this connection more clear? 

Context for questions 3 and 4: In Section 3.1, you write, “estimating the entropy of each random variable from its samples does not present a consistent difference between the entropies h(X)  and h(Y). Our method, therefore, computes an alternative complexity score for X and, independently, for Y.” You then go on to link the entropy to the reconstruction error (your method) in Lemma 1 and show competitive results in Tables 1 and 2.

3. Why do you want to link the reconstruction error to entropy if you found a purely entropy-based method did not work?

4. Why did the purely entropy-based method not work while your method worked if the two are linked?

Questions about multivariate case:

5. Are you certain that BivariateFit and ANM are the only models that you should be comparing against for this multivariate setting?

6. What is CGNN’s runtime? Would you be able to compare against CGNN in time for a potential camera-ready version of this paper?
review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: This paper proposes to use the autoencoder to measure the complexity for identifying the cause and effect in the univariate case. For the multivariate case, this paper extends the PNL model and use GAN for enforcing the independence between cause and noise.

- However, my main concerns are regarding the assumption of this work, seeing that the assumption h(X)>h(Y) in the univariate case is easy to violate. For example, let Y=f(X)+N (a special case of PNL) with some high entropy N, then h(Y) could higher than the h(X). 

- In the multivariate case, it can be seen as incremental for PNL but does not offer new insights.
review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: Update:

The authors have successfully justified my concerns. Therefore, I have increased my score to 6.

Original comments:

In this paper, the authors consider learning causal directions from observational data from both univariate case and multi-dimensional case. In the univariate case, the authors propose a new method to learn causal directions by exploiting the complexities of cause and effect variables. In the multi-dimensional case, where the complexity can be balanced, the authors proposed a method that learns causal direction based on independence loss.

1. The independence loss part looks confusing to me. Standard results in SCM yields that the error term E is a function of both the outcome Y and X. How can you learn the term E just from Y itself? In other words, I am not sure if the conditions required in Theorem 2 is feasible. The authors need to provide some examples to justify that the conditions in Theorem 2 are feasible conditions.

2. In fact, the novel idea of learning causal directions based on independence test has been extensively studied in the previous literature. I regret that this has not been mentioned in the current manuscript. Examples include:

http://www.jmlr.org/papers/v12/shimizu11a.html

In conclusion, since the idea of using independence relations for learning the causal directions is not a very new idea and a lot of discussion of the theoretical analysis is still missing. I regret that this work seems not strong enough to be accepted by ICLR.
review_title: Official Blind Review #2

Meta-review: The author response and revisions to the manuscript motivated two reviewers to increase their scores to weak accept. While these revisions increased the quality of the work, the overall assessment is just shy of the threshold for inclusion.
Decision: Reject