Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper focuses on the challenges of estimating post-nonlinear models (PNLs) for causal discovery in the bivariate case. Indeed, solving the practical issues of estimating PNLs is an important topic.

The problems with existing methods are that they can produce local minima which are trivial and meaningless solutions and that the misspecification of noise distributions can lead to wrong causal discovery results.
The paper proposes an objective that avoids relying on noise distributions (similar property as HSIC minimization methods) and enforces the independence of residual and potential direction cause. 

It combines the objective function in the alternating conditional expectation (ACE) algorithm (Breiman & Friedman, 1985) with a dependency measure as the penalty term for enforcing the residual to be independent of the potential direct cause. And it determines the causal direction by using the independent tests of the residual and potential direct cause. 

The main experiments in Table 2 cannot show that the proposed method is significantly better than the other methods empirically.

strength_and_weaknesses: ## Writing 
(+) The paper is well-written in general and makes the problem that it is solving and the contributions clear. 

(-) However, I would also suggest spending more effort on the proposed method (Page 6 - Page 7) than illustrating the problem (Page 2 - Page 5). Because I would expect the readers can know some about the problems but not really about the proposed method. It can be not the case but then perhaps readers care more about the concerns of the proposed method.

## Experiments
(+) The work provides transparent and nice-illustrated figures. 

(-) But in Table 2, it cannot show the proposed method stands out compared with the others empirically. A fairer claim is that it is better than the related works based similar framework. As for real-world data, the ANM-based method performs surprisingly well and is comparable with the proposed method. 

(-) Moreover, in this case, it would be good to show experimental results on the typical benchmark dataset, the 100 (or more) Tuebingen cause-effect pairs. Not necessary to show superiority, but to provide a comparison with the others.

## Concerns about Algorithm 2
(-) The causal score is a single number of the difference between the dependency measures in different directions. It would be more convincing to report the numbers for the experiments, especially to which degree (or within which threshold), we can believe the number is larger than zero, smaller than zero, or equal. Moreover, I am worried that in real-world data or finite samples, how much we can trust a single number that gives us the causal relationship. And it doesn't seem to show the experiments about the independent case.


## Proposed method
(+) The work clearly formulates the proposed method with a discussion of optimization process. 

(-) Nevertheless, __my main concern__ is that the proposed objective (13) lacks a justification and theoretical guarantee from the identifiability concern. ( Note that I am not asking for an identifiability proof, but the justification and theoretical guarantee for that given the PNLs under identifiability assumptions, can the method be used for determining the causal direction?)

As actionable feedback, can the authors show that  
(i) for objective (8), is it in any way related to maximum likelihood or minimizing mutual information?

(ii) as for the model in the causal direction and the one in the reverse causal direction, can objective (8) imply that the one in the causal direction has a smaller value of the optimal objective function value than the other one? 

Furthermore,

(iii) as for objective (13), how can the property maintain by adding the penalty term? A concern is that the penalty term is added as a "soft" constraint which is not necessary to be exactly the case. Then, is it possible that the optimal solution by solving (13) can be the local minima which are taken as a trade-off between the objective (8) and the penalty term? Then, will this lead to a misspecified model ? will this lead to a problem for causal discovery?


To further elaborate on my point:
The authors introduced the PNL-MLP of Zhang & Hyvärinen (2009), which uses mutual information for estimating the model and later uses independent tests for causal discovery. And the authors point out the problem of using mutual information as the objective, which can be hard with large-scale datasets. But an important fact of using the maximum likelihood or minimizing mutual information is that they are well justified by the identifiability of PNLs as illustrated in [1], especially, the independent noise assumption. 
Similarly, as for the regression by dependence minimization (Mooij et al., 2009), it directly minimizes the HSIC score to enforce the independent noise assumption and pick up the model with a smaller score as the causal direction. This is fine because it directly uses the assumption as objective from the perspective. But for the objective in (Uemura & Shimizu 2020) and this paper, they neither directly use the independent noise assumption nor have a theoretical guarantee of the identifiability as Thm.2 and Thm. 3 in [1]. Therefore, to fix my concerns, maybe the authors could consider my actionable feedback.

[1] Zhang, K., Wang, Z., Zhang, J., & Schölkopf, B. (2015). On estimation of functional causal models: general results and application to the post-nonlinear causal model. ACM Transactions on Intelligent Systems and Technology (TIST), 7(2), 1-22.



clarity,_quality,_novelty_and_reproducibility: The work focuses on an important problem for causal discovery. It is in general well-written. The related points can be found in the previous comments.
summary_of_the_review: The paper works on the practical issues of estimating PNLs and shows the problems of existing methods. It proposes to use the objective,  maximum correlation with a dependency measure as the penalty term. My main concern is about the theoretical guarantee and the identifiability of the results given by the proposed objective. My minor concern is about the experiments and the causal score used in Algorithm 2.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The authors suggest using the randomized dependence coefficient (RDC) instead of the Hilbert-Schmidt independence criterion (HSIC) for the independent test for performing bivariate causal discovery using MLPs in the post-nonlinear setting. 
strength_and_weaknesses: The practical take-aways such as the strength of RCD with finite samples could be useful. 

Authors have not tested their algorithm on Tuebingen dataset, which is the baseline for bivariate causal discovery.
clarity,_quality,_novelty_and_reproducibility: Presentation is clear in my opinion. 

Novelty is not very high, as commented above. 

Proposed methods seem reproducable. 
summary_of_the_review: Proposing using a different independence measure is not a sufficient contribution in my opinion. This is analogous to having a different algorithm around the PC algorithm by simply using different conditional independence testers. Please see below for some more minor comments.

On Justification of the Method:
"However, the state-of-the-art (SOTA) PNL-based algorithms involve highly non-convex objectives for neural network training, which are
time-consuming and unable to produce meaningful solutions with finite samples."
Causal discovery literature does not generally rely on neural network training. So I find this justification a bit inadequate. How about fitting a function class then checking residual independence as is typically done with ANM models?

"moreover, the discovered DAG may not necessarily be causal."
This is not a fair criticism either. The experts are aware that GES would return a graph in the MEC. 

"In this paper, we will focus on a more fundamental problem"
the relative fundamental-ness of bivariate vs. full graph discovery is subjective. I would suggest authors refrain from such subjective comparative statements.

On Experiments:

A typical benchmark used for bivariate causal discovery is the Tuebingen dataset. Did you test your algorithm on this real data? It would be nice to address why this was not added.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper studies bivariate causal direction learning from observational data. The authors focus on the general bivariate model, i.e., the post-nonlinear model, and propose a new method to learn the model. The contributions are as follows, the authors first analyze the drawbacks of the existing estimation methods, e.g., the PNL-MLP algorithm, and the AbPNL algorithm. Then they propose a new optimized function to estimate the PNL model by using the Soft-HCR. Finally, the authors apply their methods in both synthetic and Gene datasets to verify the efficiency. 
strength_and_weaknesses: Strength

The authors focus on the challenge of causal direction inferring. This is an important but challenging problem. 

The paper builds on prior work in the field of casual discovery in PNL model. The authors put some effort into mitigating the problem of time-consuming and unreliable results with finite samples. 

The basic idea is interesting and useful.

 This paper is well-written and well-organized.


Weakness

Some results are based on simulation analysis rather than theoretical analysis.


Some concerns or questions

1. The last but one paragraph on Page 2: a biased HSIC? It may be unbiased?

2. ﻿In Table 2, the ROC-AUCs of IGCI have better accuracy than the proposed method in the synthetic datasets. However, IGCI has worse accuracy in real-data sets. Can you explain this result?

clarity,_quality,_novelty_and_reproducibility: The idea of combining the Soft-HCR and dependence measure (HSIC) is non-trivial. This paper is well-written.
summary_of_the_review: The authors make some progress on the hard problem of estimating the causal direction of interest only using the observed variables.  The proposed method is non-trivial and useful with finite samples. However, some results are based on simulation analysis rather than theoretical analysis.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a new bivariate causal inference algorithm called MC-PNL (maximal correlation with independence regularisation). The authors propose to use the randomised dependence coefficient (RDC) instead of HSIC test often used by the causal inference community. 

strength_and_weaknesses: Strengths. The paper is in general well-written. It does a good overview of the state-of-the art methods. The proposed method is novel and worth being studied. 

Weaknesses. It is stated that the proposed method outperforms the state-of-the-art independence test-based methods (bivariate causal inference), however, the method is much worse than some other state-of-the-art causal discovery methods. The results of the numerical experiments are not so convincing. I see that the idea is to show that the method outperforms the test-based methods, however, the IGCI method which is also based on the same asymmetry (independence of mechanism) hypothesis performs much better. Do you have some intuition? Are the independence tests used not efficient enough? And the future work should focus on a more efficient tests?  

Figure 2 is unclear. What is the true causal direction? And how the plots should be interpreted? Taking, e.g., subplot a), above and below, we clearly see that the red line is well-fitted. What is the direction? And what can be deduced from the estimated residual values? 

Algorithm 2 abstains from the decision only if C = 0. It is not too strict for real (noisy) applications? Were it be not better to introduce some \epsilon, so that if abs(C) < \epsilon the decision can not be made? As it is done, e.g. in F. Liu and L. Chan,“Causal inference on discrete data via estimating distance correlations,” Neural Computation, vol. 28, 2016.  

Finally, in the problem formulation (equation 13), there are two parts: the HGR correlation and RDC (instead of HSIC which is, however, seems to be used in the paper). Could you study the efficiency of each of these parts?  

Section 2.2: generating process and the causalities: I guess a number of assumptions are made behind the description of the idea, and these assumptions need to be mentioned. 

In the numerical experiments (Section 5.2.) did you fix C = 0 (as in the Algorithm 2)? 

While reading, I have an impression that the HSIC takes too much room in the paper, finally the HSIC is criticised and is proposed to be replaced.


clarity,_quality,_novelty_and_reproducibility: The paper is well-written. 
There is some novelty in the contribution. 
summary_of_the_review: Some points need to be clarified. 
I have some concerns about the numerical experiments, since the reported results outperform some existing methods but are much worse compared to some other state-of-the-art approaches. 
A number of assumptions made in the theoretical part are not mentioned explicitly.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper focuses on the challenges of estimating post-nonlinear models (PNLs) for causal discovery in the bivariate case. The proposed method combines the objective function in the alternating conditional expectation (ACE) algorithm with a dependency measure as the penalty term for enforcing the residual to be independent of the potential direct cause. It then determines the causal direction by using the independent tests of the residual and hypothetical direct cause. The paper is in general well written, and the tackled problem is interesting and has direct practical implications. However, given that causal discovery aims to find true causal direction, the paper will benefit a lot from suitable theoretical guarantees.
justification_for_why_not_higher_score: Given that causal discovery aims to find true causal direction, the paper will benefit a lot from suitable theoretical guarantees.
justification_for_why_not_lower_score: The paper is in general well written, and the tackled problem is interesting and has direct practical implications.
Decision: Reject