Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 8: Accept
review_content: In this paper, the authors propose an RL-based structure searching method for causal discovery. The authors reformulate the score-based causal discovery problem into an RL-format, which includes the reward function re-design, hyper-parameter choose, and graph generation. To my knowledge, it’s the first time that the RL algorithm is applied to causal discovery area for structure searching.
 
The authors’ contributions are:
(1) re-design the reword function which concludes the traditional score function and the acyclic constraint

(2) Theoretically prove that the maximizing the reward function is equivalent to maximizing the original score function under some choices of the hyper-parameters.

(3) Apply the reinforce gradient estimator to search the parameters related to adjacency matrix generation.  

(4) In the experiment, the authors conduct experiment on datasets which includes both linear/non-linear model with Gaussian/Non-gaussian noise.

(5) The authors public their code for reproducibility.
 
Overall, the idea of this paper is novel, and the experiment is comprehensive. I have the following concerns.
 
(1) In page 4 Encoder paragraph, the authors mention that the self-attention scheme is capable of finding the causal relationships. Why? In my opinion, the attention scheme only reflects the correlation relationship. The authors should give more clarifications to convince me about their beliefs.
 
(2) The authors first introduce the h(A) constraint in eqn. (4), and mentioned that only have that constraint would result in a large penalty weight. To solve this, the authors introduce the indicator function constraint. What if we only use the indicator function constraint? In this case, the equivalence is still satisfied, so I am confused about the motivation of imposing the h(A) constraint.
 
(3) In the last paragraph of page 5, why the authors adjust the predefined scores to a certain range?
 
(4) Whether the acyclic can be guaranteed after minimizing the negative reward function (the eqn.(6))? I.e., After the training process, whether the graph with the best reward can be theoretically guaranteed to be acyclic?
 
(5) In section 5.3, the authors mention that the generated graph may contain spurious edges? Whether the edges that in the cyclic are spurious? Whether the last pruning step contains pruning the cyclic path?
 
 
(6) In the experiment, the authors adopt three metrics. For better comparison, the author should clarify that: the smaller the FDR/SHD is, the better the performance, and the larger the TPR is, the better the performance.

(7) From the experimental results, the proposed method seems more superiors under the non-linear model case. Why? Could the authors give a few sentences about the guidance of the model selection in the real-world? i.e., when to select the proposed RL-based method? And under which case to choose RL-BIC, and which case to selection RL-BIC2?
 
(8) What’s training time, and how many samples are needed in the training process?
 
 
Minor:
1. In the page 4 decoder section, the notation of enc_i and enc_j is not clarified.

2. On page 5, the \Delta_1 and \Delta_2 are not explained.

3. For better reading experience, in table 1,2,3,4, the authors should bold value that has the best performance.


review_title: Official Blind Review #2

final_score: rating: 8: Accept
review_content: Update: after the revision, I have decided to increase my score to 8.

Original comments:

In this paper, the authors proposed a new reinforcement learning based algorithm to learn causal graphical models. Simulations on real and synthetic data also shows promise.

Pros

1. It's great to see the authors has done a comprehensive comparison with the other methods, especially under different simulation scenarios.

2. The novel idea of applying reinforcement learning to DAG search sounds intriguing. Reinforcement learning offers a powerful tool for policy evaluation and decision making. It’s good to see that the author can successfully extend such toolbox to the field of causal structure learning. To the best of the author’s knowledge, such idea has never been considered by previous work in causal graphical models.

Cons. 

1. In the introduction section, the authors claimed that “GES is not guaranteed in the finite sample regime”. This seems to be incorrect. For example, the Nandy et al. paper tackles exactly the finite sample problem. 

In conclusion, overall this is a sensible idea, although some of the preliminaries still remain to be polished.
review_title: Official Blind Review #3

final_score: rating: 8: Accept
review_content: This work addresses the task of causal discovery. The proposed contribution is to apply prior work which uses reinforcement learning for combinatorial optimization to structure learning. Specifically, the proposed optimization problem seeks to maximize a penalized score criterion subject to the acyclicity constraint proposed by Zheng, et al. Empirical results show the proposed method performing favorably in contrast to prior art. 

Overall I think this is a sensible idea, and the authors do a nice job of exposition, and empirical evaluation. 

My concerns are as follows:

* The novelty is somewhat limited, since the paper is combining two previously proposed ideas (combinatorial search and the acyclicity constraint) for structure learning.

* The paper is loose with technical points. Specifically, the authors claim to use the additive noise model, but then make no restrictions on f(). In this setting, it is fairly well known that we can only hope to learn up to the Markov equivalence class (not the fully directed graph), but there is no mention of this in the paper. 

With all of this said, I think overall the paper is an interesting addition to the causal discovery literature. 
review_title: Official Blind Review #1

Meta-review: This paper proposes an RL-based structure search method for causal discovery. The reviewers and AC think that the idea of applying reinforcement learning to causal structure discovery is novel and intriguing. While there were initially some concerns regarding presentation of the results, these have been taken care of during the discussion period. The reviewers agree that this is a very good submission, which merits acceptance to ICLR-2020.
Decision: Accept (Talk)