Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper proposes a MAML objective to learn causal graphs from data. The data in question is randomized but the algorithm does not have access to the identity of the intervention variable. So there is an added layer of complexity of deciphering which variable was intervened on. The MAML objective, in this case, links the causal structure to the slow-moving parameter theta_slow.

The novelty of the paper seems to be in the application of the MAML framework to causal discovery which is interesting to me. I think a little theory about the sensitivity of the claim of ' theta slow changes relate to the causal structure ' is important. Even showing empirically which sort of graphs and functions become issues for the model would be useful.

Here are my issues with the paper:
 - No error bars for cross-entropy are reported in the experiments.
 - The acyclic regularizer does not reject large length cycles than 3.
 - The ability to predict interventions seems to drop off sharply as the number of nodes increases. This suggests an inability to scale to more than 20 variables.
 - The experimental setup of uniformly sampling an intervening variable seems artificial to me.
 - MLP-specification of the SCM also seemed a bit artificial to me. 

Overall, the experiments look reasonable and the method itself seems interesting although further work is needed to show it is useful.

(writing comments) The paper could use a more structured re-write. I had trouble tracking terms around the paper. For example, there seems to be a difference between P_i and P because the former uses theta_i and the latter only uses theta_slow only.

---------------------------------

Updated score to 6 after rebuttal.
review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: This paper proposes an SCM-model based on masked neural networks to capture arbitrary conditional relationships combined with meta-learning-style adaptation to reflect the effects of various unknown interventions. Overall the paper is well written and easy to follow, but some conceptual issues remain.


- How come there is hardly any discussion of the identifiability issue beyond the few sentences in A.3. This is one of the key issues in learning SCMs and it is strange that the concept of "faithfulness" is not even mentioned in the paper.

In general, there is hardly any discussion of what conditions are required for the proposed estimates to even be valid. The authors seem to be optimistically assuming that their neural network + metalearning model will somehow pick up on the correct structure, without any actual conceptual investigation of this issue.

- The massive downside of neural nets is all the various hyperparameters one has to set (eg. architecture, optimizer, activations, etc). In this setting, how do the authors propose selecting hyperparameter values? How does the reader know the authors did not simply tune their hyperparameters to best match the underlying ground truth (I assume the proposed methodology has many more hyperparameters and thus more degrees of freedom here).
I would like to see the empirical performance of different variants of your model with different hyperparameter values to assess its sensitivity to these choices. 

- Why does one even care about the graph being acylic in this setting?
The mere fact that the authors require a regularizer to ensure acylicity suggests this approach is prone to mis-identifying the ground truth structure (which is always acyclic in the experiments).

- One main reason for SCM modeling in science and policy-making is for analysts to better understand the data generating phenomena.  However your use of neural networks here seems to hamper interpretability, so how do you reconcile this issue? Also is your sparsity regularizer satisfactory to confidently diagnose presence/absence of an edge (in constrast to statistical hypothesis tests, say based on conditional independence). Isn't this heavily influenced by the particular sparsity-regularizer value that happened to be selected?


- Related papers that utilize the same idea of predicting a variable conditioned on subset of other variables via neural network + masking strategy:

Ivanov et al (2019). VARIATIONAL AUTOENCODER WITH ARBITRARY CONDITIONING. 
https://openreview.net/pdf?id=SyxtJh0qYm

Li et al (2019). Flow Models for Arbitrary Conditional Likelihoods.
https://arxiv.org/abs/1909.06319

Yoon et al. GAIN: Missing data imputation using generative adversarial nets. Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, 2018. http://proceedings.mlr.press/v80/yoon18a.html

Douglas et al. A universal marginalizer for amortized inference in generative
models. arXiv preprint arXiv:1711.00695, 2017

For clarity, the authors should highlight the differences of their approach from these works (beyond the causal setting).

- Given the lack of theoretical / conceptual guarantees that the methodology will work, our faith in the proposed methodology rests entirely on the empirical experiments.  However, I find these a bit too basic to be very convincing, and would at least like to see more methods being compared (in particular for the simulated graphs as well).

- The authors should describe what are the underlying interventions in each dataset a bit more.

- The Figures should be better explained (took me a while to figure out what dots/colors represent).

- Why do the authors report cross entropy loss in Table 1? To my knowledge this is not a standard metric for measuring the quality of structure-estimates.
 
- Instead of ICP (which is constrained to be linear which is unrealistically simple), why don't the authors compare against nonlinearICP (which is more flexible like their neural networks):  

Heinze-Deml et al (2018). Invariant Causal Prediction for Nonlinear Models. https://arxiv.org/pdf/1706.08576.pdf
review_title: Official Blind Review #1

final_score: rating: 8: Accept
review_content: The paper develops a learning-based causal inference method that performs multiple tasks jointly: 

i) scalable discovery of the underlying Structured Causal Model (SCM) by modeling both structural assignments and the SCM as a continuously parameterized chain of distributions, 

ii) identification of the intervened variables, which are not known to the model a-priori unlike the mainstream causal inference setups,

iii) achieving the two aforementioned goals using meta-learning driven heuristics, i.e. interventions cause distributional shifts. 

While the paper adopts the core design choices from recent prior art (Bengio et al., 2019), the proposed methodology (especially ii)) is sufficiently novel to be published as a main-track conference paper. The paper is very well-written, follows a concrete and easy-to-follow story line. It solves multiple ambitious problems end-to-end and justifies the methodological novelty claims by a properly conducted set of experiments. The paper also successfully employs simple and useful but forgotten old techniques such as fast/slow parameter decomposition in the proposed model pipeline.

The intervention prediction heuristic is splendid. It is simple, sensible, and has been proven by experiments to be very effective. I would rate this as the primary novelty presented in this paper.

The paper can be improved if the below relatively minor concerns are addressed:

 i) It would be informative if the paper had a paragraph discussing also the fundamental limitations of the approach more openly. For instance, the choice of the neural net architecture used for the structural assignment might have a huge impact on the outcome, especially because the same architecture is repetitively used for all variables of the SCM. Furthermore, treatment of each variable with a fully independent neural net could cause overparameterization as the SCM grows in number of variables.

 ii) The paper makes a strong scalability claim across the variable size thanks to independent Bernoullis assigned on the adjacency matrix entries. However, it reports results only for very small SCMs. It is understandable that given the premature stage of the causal inference research might not grant standardized data sets at a larger scale, but at least lack of this quantitative scalability test could be acknowledged and the related claims could be a little bit softened.

 iii) I do not buy the argument in the first paragraph of Sec 3.5 about why the structural assignment functions need to be independent. As the model does not pose a distribution on neural net weights, sharing some weights (i.e. conditioning on them) would only bring conditional independence across the variables. I do not see a solid reason to try to avoid this. What is wrong for multiple variables to share some functional characteristics in their structural assignment? After all, some sort of conditional independence will be inevitable in modeling. If the variables share the same architecture, this is also conditional independence, not full independence. Relaxing the independence assumption and allowing some weight sharing could be beneficial at least for scalability of the model, could even bring about improved model fit due to cross-variable knowledge transfer.

Overall, none of the aforementioned three weaknesses is fundamental. In the status-quo, this is a spectactular research paper and my initial vote is an accept.
review_title: Official Blind Review #2

Meta-review: This paper proposes a metalearning objective to infer causal graphs from data based on masked neural networks to capture arbitrary conditional relationships. While the authors agree that the paper contains various interesting ideas, the theoretical and conceptual underpinnings of the proposed methodology are still lacking and the experiments cannot sufficiently make up for this. The method is definitely worth exploring more and a revision is likely to be accepted at another venue.
Decision: Reject