Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: In this paper, the authors aim to address the problem of causal discovery. Considering that the existing methods for causal discovery are hard to specify realistic inductive biases and require different assumptions, the authors propose an amortized variational inference model for causal discovery, which amortizes the process of causal structure learning and approximates the posterior over causal graphs given a data-generating distribution. The authors evaluate the performance of the proposed method on the synthetic data 13 and semi-synthetic gene expression data. However, there are some concerns as follows:
strengths_and_weaknesses: 
1.	One of the main concerns is that the authors do not provide sufficient theoretical validation, such as in the form of identifiability or consistency proofs. Identifiability is an important problem for causal discovery from observational data, and the authors seem to ignore this problem. 
2.	The main contribution of this paper is to leverage the variational inference to discover the causal structure. However, similar ideas have been proposed by other researchers like [1]. The main difference between the proposed method and [2] is that [2] is used for time-series data.  Therefore, the contributions of these are limited. 
3.	The authors assume causal sufficiency, so all the data should be extracted from the same distribution with one causal mechanism. And the domain variables that result in the circumstance of domain-shift or out-of-distribution [2], however, the authors consider the OOD generalization and cannot observe the domain variables. Hence, these experiments seem to be not reasonable. 
4.	Technologically, since the training data are extracted from one distribution, so only one causal structure can be learned. However, since the authors employ mini-batch optimization to learn the structure. What if different structures are extracted from different samples? 
5.	Finally, it is suggested that the authors should consider more latest baselines. 


[1] Amortized Causal Discovery: Learning to Infer Causal Graphs from Time-Series Data 
[2] domain adaptation as a problem of inference on graphical model 

questions: Please refer 'Strengths And Weaknesses'
limitations: Please refer 'Strengths And Weaknesses'
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes a method for learning a causal graph from data, by training a neural network on similar data from a simulator that generates both random graphs and data from those graphs, so that the model learns to predict graph structures given data. Data points can be marked as observational or interventional in arbitrary ways.
strengths_and_weaknesses: The method proposed seems extremely powerful and general, both judging by its design and by its empirical performance. I expect it will have a very high impact.

The paper is also very clearly written.

I am not familiar enough with related work on amortized variational inference to assess the originality of this approach.

UPDATE: I learned from the other reviewers that this submission is more closely related to existing work than I thought. I have reduced my score (as well as my confidence) accordingly.
questions: 1. (Section 2.1) I wasn't sure at first what definition of causal structure you meant here. But having read the paper, I suppose it doesn't really matter: whatever the relation between graph and data might be, if you have a simulator that captures this relation, AVICI will learn how to recover graphs given datasets. Or are there assumptions I'm overlooking here?

2. (Section 2.2) The sentence on lines 80-82, comparing AVICI to work leveraging multi-domain data, came across as odd. The methods just have different (orthogonal?) goals, but here it sounds like those other methods are weakened by some assumption. ("Orthogonal", because maybe if AVICI or a method like it is fed multi-domain datasets, it could learn to do the same task.)

3. (Figure 3) The SID- and SHD-ratios have a counterintuitive definition, and furthermore are apparently usually larger than 1, often by quite a bit it seems from the direction of the red and green lines at F1. Is there a better way to visualise these results?

Minor points:
* below (1): "for any" -> "for some" ($\exists$, not $\forall$)

limitations: Yes
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper introduces an amortized inference approach to learn causal structures from observational/interventional data, by leveraging variational inference method and permutation invariant/equivarient models. Experiment results on synthetic and semi-synthetic data are provided to show that the proposed approach exhibits generalization capabilities under distribution shifts.
strengths_and_weaknesses: Pros:
- The proposed method is sound and novel.
- The problem studied is interesting and relevant to the field of causal discovery, which helps bridge the gap between causality and machine learning.
- The empirical studies are conducted across different data generating processes, number of variables, graph types, etc.

Cons:
- The proposed idea of amortized inference for multivariate causal discovery is not completely novel. It has been explored by existing work "Supervised Whole DAG Causal Discovery", which should be discussed in the introduction and compared in the experiments.
- The model architecture in Sec. 4.2 appears to be very similar to the encoder-decoder architecture for causal graph generation in paper "Causal Discovery with Reinforcement Learning", and should be discussed/compared in the paper. The latter also applies a transformer encoder that attends over axis $d$ and a similar decoder to generate graph. The only difference appears to be that the latter does not apply attention over axis $n$.
- It would be great to provide some ablation studies on the attention over axes $n$ and $d$ to study the contribution of each module.
- The current OOD experiments change only the linear function weights, kernel scales, and noise distribution (L224-233). I believe that the ability to generalize to a completely different data-generating process is also important, e.g., train distribution contains RFF dataset while test distribution contains MLP dataset. Another aspect is to generalize from synthetic data to real data, e.g., train distribution contains RFF dataset while test dataset is a real dataset (such as the one by "Causal Protein-Signaling Networks Derived from Multiparameter Single-Cell Data").
- Could the authors explain how users should apply the learned amortized model in practice? Will there be a separate learned model for different data-generating processes like linear, RFF, MLP, and GRN? That means users have to manually train/pick an amortized model for a dataset of interest based on their understanding of the dataset. Could we combine all different data-generating processes into a large dataset and train a single, more powerful, amortized model? This is what I might expect from a amortized/supervised causal discovery method.
- Stronger baselines:
    - DAG-GNN is only applicable for a limited type of nonlinear data-generating process (and not RFF). I suggest the authors to include GRAN-DAG ("Gradient-Based Neural DAG Learning") with preliminary neighborhood selection and pruning as a baseline.
    - The authors should also include non-parametric scoring criterion and test for GES and PC.

Minor:
- It was surprising to see that LiNGAM does not perform well on the linear non-Gaussian synthetic data. Have the authors considered the improved method in "DirectLiNGAM: A Direct Method for Learning a Linear Non-Gaussian Structural Equation Model"?
- The proposed method cannot ensure convergence to acyclic solution, which might be important in some cases. Could the authors explain their choice of dual ascent to solve the acyclicity-constrained problem, instead of penalty/augmented Lagrangian method (which is more widely adopted by existing works on differentiable causal discovery)?

Post-rebuttal:

I have read the rebuttals and further responses from the authors. I think the paper still lacks a number of experiments:
- Systematic ablation study (this is particularly important in my opinion)
- Comparison with existing decoders in Zhu et al., 2020
- Benchmarks for in-distribution setting, i.e., homogeneous noise (the proposed method should ideally perform well as compared to the baselines for both in-distribution and out-distribution settings)
- Stronger baselines like GRAN-DAG and GES using nonparametric scoring criterion

Given the limited time of rebuttal period, it is understandable that it is difficult to finish conducting all experiments, and the authors have promised to include them in the final version. Therefore, my concerns have been mostly addressed and I have increased my score to 6.
questions: The suggestions and questions for the authors are outlined in my comment to the previous question.
limitations: The authors have adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

Meta-review: metareview: In this paper, the authors introduce an amortized inference approach to learn causal structures from observational/interventional data. Most of the reviewers consider the proposed method is sound and novel and their questions have been well addressed.
confidence: Certain
Decision: Accept