Venue: CLeaR 2022 Conference

Reviews:

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: Summary: This paper proposes a causal discovery method using leveraging interventional data. This study extends Brouillard et al. (2020). The previous study assumes that the correspondence between samples and interventions is known. This paper relaxes the assumption and propose a method based on neural networks and variational inference.
Main review: # Quality and clarity
- This paper is well written, well organized, and clear.
- It contains a theoretical reasoning of their method, and their method is likely reproducible.
- The paper is self-contained, with a good description of previous studies and existing methods.
- The paper contains the results of the experiments. Some of the experimental results of the proposed method are inferior to those of the existing methods, but reasonable explanations are given for the results.

# Originality and significance
-  This paper extends Brouillard et al. (2020) by relaxing their assumptions. Using interventional data for causal discovery is a very important problem but little has been studied. The relaxing of the assumption is important for the causal discovery from interventional data.

# Feedback to the authors
- What variable does the S in the first line of Section 2.1 refer to?
review_title: null

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary: Interesting problem, sound approach, empirical evaluation should be improved
Main review: The paper focuses on the under-explored problem of score-based causal discovery, where for each sample we don't know which was the interventional distribution that generated it. It proposes a differentiable structure learning approach based on variational inference and Dirichlet processes. The framework can be extend to the case of partially knowing the intervention targets, or at least knowing which batch of samples is generated by the same intervention.

The paper is generally well-written and to the best of my knowledge technically sound. I thought the setting was particularly interesting, with many applications.

One of my concerns is that results are only on single-node interventions on simulated data (and with single node interventions on each of the variables) and the [Sachs et al. 2005] dataset, which is not the best benchmark for this case. I would really be curious about any result on multiple-targets and in the case in which one doesn't have an intervention per variable. Even in this case, the stochastic interventions results are a bit lukewarm, and I'm confused why there are no perfect interventions.


# Minor points

Currently the method assumes the parents cannot change in an intervention - would it then make sense to just consider as the "original" SCM M the one in which you can have the union of the sets of parents (assuming it is still acyclic) in the different environments and then switch them off in each specific environment? 

If the $\theta_j$ are shared across interventions, I'm not sure I understand exactly how the soft interventions are modelled
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: Summary: The authors propose a score-based causal discovery method for the setting where interventions are performed latently (i.e., the targets are unknown and even the environments from which they are sampled are unknown). More specifically, the authors propose a continuous-constrained optimization method that relies on neural networks and variational inference. The authors show competitive results compared to the case where the environment is known but the target is unknown.
Main review: The problem addressed is important since the setting considered is realistic and challenging. However, I have a few concerns: 1) since the authors don’t show the consistency of their methods, I would expect the experiments to be more thorough and 2) the method is not well situated within the causal literature, more particularly the link with latent confounders.

It is not clear how this setting is different from causal discovery in the presence of latent confounders. The differences should be clearly explained. If there are no differences, then several other state-of-the-art methods exist (e.g., Spirtes et al., 1999) and, moreover, compared to the proposed method, they often have guarantees to recover the equivalence class. In this case, the authors should definitely include these methods in their related work section, compare their method to at least one of these, and explain clearly how their proposed method differs. To develop the latter point, the related work is really short and doesn't situate the present work in the causal discovery literature.

This work has a limited novelty. It is based directly on Brouillard et al., 2020 that is also a continuous-constrained optimization method that relies on neural networks and can deal with interventional data, possibly with unknown targets (also it uses the same components: random matrices trained with the Gumbel-Softmax trick, normalizing flows (DSF), augmented Lagrangian, etc). The main contribution of the present work is to add variational inference in order to deal with latent interventions. However, contrary to previous works, in this new condition, the authors don’t have any identification results and thus, the graph with the best score can be an element that is not part of the I-Markov equivalence class. As mentioned in the previous paragraph, if their setting is equivalent to the presence of latent confounders, then the novelty is furthermore reduced since other methods exist with theoretical guarantees.

**Experiments**
Since they don’t have theoretical guarantees, I would have expected experiments to be more convincing. 

For the synthetic dataset experiments, they don’t have any external baseline method. They show that “latent” is comparable to “known”. I wonder if it is simply explained by the fact that the different environments are easily distinguishable (see Fig. 3, it seems to be the case, more obviously for atomic interventions). If it is the case, then a method that first finds the environments (performing some kind of clustering for example) and then performs classical causal discovery with unknown targets would have a similar performance. This would also explain the results of the semi-supervised learning experiments where the knowledge of the environment doesn’t help.

All the experiments are performed on very small graphs (2-11 vertices). Furthermore, the authors state that their method scales well w.r.t. the number of interventions. It would be interesting to see if this method is applicable to larger graphs and how well it scales in practice. 

For the real-world dataset, since their assumptions are less restrictive, I would expect to have at least another dataset than the classical Sachs et al., 2005 (which is imperfect as noted by the authors).

For the hyperparameter selection, the procedure is not clearly explained. How many experiments were performed in order to determine the optimal $\alpha$, h, K, learning rate, etc; and was it done on only one type of dataset? If I understand correctly, once determined, the same hyperparameters are applied for all the experiments and a grid-search was only performed on $\lambda_G$ and $\gamma$? The authors state that the models were trained for 500 epochs for the first iteration of the augmented Lagrangian. What are the parameters used for the augmented Lagragian? Why do the authors use a fixed number of iterations?

For the reproducibility of the experiments, while the authors state that the code will be provided, all the hyperparameters used should be clearly described in the appendix (how many layers for each neural network, the hyperparameters related to the augmented lagrangian, etc).

**Method**
In Section 3.2, if I understand correctly $\lambda_G$ is a scalar defined by the user. Why are all the $\lambda_{ij}$ set to this value? It seems they should be learned independently.

In Section 3.3, instead of using different neural networks for each interventional distribution, the authors use an intervention embedding $\mathcal{U}$ and share the parameters $\theta_j$ between all interventions. By doing this, they have a method that scales well w.r.t. the number of interventions, but they don’t fully use the invariance of the conditional for different interventional settings. Also, it gives them a prior that causal mechanisms should be similar when intervened upon, which is not always the case.

**Typos and short comments**
- In abstract, “a infinite” -> an infinite
- Abstract, extra space after “models”
- Section 2.1, what is S in the SCM?
- Section 2.4, to be correct, Zheng et al., 2018 proposed the constraint $tr(e^{W \odot W}) -d = 0$
- Section 4: “All of the possible graphs share the MEC”. The same MEC? It is not true, the two first graphs share the same MEC (A -- B), while the last one has its own MEC (A  B).
- Section 4: “is worse than”
- Section 4: “with a full batch (” (space before the parenthesis)
- Section 5, p.12: “representation”
- Section 5, real-world dataset: should cite Joris et al., 2020 for the observations related to Sachs.
- Appendix B, what is $k$ and $l$ in the Table 3 caption?
- Appendix C, “Given an vector”
- Appendix C, “as show in Figure 4”
- Appendix D, “a the correspondences”
- Appendix D, there are extra spaces in parentheses
- When citing, usually the publications are in order starting from the most recent to the oldest
review_title: null

Meta-review: I recommend acceptance and editing the manuscript to highlight the limited nature of experiments (more datasets and more realistic applications would be welcome). Still, all reviewers see this submission as interesting and worth presenting in the conference.
Decision: Accept (Poster)