Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes transportability in DAG structure learning problem, which can be seen as a multi-task DAG learning problem or a cross validation method from a single task. The changes from NOTEARS methods is the addition from an average graph loss term additional to the differential scoring function. The method is compared with one baseline NOTEARS to show its superior performance in accuracy.
strength_and_weaknesses: Strength:
It seems this is a first work to learn one single DAG from multiple domain.
Subset construction is interesting, as it may improve the statistical property of the estimation. 

Weakness:
The technical contribution is rather limited, with a MSE terms on graphs. It would be interesting for authors to discuss the choice of such a regularization against potential other alternatives. 
Only one baseline is compared. Understandably this may be first work, but other works could and should be adapted as a baseline. For example, the multi-task DAG learning from [62].

Comments:
- subset construction: it is not fully clear how much the performance gain is from d-struct formulation, compared against an ensembled approach on different subset of data (maybe taking an average to obtain the final result with measure to ensure acyclicity).  This could also serve as a baseline.
- While transportability definition is clear, authors did not discuss much about it with respect to other standard problem regimes. For example, how is transportability problem different from a typical multi-task learning or transfer learning for DAGs?
- another way to improve technical contribution is to study feature differences or absences in different domains, and how to integrate them into one single DAG. 
clarity,_quality,_novelty_and_reproducibility: clarity: good

quality: good

originality: problem is new, but technically it is limited. 
summary_of_the_review: the paper addressed a new problem in DAG learning but with straightforward technical contribution.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper studies the problem of learning DAGs from observational data. In contrast to previous work under the continuous framework, the authors aim to learn a DAG that is transportable to other distributions over the same observables that respect a set of conditional independences. In a few words, the approach consists of learning K different DAGs where each DAG is penalized by how far away it is from the mean of the K DAGs, in this way, the approach remains fully differentiable and the resulting DAGs are somehow forced to be similar or equal for the K different datasets. The authors apply a preprocessing sampling step to make the method work even for the case of observing a single dataset. Some experiments are provided to demonstrate the efficacy of the proposed method against the nonparametric NOTEARS algorithm.
strength_and_weaknesses: ### Strengths

The core ideas of the paper are well-written and well-motivated. The authors do a good job of stating the problem under study and the contributions of their work. The method is simple to grasp and simple to adapt to existing differentiable approaches to learning DAGs.

### Weaknesses

The main weakness is the experimental section. 

* Lack of proper comparison to other methods. First, in my opinion, the correct baseline should not be plain NOTEARS but the version of NOTEARS with the same exact sampling mechanism for Dstruct and taking the average or median as the final output, i.e., the only difference to Dstruct should be the L_MSE step. I stress this point because we need to make sure that (part of) the improvements are not due to the ensemble mechanism (which would be a very incremental contribution) but to the addition of the L_MSE regularization. Second, there really should be comparisons to other methods, especially Bayesian methods such as DiBS [1]. This is because in those methods we can sample DAGs from a posterior distribution and, again, perhaps taking the mode or mean would result in a transportable DAG. For these Bayesian methods, I would feed all the data---even from different distributions---and see if they are robust to these changes in distributions. 

* The speed-up claims are rather disappointing, the graphs have a very small number of variables, $d=5$. To make a more precise claim about computational speed-ups there must be experiments for different values of d and not just n. This is because increasing $n$ really just makes the training of the neural nets longer but does not increase the complexity of enforcing acyclicity, for which $d$ does. Recent work [2] for instance ran the NOTEARS-MLP for up to 100 nodes leveraging a new acyclicity function. 

[1] Lorch et al. (2021), "DiBS: Differentiable Bayesian Structure Learning"
[2] Bello et al. (2022), “DAGMA: Learning DAGs via M-matrices and a Log-Determinant Acyclicity Characterization”

clarity,_quality,_novelty_and_reproducibility: The writing is a bit sloppy in some places. 

* The objective in equation (2) is many times referred to as the "score", which can be misleading since the score in a score-based method is simply what the authors write as $F(A)$ or the "regularized" score $F(A) + \lambda_1 * ||A||_1$ algorithm. The other terms w.r.t. $h$ stem from the augmented Lagrangian to solve the constrained problem. 

* In Section 3.1., the authors also mention $K$ "distinct" DSFs. I believe this *cannot* be the case. For instance, consider linear SEMs with equal variances, from Loh & Buhlmann (2014) we know that the ground-truth DAG is the global minimizer of the least squares (LS) score, i.e., these are identifiable models. Now, suppose one uses LS for dataset D1 and another score, call this S2, for another dataset D2. Moreover, suppose we are able to find the global minimizes in each case. Then, using LS on D1 already finds the correct DAG, while using S2 on D2 will find an incorrect DAG. Forcing both DAGs to be the same, as in Dstruct, will fail to find the correct DAG. Thus, to me, it does not make sense to use "distinct" DSFs.

* Algorithms 1 and 2 are a bit confusing, Alg 1 does not use h_tol and Alg 2 does not have rho_max as input. Please polish these minor details.

* In the paragraph on transportability on Page 8, what do "internal" graphs mean?

As per novelty, I like the idea of trying to find a transportable DAG, but I am unclear if the approach really does make sense due to my questions above. I am under the impression that really the core technical contribution is simply adding an average of DAGs as a measure to force the different DAGs to be the same, which looks like a fine contribution but perhaps not enough to grant publication at this time.

I don't have many concerns about quality and reproducibility.




summary_of_the_review: My main concerns are listed above which currently make me inclined toward rejection.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The authors consider the problem of learning DAG from heterogeneous datasets that share the same DAG. This problem statement is quite restricted. From Eq(4) we can see that the authors not only encourage the DAG to have the same structure, but also to have the same parameter. While in Figure 1, the authors assume that multiple datasets to have different distribution. In this case, there is only one possibility, that is the distribution of noise variable may be different. In the NOTEARS framework, the only possibility is that the Gaussian noise may have different mean and variance in different datasets. This setting is in fact quite restricted. 
strength_and_weaknesses: Weakness: 

1. The problem setting is quite restricted.
2. There is no theoretic guarantee that the proposed algorithm will find a set of consistency DAG.
3. The experiment is quite weak. Instead of only using nonlinear models, the authors may also add some linear experiments, where the DAG is known to be identifiable. 
clarity,_quality,_novelty_and_reproducibility: The problem statement is not very clear, and the novelty is quite limited. 
summary_of_the_review: The paper proposed a method to learn DAG from heterogeneous datasets, but the setting in the paper is quite restricted and the experiments are also weak.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper develops an extension of Notears, a recent approach for structure learning with continuous optimization, to learn transportable structures (transportable in the sense that the learned structures are the same across multiple datasets with potentially different distributions). This is achieved by incorporating a regularization term in the optimization procedure to encourage the structure parameters over different dataset to be equal to one another. Empirical results are provided to validate the proposed method.
strength_and_weaknesses: ### Strengths
- The problem studied is highly relevant, since learning transportable structures may be an importance topic especially because of data heterogeneity across different datasets.
- The paper is well written and easy to follow.

### Weaknesses
- The setting studied is very similar to (and seems to be a special case of) differentiable federated structure learning [1, 2], which should be discussed. E.g., [1] considered data heterogeneity difference (i.e. different data distribution across multiple clients, but the underlying DAG is the same), which is essentially the same as transportability in this paper. If I understand correctly, the only key difference appears to be that [1, 2] additionally considered privacy issue.
- Therefore, the resulting approach is also similar to [1, 2]. Specifically, [1] compute the average adjacency matrix in each step and enforces different clients to use the same average matrix, while [2] enforces the adjacency of different clients to be the same via ADMM. The only key difference appears to be that the proposed approach uses a soft regularization scheme to *encourage* the same structure, while [1, 2] used a hard scheme to *enforce* the same structure.
- Following the comment above, although adding a soft regularization scheme encourages the same structure, it does not guarantee/enforce it, which I think might be a limitation (in practice the participating clients may want to learn a shared structure that is the same). This is also acknowledged by the experiment in Sec. 4.
- Adding a soft regularization directly over the parameters $A$ (which correspond to both structure and model parameters) may lead to the wrong structure. I.e., this regularization may conflict with the original least squares loss of the specific dataset when the data distributions are different. In my opinion, the backbone method Mcsl used by [1] might seem like a better choice (as compared to Notears-MLP) because it decomposes the structure and model parameters, and so the proposed regularization can be applied w.r.t. the structure parameters only.
- I did not manage to find experiments of different datasets having different data distributions in Sec. 4, which I think are much more relevant (considered by [1]), since that's the key motivation of using transportability (Definition 1) or invariance.
- The considered setting is highly similar to (invariant) causal discovery from multiple domains or heterogenous data, e.g., [3, 4, 5], which should be discussed and compared.

### Minor/Other Comments
- Sec 2.2: $\rho$ and $\lambda_2$ are not "hyperparamers", but rather the parameters of augmented Lagrangian method that will be updated during the optimization process. 
- Sec. 2.2 "One can traverse G_X smartly to arrive at a DAG much faster": There was no guarantee that DSF arrives at a DAG in (Zheng et al, 2018); this was proved by [6, 7] and should probably be mentioned. Also, to the best of my knowledge, Zheng et al. (2018) did not claim/demonstrate that Notears runs "much faster" than CIT-based methods like PC, and I would encourage the authors to add a reference regarding this.
- For Notears-MLP, is the regularization scheme applied to the weighted adjacency matrix constructed w.r.t. the MLP weights, or all weights of the MLPs? Alg. 2 seems to indicate the latter, but the former sounds more intuitive to me.
- The efficiency gain in Fig. 5 is interesting and surprising. Is it because the for loop in fifth line of Alg. 1 can be run in parallel?

1. Federated causal discovery, 2022.
2. Towards Federated Bayesian Network Structure Learning with Continuous Optimization, 2022.
3. Multi-domain Causal Structure Learning in Linear Systems, 2018.
4. Causal inference using invariant prediction: identification and confidence intervals, 2015.
5. Causal Discovery from Heterogeneous/Nonstationary Data, 2020.
6. On the Convergence of Continuous Constrained Optimization for Structure Learning, 2022.
7. DAGs with No Fears: A Closer Look at Continuous Optimization for Learning Bayesian Networks, 2020.
clarity,_quality,_novelty_and_reproducibility: Details are provided in the comments above.
summary_of_the_review: - A number of highly related works were not discussed and compared to. Specifically, the proposed method is similar to [1, 2].
- The proposed method adds a regularization term w.r.t. both structure and model parameters, which may lead to the wrong structure in practice especially when the data distributions are different for different datasets.
- The paper does not consider experiments of different data distributions for different datasets, which may be more common in practice and is the key motivation of transportability/invariance.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The authors consider the problem of learning DAG from heterogeneous datasets that share the same DAG, called transportability, in the causal inference literature. 

Overall, the problem is real and significant, and it's commendable the authors attempted to solve such a difficult task. Also, I appreciate the goal of finding a differentiable approach to the problem, which is usually solved through non-neural approaches. On the other hand, reviewers found the particular setting restricted, the method with no theoretical guarantees and/or incremental, and the experiments somewhat weak. The reply was appreciated but was not strong enough to revert some of the reviewers' concerns. 

Since the work is quite interesting, they should try to improve the writing, taking into account all the feedback provided. Furthermore, the data structure that is the target of the learning process in this paper is an object already understood in the literature under the name of selection diagram (Pearl & Bareinboim,  (2011). As I understand, the study of transportability has been introduced in various papers by them (e.g.,  2013, 2016), and readers familiar with this literature may wonder whether the constraints implied by the causal system (formally, the selection diagram) in the data are encoded and leveraged by the method proposed in the paper. Is the proposed method complete (necessary and sufficient) relative to these constraints? Subsumed? More general? Making this connection could shed some light on the theoretical properties of the proposed work. Finally, other works have leveraged invariances across environments and proved their completeness (Jaber et al., NeurIPS, 2020) in some IT sense,  and it would be important to understand how these methods differ. 
justification_for_why_not_higher_score: The paper seems to have serious problems in many dimensions, as mentioned in the discussion above.
Decision: Reject