Differentiable and transportable structure learning
Authors: Jeroen Berrevoets
Nabeel Seedat
Fergus Imrie
Mihaela van der Schaar
Venue: Submitted to ICLR 2023 (2023)
Keywords: graph learning

Abstract: Directed acyclic graphs (DAGs) encode a lot of information about a particular distribution in its structure. However, compute required to infer these structures is typically super-exponential in the number of variables, as inference requires a sweep of a combinatorially large space of potential structures. That is, until recent advances made it possible to search this space using a differentiable metric, drastically reducing search time. While this technique– named NOTEARS –is widely considered a seminal work in DAG-discovery, it concedes an important property in favour of differentiability: transportability. To be transportable, the structures discovered on one dataset must apply to another dataset from the same domain. In our paper, we introduce D-Struct which recovers transportability in the discovered structures through a novel architecture and loss function, while remaining completely differentiable. Because D-Struct remains differentiable, our method can be easily adopted in existing differentiable architectures, as was previously done with NOTEARS. In our experiments, we empirically validate D-Struct with respect to edge accuracy and structural Hamming distance in a variety of settings.

Under review as a conference paper at ICLR 2023
DIFFERENTIABLE and TRANSPORTABLE STRUCTURE LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Directed acyclic graphs (DAGs) encode a lot of information about a particular distribution in its structure. However, compute required infer these structures is typically exponeexponetiall in the number of variables, as inference requires a sweep of a combinatorially large space of potential structures. That is, until recent advances made it possible to search this space using differentiable metric, drasti- cally reducing search time. While this technique- named NOTEARS -is widely considered seminal work in DAG-discovery, it concedes an important property in favour of differentiability: transportabiliiy. To be transportable, the structures discovered one dataset must apply another dataset from the same domain. In our paper, we introduce D-Struct which recovers transportability in the discov- ered structures through novel architecture and loss function, while remaining completely differentiable. Because D-Struct remains differentiable, our method be easily adopted in existing differentiable architectures, done with NOTEARS, In our experiments, we empirically validate D-Struct with respect to edge accuracy and structural Hamming distance in variety of settings.
previously
INTRODUCTION
Machine learning has proven to be crucial tool in many disciplines. With successes in medicine [1-5], economics [6-8], physics [9-14], robotics [15-18], and even entertainment [19-21]. machine learning is transforming the way in which experts interact with their field. These successes are in large part due to increasing accuracy of diagnoses, marketing campaigns, analyses of experiments, and forth. However, machine learning has much more offer than improved accuracy alone. Indeed, recent advances seem to support this claim, as machine learning is slowly recognised as tool for scientific discovery [22-25]. In these successes, machine learning helped to uncover previously unknown relationships between variables. Discovering such relationships is the first step of the long process of scientific discovery and are the focus of our paper as D-Struct- the model we propose in this paper aims to help through differentiable and transportable structure learning,
The structures. We focus on discovering directed acyelic graphs (DAGs) in domain X. DAG helps us understand how different variables in X' interact with each other. Consider a three-variable domain X {X, Y, 2), governed by a joint-distribution, x- A DAG explicitly models variable where G depicts IP as a DAG. Such a DAG allows useful analysis of dependence and independence of variables in Px [26, 27]. From G, we learn that X does not directly influence Y, and that X 1L Y as X does not give any additional information on Y once we know Z. While DAGs are the model of choice causality [28), impossible to discover a causal DAG from observational data alone [29-32]. As we only wish to assume access to observational data, our goal is not causal discovery.
interactions in Px. For example, consider the following DAG: G =
Z
The above forms the basis for conventional DAG-structure learning [33]. In partieular, X JI YIZ strongly limits the possible DAGs that model Px. Given more independence statements, limit the potential DAGs further. However, independence tests are computationally expensive which is problematic as the number of potential DAGs increases super-exponenntilly in |22'] [34].
This limitation strongly impacted the adoption of DAG-learning, until Zheng et al. [35] proposed NOTEARS which incorporates differentiable metric evaluate whether not a discovered struc- ture is DAG [35, 36). Using automatic differentiation, NOTEARS learns a DAG-structure a much more efficient way than earlier methods based on conditional independence tests (CITs).

---

Under review as a conference paper at ICLR 2023
Population of patients.
Different hospitals.
Different datasets.
The same DAGs.
These DAGs are transportable!
Different
distributions.
Figure 1: Transportability in DAG discovery. Different patients go different hospitals (left), yet we wish to infer general structure (right) across hospitals. A structure can only be considered discovery if generalizes in distributions over the same domain. For example, the way blood pres- sure interacts with heart disease is the same for all humans and should be reflected in the structure.
While NOTEARS makes DAG inference tractable, we recognise an important limitation in the ap- proach: a discovered DAG does not generalise to equally factorisable distributions, i.e. NOTEARS not transportable. While we explain why this is the case in Section 2.2 (and confirm it empirically in Section 4), we give a brief description of the problem below, helping to state our contribution.
Transportability. Consider Fig. 1, depicting two hospitals: t and Pil, named hospitals A and onward. Each hospital hosts patients described by the same set of features such as age and gender. However, the hospitals may have different patient-distribution
e.g. patients in A are older compared to B. Bul their underlying biology remains the same. Using NOTEARS to learn DAG from data on hospital A, actually does not guarantee the same DAG is discovered from data in hospital B.
Learning from multiple data-sources is not new. In particular, papers focusing on federated structure learning solve similar objective as described above [37, 38.. However, we believe transportability more general property than only training from multiple data sources. Crucially, transportabilityy is very explicit about the domains we learn from, allowing their distributions to vary across domains.
Interestingly, despite computational limitations, transportabilityy is actually guaranteed when using a CIT-based discovery method [39, 40], assuming that patients in both hospitals exhibit the same (in)dependencies in X. Being unable transport findings across distributions is major shortcom- ing, as replicating a found discovery is considered hallmark of the scientific method [41-44].
Contributions. In this paper, we present D-Struct, the first transportable differentiable structure learner. Transportability grants D-Struct several advantages over the state-of-the art: D-Struct is more robust, even in the conventional single-dataset case (Table I); D-Struct is fast, in fact, we report time-to-cone convergence often up to 20 times faster than NOTEARS (Fig. 5); and given its completely differentiable architecture, D-Struct is easily incorporated in existing architectures (e.g. [45 (4)).
While transportable methods have clear benefits over non nonttranspo methods in settings with multiple datasets (as illustrated in Fig. 1), we emphasise that our method not limited these settings alone. In fact, we find that enforcing transportability significantly increases performance in settings with one dataset, which arguably most common. In Section we introduce D-Struct and how to use our ideas in the single dataset setting. We then empirically validate D-Struct in Section 4.
2 PRELIMINARIES AND RELATED WORK
Our goal is to build a transportable and differentiable DAG learner. Without loss of generality, we focus our discussion mostly on NOTEARS [35] (and refinements [36, 50-52)) as it is the most adopted differentiable DAG learner. For a more in depth overview of structure learners (CIT-based as well score-based), we refer to Appendix G or relevant literature [26, 28, 34]. First, we formally introduce transportability, and then explain how NOTEARS works and why not transportable.
2.1 TRANSPORTABILITY
Factorisation and independence. Consider a distribution, Px, which we can factorise into,
II P Px
with E [d]. where [d] = 1,...,d, d, and X; representing the 2th element in X. Eq. (1) may get quite long with increasing the conditions may contain up to d different variables. This becomes
2

---

Under review as a conference paper at ICLR 2023
restrictive when estimating and decomposing Px from data. Instead, simplify eq. (1) using independence statements, c.g. X T X3. invokes the equality: Px,,,,,. = Px A simplified version of eq. (1) translates into a smaller Markov blanket and boundary [53] (see Appendix D).
Non-symmetrical statements. Contrasting the above, we are interested in directed and acyelic graphical (DAG) structures. Let Ox = {X, E be DAG, where & C X x X is set of edges connecting nodes in X' (i.e. random variables), with (g) E & implying (2,, X,) & & [54].
While independence is symmetric, it is still possible to infer non-symmetric structures with only independence statements using d-separation [28, 30, 55-58J. Given a collection of conditional in- dependence statements, which renders two variables independent given third, e.g. X 1 Y d-separation (defined in Def. 4 in Appendix D) helps us narrow down a directed structure from inde- pendence statements [59, 60]'. If set Xy blocks all paths between A and B, they are d-separated, d-sep, (A; B)),),, where blocking is directly related to independence between A and B.
With d-separation and the common faithfulness assumption (see Appendix G), we have link be- tween 9x and Fx. Specifically, the conditional independence relationships in Gx have a one-to-one correspondence with those x [61], i.e. if 1 p Y Z then X IL g 2, where IL s denotes independence in S. The reverse is not necessarily true there can be many graphs that correspond with PP in terms of (in)dependence- these graphs are termed Markov equivalent [26].
The set of conditional independence assertions in IP is denoted as I(P).. Similarly, all indepen- dence statements implied by d-separation in a graph G are denoted as I(g) = {(X' 1 Bl(aa) : d-sepg (A; B) referred to the set of global Markov independencies [26, Chapter 3].
Invariance and discovery. Consider two datasets: D1 (x(m) EX: : [N]] and D2 (X(m) X n E [M]] spanning the same space X. As a sample x()) from Di depicts the same variables sample ((m) from D2, both datasets reflect the same underlying mechanisms. For example, if hospital A collected a dataset on its patients in A' (say D)) and associated smoking with cancer, then if this statement is true this should also be found in data collected by hospital B (D2).
Of course, while the samples in DI and D2 come from the same domain X, they may be sampled from different distributions, IPL and IP respectively. Recall Fig. where hospitals A and B may be located in different regions, resulting in the characteristics of patients to also be different. However, key in a scientific discovery is that it generalises beyond distributions and is carried over through the entire domain X. In other words, any structure we may find in D,, should also be found in D2, as for almost all distributions X E P that factorise over G I(Px) = I(G) I(Px) where Px + P2 [26, Theorem 3.5]; if this is not the case, we haven't 't discovered anything at all.
Definition 1 (Transportabilitty. With multiple datasets, {Dr Pt k [K]} over the same
domain X, sampled from potentially different distributions IP's # Pix if if for all i, [K], we call a method transportable if it learns a structure that is the same across all datasets: (Dr + Gk GK.
k E [K] s.t. G1
Def. 1 states that when a DAG found in D2 is also found in D2, we consider that DAG, and the method proposing it, transportable. Transportability in CTT-based methods is only satisfied when we assume that each distribution contains the same global Markov independencies. This assumption not strict all, as we are only concerned with distributions that span the same domain X, and thus inherit the same interactions as posed by X, i.e. the discoveries to be made. As such, CIT-based methods learn transportable DAGs automatically as transportabilitty a property directly related to the set of independencies of both distributions and DAGs, which we assume to be the same.
From the above, is clear that transportabilitty a property of the structure learning method itself. In the case of CIT-based methods, we are guaranteed transportability in our setting, but not so for differentiable structure learnings. However, given that the method responsible for transportability, we can come up with ways to include this property in differentiable structure learners also.
I Note that in these earlier works, DAGs were named influence diagrams. For all distributions except for set of measure zero in the space of conditional probability distribution parameterizations [39].
3

---

Under review as a conference paper at ICLR 2023
LDSF (g))
Subset Construction (Section 3.2)
M (9,,..., 9k)
D1
9
{IP ~ P
D2
92
D
G
Forward pass
Db
9k
Backward pass
Composes
Transportable Structure Learning (Section 3.1)
Figure 2: D-Struct architecture. Above architecture enforces transportabilityy. D-Struct is split in two major parts: subset construction (Section 3.2) and the transportable structure learning algorithm (Section 3.1). There are three components: P, LDSF. and LMSE The losses are combined and backpropagated through the architecture. Lastly, all DSFs are merged into final DAG structure G.
2.2 DIFFERENTIABLE STRUCTURE LEARNING
ClT-based methods evaluate each possible DAG that spans X' with respect to its factorisation over Px using I(G Gx), where Gx denotes the space of all possible DAGs in the domain X. The major issue with this is computation. Essentially, there are two aspects which negatively impact computation time: first, the number of to-be-evaluated DAGs in Gx increases super-exponentially in |X| (e.g. 10 variables result in > 4 x 1 possible DAGs [34, 62, 63]); second, simply recovering I(Px) evaluate each G E Gx requires many independence tests, each with additional compute. Appendix G includes an overview the most well known ClT-based (and score-based) methods.
Differentiable score functions. Enter differentiable score functions (DSFs). With DSFs can traverse Gx smartly to arrive at DAG [64, 65] much faster [63]. Furthermore, a differentiable method is straightforward to include variety of differentiable architectures, allowing joint opti- misation of both the graphical structure as well as the accompanying structural equations or another downstream use.
Most notable is NOTEARS by Zheng et al. [35], which proposes the following learning objective:
min F(A) + A11 + (I((A) A2h(A), AEA
where Raxd (transformed to binary with threshold) denotes an adjacency matrix; F(A) is likelihood based loss (like the MSE); p and λ1 are parameters of the augmented Lagrangian; and
h(A)= == tr(exp(A A)) - d,
is the actual differentiable score function, where tr(-) is the matrix trace operator, and is the element-wiss (Hadamard) product. Importantly, h(A) = O indicates A is a DAG. Considering that eq. (3) is differentiable, can take its derivative with respect A and minimise eqs. (2) and (3).
As is always the case for gradient-based learning, the random initialisation of A may guide the optimisation in different directions, potentially arriving a local minimum in the case of non-convex targets. The latter is certainly the case in more recent improvements of NOTEARS as they almost exclusively focus on non-linear structural equations which result non-convex losses [36, 51, 52].
Transportability of DSFs. Current DSFs are not transportable, due to eq. (2) having conflicting solutions- contrasting the single solution (set) that transforms ((P) to G. Essentially, the approxi- mate nature of (stochastic) gradient- based learning can result in conflicting estimate structures given different datasets. In fact, other works recognise this [66]. Rather than trying to recover one true DAG, their objective is to learn multiple (potentially different DAGs) through multi-task objective. In our work, we assume there does exist one (unique) DAG, which is the one we aspire to recover.
D-STRUCT: DIFFERENTIABLE AND TRANSPORTABLE StrucTURE LEARNING
The goal of any structure learner (differentiable or not) is to transform finite data into structure:
D g,
4

---

Under review as a conference paper at ICLR 2023
Sample according to
Results in
D D2
X
D
P
(a) We presented with (b) With two distributions Px (c) Sampling according the dataset over the domain X.
and Fx, we can sample from D.
two distributions results two
subsets Dj U UD2 D.
Figure 3: Differently distributed single-origin data. On the left we illustrate a single-origin dataset
D, sampled from one distribution. In the middle we illustrate two distributions over the domain of D. which are used to resample two subsets from D, thereby creating a new multi-origin datasource.
and so is the case for D-Struct. We introduce D-Struct in Section 3.1 and provide implementation details using NOTEARS in Section 3.3. While transportability seems most natural in setting with multiple datasets (e.g. data from two hospitals illustrated in Fig. 1), we show in Section 3.2 how one can leverage transportability in the setting where there is only dataset. In fact, our experiments in Section 4 show that D-Struct consistently offers more accurate DAGs in this "single-origin" setting.
3.1 D-STRUCT: TRANSPORTABLE STRUCTURE LEARNING
In order enforce transportability. D-Struct employs an weeemble-architectuee of multiple initialisa- tions of chosen DSF and their appropriate (differentiable) architecture. Each loss is then combined with regularisation function based on the D-Struct architecture. Fig. 2 depicts this architecture and highlights how our regularisation scheme is backpropagated throughout the entire network.
Given datasets Di..... DR. , we can use any DSF (e.g. [35, 36, 50-52)) to learn a DAG. Specifically, we let K distinct DSFs learn a DAG from one of the K datasets, agnostic from each other. We consider these learning objectives be K parallel objectives, as is also illustrated in Fig. 2 in the rightmost part. Crucially, D-Struct does not restrict which type of DSF we can use. In linear setting one can use vanilla NOTEARS [35], whereas in a non-linear setting, one can use the non-parametric version [36]. Naturally, any restriction posed by the chosen DSF will carryover to D-Struct. We use NOTEARS- MLPP [36] in Sections 3.3 and 4, while Appendix A includes pairings with other DSFs.
At this point, we identify first loss term: LDSF(GR), which depends on the chosen DSF (illustrated in red in Fig. 2). In the case of NOTEARS, (Gs) corresponds with eqs. (2) and (3). Whenever data is passed through the architecture- without mixing distinct datasets structure as LOSF (9 X ~ Dz). where X C D. If the chosen DSF requires hyperparameters (such as λ1,2 and p eq. (4)), we have also include these in D-Struct's set of required hyperparameters. While it is possible to set different hyperparameter values for each of the DSFs separately (which is potentially helpful when there is lot of variety in the K distinct datasets), we fix these across DSFs in light of simplicity. A discussion on D-Struct's hyperparameters can be found in Appendix A.1.
evaluate the discovered
[K]] we enforce transportability across each D3 by comparing the structures I....., Gr. We do this by calculating the difference of the adjacency matrices A& E Rd Specifi- cally, for each gradient calculation (before we perform backward pass), we take the (element- wisee) mean adjacency matrix, ALK R E. As. detach it from the gradient and backpropagate the MSE for each parallel DSF. In particular, we include the following regularisation term in D-Struct's loss:
Given (CD)
CMSE (As) == ||A 1
Minimising eq. (4) results in transportable structures (see Theorem 1), Note that eq. (4) (green in Fig. 2) remains differentiable, which was our goal for D-Struct. We add (MSE(Gx) to the DSF loss,
((G.lP)) CoSE (g )+ + aAMsE(A(Ge)),
(5)
where A(G) indicates the adjacency matrix of G, and is scalar hyperparameter (refer to Ap- pendix A. for hyperparameter settings, details, and further insights). Note that the second term in eq. (5) does not depend on De. Having LMSE be agnostic to the data makes sense transportability
5

---

Under review as a conference paper at ICLR 2023
is not property of the data. Indeed, recall from Section 2.1 that transportability is a property of the structure learner instead. This additional term (based solely on the parallel learners) enforces transportability as the architecture encourages the DSFs to converge to the same adjacency matrix.
Theorem (Minimising eq. (4) yields transportable structures.). The only way in which eq. (4) is equal to O- for every adjacency matrix A -is when A =... Ag- Even a slight difference in one of the Ag will result in non-zero equation 4 as ALKK will be affected by this difference. Having every A1 = =... Ag and thus equal structures in Gk- where each Aş. is learned from distinct Da- corresponds with transportable structures as we have defined in Def.
3.2 D-STRUCT: SUBSET CONSTRUCTION
So far, we have only discussed the rightmost part of Fig. 2. In Section 3.1, we assumed data is provided in multiple distinct datasets, i.e. they stem from multi- origin datasource. However, here we explain how even in the single-origin case D-Struct is applicable, irrespective of which DSF we end up choosing. As such, we continue our discussion by explaining the leftmost part of Fig. 2. Naturally, if one already has distinet Dr ~ Tpk, this step can be skipped and D-Struct can be used as in Section 3.1.
Selected indices:
10
IPT P
Different distributions will guide each (distinct) optimi- sation target into different direction. Combining their results, will encourage the total model to be more ro- bust and generalisable. However, while multi-origin datasource may be governed by multiple distributions, a single-origin one not. Our task clear: from single- origin datasource, we have to mimic multi-origin data- source in such a way that we know each subset has a dif- ferent distribution, yet maintains the properties of the original single-origin-distrine global Markov independencies). Doing so allows us to enforce transportability through eq. (4).
Figure 4: K distributions. We have il- lustrated the subset sampling with beta- distributions above, for K 3. For each density, we evaluate its PDF for each index, normalize it and perform Bernoulli experiment. The selected in- dices are plotted below the density func- tions (black indicates a selected index). (such the
The lefthand side of Fig. 2 shows that we handle the single-origin case by transforming it multi-origin case. With this constructed multi origi- setup. we can continue using D-Struct as we have done in Section 3. We preface the multi-origin case with a step that divides D into subsets (DI,..., Dx), according to different distributions P == (IPx...., } Consider Fig. where we illustrate how we sample from D using pk E P. In principle, each element (() E D has a 1a (X(")) probability to be sampled from D, for each Pk E P. As such, each distribution leads to a subset Tpk x D + Dr where U₁ Dr = D, and Dj need not be disjoint but is not equal to D.
In our experiments, we perform this preprocessing step by first correlating the index of each element in D with their values in X. Next, we define K distributions over [N] and then use these distributions to sample indices. The sampled indices compose the subset. While we have included a detailed description of our implementation in Appendix F, we give a brief step-wise explanation below.
Step Correlating indices and values. Sorting and reindexing elements in according to the covariates in X ensures dependency between X and e [NV. where < j indicates X < ()),, i.c. the order of X's 's in the data structure representing D is correlated with the values of the X's.
Step 2 Distributions over [N). Step allows us to create subsets based on one-dimensional distributions { k E [K]] rather than much more complicated distributions over X. An added bonus to these die-dimensional distributions is that they easily scale to more dimensions in X. Of course, the number of distributions, and consequentially their shape, should change in function of K. Specifically, with higher K, we have to ensure that the probability mass of each distribution is concentrated in different areas of [NV. As such, we chose to model these as beta-distributions with,
a, B E {i,, K), (K, K), (K,j) : interp(1, 1, K - 1),5 E interp( K - 1,1)),
where interp(a, b) is a linear interpolation between a and b, used to sample _ i's and j's. When Ki is even we leave out (K, K) such that the total number of distributions always equals K.
Step 3 Selecting indices.
Our final task is to create K subsets, which due to the first step is

---

Under review as a conference paper at ICLR 2023
simplified choosing indices instead. These indices are selected based on the distributions defined in the second step. First, we evaluate each density's s PDF for every index (after normalisation: x), and normalise the output to be a value between 0 and 1. Once have K values for each index, perform a Bernoulli experiment where the output determines whether or not the index is selected to be part of the subset & [K]. This process is illustrated in Fig. 4 for K 3 using beta distributions.
With each subset, we apply D-Struct as explained in Section 3.1. In our experiments (Section and Appendix A), show that D-Struct greatly improves performance of non-transportable DSFs. Furthermore, we also empirically validate our subsampling routing compared to random sampling.
3.3 EXAMPLE IMPLEMENTATION USING NOTEARS
D-Struct works with any DSF, though is instructive to illustrate this with an example. For this, we chose NOTEARS-MLE [36] which is non-parametric (cfr. the structural equations) extension of the classic NOTEARS paper [35]. The main challenge to incorporating D-Struct into HOTEARS-MEN integrate into its dual ascent strategy which solves the (non-convex) constrained optimisation problem in eq. (2) [50], solved with an augmented Lagrangian method [67, Chapter 5J.
The constraint in the optimisation problem can only contain zeros [35, 36, 50J. NOTEARS (and its extensions), solve this problem by using the L-BFGS-B optimizer [68] which can handle parameter bounds out-of-the-box, making a suitable choice to optimise the augmented Lagrangian', This is made explicit in Algs. and 2.
from, for example, knowing that the diagonal of A
Init.: 02 for each & [K]
Input: 0z, batch
Input: hool Pmax
while Pmax do
Setup: h < 0C, %i......K K t 1, 1
Im LMSE (01,...,................. Ok):
for maximum amount of epochs do
la + LUS (batch::
for k [K] do
0 t BL--BFGS-BE update (mm 4m;
for batch ~ D do training.step (0k, batch); ;
h' + h(A(0x));
if h > 0.25h then
h max; h(A(O:));
Ph +- 10px:
p ming Ps:
else break;
Algorithm 1: Outer-loop of dual ascent procedure for D-Struc(NNOTEARS -MPP)
Algorithm 2: training.step for D- S-Stuuc(NNOTEARS -MLP)
Algorithms 1 and highlight where D-Struct requires edits in the standard NOTEARS-ME -MLP al-
gorithm. Most obvious is the creation of multiple parameters, 0% for each k [K]. where each 0g indicates the set of parameters for one initialisation of NOTEARS-MLP, exactly following the architecture depicted in Fig. 2. The set (0,, le E []] then denotes the parameters for D-Struct. As such, the number of parameters for D-Struct scales linearly K, compared to the used DSFs.
From Algs. and 2, learn that information across the different NOTEARS-MLE3 is shared in ra wwwningnseeeee (corresponding with Alg. 2). Typically, training step is solely focused on one structure learner leaving the learner unaware of the other DSFs, is also implied in Alg. which it- erates over each learner separately. The sharing of information across each learner though- through CMSE(O)..... (x) computed in the first line in Alg. 2's while loop iis enforcing transportability.
D-Struct hardly increases implementation complexity. In fact, besides architectural alterations (as explained in Section 3.1 and Fig. 2), the optimisation strategy mostly adopted from the underlying DSF. This is an important advantage. In fact, Zheng et al. [35] already state the importance of an easy to implement model; they use approximately 60 lines of code, and we only add approximately 10 lines that. Furthermore, we also noticed a major improvement in efficiency as D-Struct drastically reduces computation time compared to NOTEARS. We report computation time the next section.
TThis also allows including prior knowledge on I(P). We discuss this in detail in Appendix

---

Under review as a conference paper at ICLR 2023
Table I: Results on Erdos-Renyi (ER) graphs. First block: We sample ten different ER random graphs, and accompanying non-linear structural equations as in Zheng et al. [36]. From each system we then sample varying number of samples, and evaluate NOTEARS-ML. with D-Struct (indicated as "V") and without D-Struct (indicated as "X"). Second block: For each row we sample new ER graph with varying degree of connectedness (s indicates the expected number of edges). In both cases, we report the average performance in terms of SHD, FPR, TPR, and FDR, with std in scriptsize. Unless otherwise indicated, n = 1000, d = 5, K = 3, = 2d.
metric
SHD (4)
FPR (1)
TPR (1)
FDR (L)
D-Struct
X
/
X
varying sample size
200
3.60
4.20
2.00
4.20.10.4
0.67
0.64.10.5
0.25
0.42.10.04
500
3.94 3.20
1.244
3.94
0.66110.2
0.56.10.04
0.18:0.00
0.44.10.04
1000
2.75=00.7
3.67+0.82
1.00
2.67+0.63
0.75+0.08
0.63+10.13
0.18=0.03
0.39+0.11
2000
2.66=0.80
3,54
1..6
2.09+0.31
0.
0.75=+0.03
0.27=0.07
0.33=+0.00
varying graph connectedness
0.5d
1.75
7.33:10.13
0.50
1.05:10.02
1..31
0.88
0.42
0.73:10001
1d
3.80
7.67::0.45
0.55 00.22
1.53:10.99
0.75:10.09
0.46:10.09
0.40
0.77:10.07
1.5d
3.00
5.67
1000
1.55
0.89
0.62:10.06
0.32+10.0
0.53++004
2d
2.28:10.0
3.67+1..2
1.00
2.67:1063
0.67
0.70:10.09
0.11
0.32.+0.88
d
varying dimension count
5
2.28=
3.67+:082
1.00
2.67:4063
0.67
0.70=
0.11
0.32-+0.08
7
8.67=
12.9:0.15
0.72
1.07
0.96=2002
0.83++0011
0.49
0.63++001
10
19.71
30.8.10.8
0.42
1.1810004
0.70
0.71
0.34
0.70.0.02
EXPERIMENTS
Recall from Section that D-Struct's objective is exactly the same as any differentiable structure learner: transform dataset into DAG, whilst remaining differentiable. With D-Struct, our aim to increase performance of any DSF by enforcing transportability on the learner's outcome structure. As such, the most pressing questions are: (1) Are the discovered structures transportable?. (2) Does D-Struct improve existing learners?, (3) Does D-Struct give up on efficiency?, and (4) Do really need our subsampling routine? We answer these questions one-by-one below with experiments.
(1) Transportabilitt. Before testing accuracy, we first empirically confirm that NOTEARS is not transportable while D-Struct is. We compare NOTEARS with D-Struct using 1000 samples drawn from an Erdos-Renyi (ER) random graph, and split the samples into two equal sized subsets. We evaluate the structural Hamming distance (SHD) between the graphs learned by NOTEARS on each dataset, and the same for the internal graphs learned by D-Struct. The DAGs learnt by D- Struct are perfectly transportable (SHD= 0) in 8/10 runs (mean SHD 0.46 + 0.27), with only minor discrepancies in the other cases. Conversely, NOTEARS has mean SHD of 1.14 + 0.20, only displaying transportability in 2 cases. Similar results for other DSFs are reported in Appendix A.
(2) Accuracy. The most straightforward way to see D-Struct is better is by repeating the experiments in Zheng al. [36]. We report only subset of our outcomes in the main text, mainly on D-Struct's improvement over NOTEARS -LLP. However, more metrics and exper- iments on different DSFs can be found in Appendix A. In Table we report the false positive rate (FPR), true posi- tive rate (TPR), false discovery rate (FDR), and structural Hamming distance (SHD) of the estimated DAGs using data sampled from different ER random graphs with vary- ing sample size (n), expected number of edges (s), and dimension count (d). In all cases we find that D-Struct significantly improves NOTEARS-MLP (other DSFs in Appendix A). A similar conclusion can be drawn from Fig. 6, where we report the SHD for more parameters and data from Erdos- Erdos-Renyi as well as Scale Free graphs [35].
25x
20x
Speedup
15x
10x
5x
200 500 1k 2k
Number of samples (n)
Figure 5: Speedup of D-Struct over NOTEARS. Difference in computation time between NOTEARS- - and D- Struct, as function of n. On average, D-Struct is about 10x quicker.
8

---

Under review as a conference paper at ICLR 2023
varying n
varying 8
varying
varying K
Erdos- Renyy
20
10
200
500
1,000
2,000
0.5
1.5
10
K
300
Scale Free
SHD
6
20
4
10
200
500
1,200
2,000
0.5
1.5
10
Figure 6: Structure recovery. We report the SHD (4) compared to the true graph. We report perfor- mance as a function of four different parameters (changing the properties of the task). The results demonstrate that D-Struct outperforms NOTEARS MLP in all these settings. Additional results are reported in Appendix A. Unless otherwise indicated, 1000, d = 5, K 3, = 2d.
(3) Computational efficiency. In Fig. we learn that despite its parallel ensemble architecture, D-Struct is actually much faster than NOTEARS. Note that D-Struct is built on top of NOTEARS, meaning this computational gain not due differences in implementation. Instead, we believe computation gains are largely due to D-Struct's learning scheme. Rather than using the entire dataset at once to learn one (computationally intensive) DSF, D-Struct splits the data and learns multiple DSFs from several smaller datasets. We believe this is important result: the whole reason for having differentiable structure learners is due to their efficiency gains over CaT-based methods.
(4) Subset construction. A final property we wish to validate is the need for sampling K different subsets us- ing our subsampling routine from Section 3.2. This is an important validation it shows that D-Struct does not only gain in performance due to its ensemble architecture. For this, we compare D-Struct's performance with and without our subsampling routine. Using D-Struct without our subsampling routine amounts to providing K random splits, rather than carefully sampling K distinct D₂ ~ pk Table 2 shows that our subsampling routine does improve D- Struct's 's performance as expected, validating our goal to explicitly optimise for transportable structure learners.
Usefulness of our subsam- pling routine. We sample ten different ER random graphs like in Table 1. From each system we then sample n 2000 samples, and evaluate NOTEARS-MLP with our subsampling routine from Sec- tion 3.2 (indicated as "V") and without the subsampling routine, using random splits instead (indicated "X"). For each row we repeat our experiment with different K. We report the average (and std) performance in terms of the SHD.
Table 2:
We believe that these experiments offer a broad view of how D-Struct and transportability can help us create use- ful structure learners. However, as we were unable to in- clude all our experimental results in the main text of this paper, we refer the interested reader to our Appendix A for completed and additional validation. In our appen- code
metric
SHD (1)
Subsample
X
K
varying amount splits
dices also include anonymous link repository, encouraging readers to reproduce our results.
2
2.80=0.53
3.40:1055
3.00=0.37
4.00::0.39
5
2.80
1.40.11.29
DISCUSSION
D-Struct advances differentiable structure learning by introducing transportability, property guar- anteed by CIT-based methods. We believe D-Struct can have positive impact on architectures and problems relying on differentiable structure learners, well as on general scientific data analysis.
Relating DSFs to causality. As pointed out by Kaiser and Sipos [69] and Reisach al. [70], DSFs often wrongly used to recover causal DAG. While DAGs are indeed the model of choice to describe causal interactions, there is currently no guarantee that a DAG discovered using any DSF can be identified (and thus used) as such. With this wish to state explicitly that DSF's output is not to be interpreted as a causal model. We discuss more about this particular topic in Appendix B.
Future work. The inability recover causal structure is consequence of there existing many more useful properties stemming from a baseb approach (multiple books concern this very topic [26, 53, 71, 72]). Bridging the gap between these methods is clear path forward, hopefully

---

Under review as a conference paper at ICLR 2023
increasing differentiable structure learners' potential even further. Specifically, using structure learn- ers to uncover a causal structure from observational data requires stricter assumptions. As such, one particularly interesting avenue of future work is to allow DSFs (not only D-Struct) to adhere to some of these assumptions and use them to guarantee causal discovery, taking DSFs to a next level.
Finally, D-Struct is only the first step in the process of scientific discovery. As D-Struct (or any other DSF) suggests a link between variables, is still the scientist's task to confirm this link in the lab.
Ethics Statement. We envisage D-Struct sise the discovered structures and links would need to be further verified by a human expert or in an experimental setting. Furthermore, the data used in this work is synthetically generated from given random graphical models, and no human-derived data was used.
tool to help the scientific endeavour, however empha-
Reproducibility Statement. To ensure reproducibilityy we include experimental details in Ap- pendix A. These details include: (1) hyperparameter settings, (2) evaluation metrics, (3) the synthetic data generation procedure, and (4) additional implementation details of D-Struct. Finally, all code is readily available at our anonymous online code repository: https: : /anonmoous 4open, , sci ence r/d-//d- struct. Beyond documentation and instructions, this code includes benchmark models, synthetic data generation, and our D-Struct implementation.
REFERENCES
[1] Rohan Bhardwaj, Ankita R Nambiar, and Debojyoti Dutta. A study of machine learning in healthcare. In 2017 IEEE 41st Annual Computer Software and Applications Conference (COMPSAC), volume 2. pages 236-241. JEEE, 2017.
[2] Jeroen Berrevoets, James Jordon, loana Bica, Mihaela der Schaar, al. Organite: Opti- mal transplant donor organ offering using an individual treatment effect. Advances in neural information processing systems, 33:20037-20050, 2020.
[3] Mihaela van der Schaar, Ahmed M Alaa, Andres Floto, Alexander Gimson, Stefan Scholtes, Angela Wood, Eoin McKinney, Daniel Jarrett, Pietro Lio, and Ari Ercole. How artificial in- telligence and machine learning can help healthcare systems respond to covid-19. Machine Learning, 1 10(1):1-14, 2021.
[4] Alvin Rajkomar, Jeffrey Dean, and Isaac Kohane. Machine learning in medicine. New England Journal of Medicine,
2019.
[5] Jeroen Berrevoets, Ahmed Alaa, Zhaozhi Qian, James Jordon, Alexander ES Gimson, and Mi- haela Van Der Schaar. Learning queueing policies for organ transplantation allocation using interpretable counterfactual survival analysis. In International Conference on Machine Learn- ing, pages 792-802. PMLR, 2021.
[6] Susan Athey et al. The impact of machine learning on economics. The economics of artificial intelligence: An agenda, pages 507-547, 2018.
[7] Susan Athey and Guido W Imbens. Machine learning methods that economists should know about. Annual Review of Economics, 11 11:685-725, 2019.
[8] Sendhil Mullainathan and Jann Spiess. Machine learning: an applied econometric approach. Journal of Economic Perspectives, 31(2):87-106, 2017.
[9] Giuseppe Carlco, Ignacio Cirac, Kyle Cranmer, Laurent Daudet, Maria Schuld, Naftali Tishby, Leslie Vogt-Maranto, and Lenka Zdeborová. Machine learning and the physical sciences. Reviews of Modern Physics, (144000450 2019.
[10] Alexander Radovic, Mike Williams, David Rousseau, Michael Kagan, Daniele Bonacorsi, Alexander Himmel, Adam Aurisano, Kazuhiro Terao, and Taritree Wongjirad. Machine learn- ing the energy and intensity frontiers of particle physics. Nature, 560(7716):41 1-48, 2018.
[11) Sankar Das Sarma, Dong-Ling Deng, and Lu-Ming Duan. Machine learning meets quantum physics. arXiv preprint arXiv: NeXXvv:/03..00 036 2019.
[12] George Em Kamiadakis, Toannis G Kevrekidis, Lu Lu, Paris Perdikaris, Sifan Wang, and Liu Yang. moysics-informin machine learning. Nature Reviews Physics, 3(6):422-444 -40, 2021.
[13] Philip G Breen, Christopher N Foley, Tjarda Bockholt, and Simon Portegies Zwart New- ton versus the machine: solving the chaotic three-body problem using deep neural networks. Monthly Notices of the Royal Astronomical Society, 494(2):2465-2470) 2020.
10

---

Under review as a conference paper at ICLR 2023
[14] Silviu-Marian Udrescu and Max Tegmark. Ai feynman: A physics-inspired method for sym- bolic regression. Science Advances, 6166)aaay2631, 2020.
[15] Jan Reinhard Peters. Machine learning of motor skills for robotics. University of Southern California, 2007.
[16] Xue Bin Peng. Marcin Andrychowicz, Wojciech Zaremba, and Pieter Abbeel. Sim-to-real transfer of robotic control with dynamics randomization. In 2018 IEEE international confer- ence on robotics and automation (ICRA), pages 3803-3810. IEEE, 2018.
[17] Ben Kehoe. Sachin Patil, Pieter Abbeel, and Ken Goldberg. A survey of research on cloud robotics and automation. IEEE Transactions automation science and engineering, 12(2): 398 409, 2015.
[18] Pieter Abbeel, Adam Coates, and Andrew Ng. Autonomous helicopter aerobatics through apprenticeship learning. The International Journal of Robotics Research, (9(13)1608-116 2010.
[19] Yanir Kleiman, Simon Pabst, and Patrick Nagle. Boosting vfx production with deep learning. In ACM SIGGRAPH 2019 Talks, pages 1-2. 2019.
[20] Dan Ring, Johanna Barbier, Guillaume Gales, Ben Kent, and Sebastian Lutz. Jumping in at the deep end: how to experiment with machine learning in post-production software. In Proceedings of the 2019 Digital Production Symposium, pages 1-5, 2019.
[21] Yi Wang. Film and television special effects production based on modern technology: from the perspective of statistical machine learning. In 2022 4th International Conference on Smart Systems and Inventive Technology (ICSSIT), pages 833- 836. IEEE, 2022.
[22] Alex Davies, Petar Veličković, Lars Buesing, Sam Blackwell, Daniel Zheng, Nenad Tomašev, Richard Tanburn, Peter Battaglia, Charles Blundell, András Juhász, et al. Advancing mathe- matics by guiding human intuition with ai. Nature, 600(7887):70-74, 2021.
[23] John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ron- neberger. Kathryn Tunyasuvunakool. Russ Bates, Augustin Židek, Anna Potapenko, et al. Highly accurate protein structure prediction with alphafold. Nature, (96(7873) 2021.
[24] Kathryn Tunyasuvunakool, Jonas Adler, Zachary Wu, Tim Green, Michal Zielinski, Augustin Zidek, Alex Bridgland, Andrew Cowie, Clemens Meyer, Agata Laydon, et al. Highly accurate protein structure prediction for the human proteome. Nature, 596(7873):590-599 5966 2021.
[25] Kiersten M. Ruff and Rohit Pappu. Alphafold and implications for intrinsically disor- dered proteins. Journal of Molecular Biology, 433(20):167208, 2021. ISSN 0022-2836. doi: https://ooi.orr/100 101660 j0t...m...2...... 1672088 URL https: /www. sciencedirectt com/ sessecieeeeeee art iicll e /pi i (0000222222220440200 Structure at Warp Speed: How Alphafold Impacts Biology.
From Protein Sequence to
[26] Daphne Koller and Nir Friedman. Probabilistic graphical models: principles and techniques. MIT' press, 2009.
[27] Sewall Wright. The method of path coefficients. The annals of mathematical statistics, 5(3): 161-215, 1934.
[28] Judea Pearl. Causality. Cambridge university press, 2009.
[29] Clark Glymour, Kun Zhang, and Peter Spirtes. Review of causal discovery methods based on graphical models. Frontiers in genetics, 10:524, 2019.
[30] Dan Geiger and Judea Pearl. On the logic of causal models. In Machine Intelligence and Pattern Recognition, volume 9, pages 3-14. Elsevier, 1990.
[31] Christopher Meek. Strong completeness and faithfulness in bayesian networks. arXiv preprint arXiv:1302.497. 2013.
[32] Frederick Eberhardt. Introduction to the foundations of causal discovery. International Journal of Data Science and Analytics, 3(2):81-9) -91, 2017.
[33] Max Chickering, Dan Geiger, and David Heckerman. Learning bayesian networks: Search methods and experimental results. In Proceedings of the fifth international workshop on arti- ficial intelligence and statisfies, 1995.
11

---

Under review as a conference paper at ICLR 2023
[34] Jonas Peters, Dominik Janzing, and Bernhard Schölkopf. Elements of causal inference: foun- dations and learning algorithms. The MIT Press, 2017.
[35] Xun Zheng, Bryon Aragam, Pradeep Ravikumar, and Eric P. Xing. DAGs with NO TEARS: Continuous Optimization for Structure Learning. In Advances in Neural Information Process- ing Systems, 2018.
[36] Xun Zheng, Chen Dan, Bryon Aragam, Pradeep Ravikumar, and Eric P. Xing. Learning sparse nonparametrie DAGs. In International Conference on Artificial Intelligence and Statistics, 2020.
[37] Erdun Gao, Junjia Chen, Li Shen, Tongliang Liu, Mingming Gong, and Howard Bondell. Federated causal discovery. arXiv preprint arXiv:2112.0355. 2021.
[38] Ignavier Ng and Kun Zhang. Towards federated bayesian network structure learning with continuous optimization. In International Conference Artificial Intelligence and Statistics, pages 8095-8111. PMLR, 2022.
[39] Christopher Meek. Strong completeness and faithfulness in bayesian networks. In Proceedings of the Eleventh Conference on Uncertainty in Artificial Intelligence, 1995.
[40] Ann Becker, Dan Geiger, and Christopher Meek. Perfect tree-like markovian distributions. In Proceedings of the Sixteenth Conference Uncertainty in Artificial Intelligence, 2000. URL https: arxiv. org/ abs/ 1110/111110 3834.
[41] Monya Baker. 1,500 scientists lift the lid on reprodueibility. Nature, 533(7604), 2016.
[42] Colin F Camerer, Anna Dreber, Eskil Forsell, Teck-Hua Ho, Jürgen Huber, Magnus Johan- nesson, Michael Kirchler, Johan Almenberg, Adam Altmejd, Taizan Chan, et al. Evaluating replicability of laboratory experiments in economics. Science, 351(6280) 2016.
[43] Robert K Merton. The sociology of science: Theoretical and empirical investigations. Univer- sity of Chicago press, 1973.
[44] Victoria Stodden. The scientific method in practice: Reproducibility in the computational
sciences. 2010.
[45] Rohit Bhattacharya, Tushar Nagarajan, Daniel Malinsky, and Ilya Shpitser. Differentiable causal discovery under unmeasured confounding. In International Conference Artificial Intelligence and Statistics, pages 2314-2322. PMLR, 2021.
[46] Trent Kyono, Yao Zhang, and Mihaela van der Schaar. Castle: Regularization via auxiliary causal graph discovery. Advances in Neural Information Processing Systems, 33:1501-1512, 2020.
[47] Roxana Pamfil, Nisara Sriwattanaworachai, Shaan Desai, Philip Pilgerstorfer, Konstantinos Georgatzis, Paul Beaumont, and Bryon Aragam. Dynotears: Structure learning from time- series data. In International Conference on Artificial Intelligence and Statistics, pages 1595- 1605. PMLR, 2020.
[48] Trent Kyono, Yao Zhang, Alexis Bellot, and Mihaela van der Schaar. Miracle: Causally-aware imputation via learning missing data mechanisms. Advances in Neural Information Processing Systems, 34, 2021.
[49] Boris van Breugel, Trent Kyono, Jeroen Berrevoets, and Mihaela van der Schaar. Decaf: Generating fair synthetic data using causally-aware generative networks. Advances in Neural Information Processing Systems, 34, 2021.
[50] Sébastien Lachapelle, Philippe Brouillard, Tristan Deleu, and Simon Lacoste-Julien. Gradient- based neural dag learning. In International Conference on Learning Representations, 2020. URL httph:tpp/... /ttp:://ppppeee http//ppeeeevieee
net/ forum? 1id=rkkkk4444DDD
[51] Yue Yu, Tian Gao, Naiyu Yin, and Qiang Ji. Dags with curl: An efficient dag structure learning approach. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th Inter- national Conference on Machine Learning, volume 139 of Proceedings of Machine Learning PMLR, 18-24 Jul 2021. URL pettpss:/proceeeeee
Research, pages 12156-1211 mir. www.speesseeeo /v1lll1 39/33//222 ///yyl2llll html.
[52] Yue Yu, Jie Chen, Tian Gao, and Mo Yu. DAG-GNN: DAG structure learning with graph neural networks. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of
12

---

Under review as a conference paper at ICLR 2023
Machine Learning Research, pages 7154-7163. PMLR, 09-15 Jun 2019. URL https: /procceddigg. mlr. www...peess/v/// http///7yylll htt.//yll9aa........ ht.htm..
[53] Judea Pearl. Probabilistic reasoning in intelligent systems: networks of plausible inference. Morgan kaufmann, 1988.
[54] Martin J Wainwright, Michael I Jordan, al. Graphical models, exponential families, and variational inference. Foundations and Trends@ in Machine Learning, 1(1-2)11-305, 2008
[55] Judea Pearl. Fusion, propagation, and structuring in belief networks. Artificial intelligence, 29 (3):241-288, 1986.
[56] Thomas Verma and Judea Pearl. Causal networks: Semantics and expressiveness. In Machine intelligence and pattern recognition, volume 9, pages 69-76. Elsevier, 1990.
[57] Dan Geiger, Thomas Verma, and Judea Pearl. d-separation: From theorems algorithms. In Machine Intelligence and Pattern Recognition, volume 10, pages 139-148. Elsevier, 1990.
[58] Dan Geiger, Thomas Verma, and Judea Pearl. Identifying independence in bayesian networks. Networks, 2075):507-534, 1990.
[59] Ronald Howard and James E Matheson. The principles and applications of decision analysis. Strategic Decisions Group, Palo Alto, CA, pages 719-762, 1984.
[60] JQ Smith. Influence diagrams for statistical modeling. The Annals of Statistics, 1, 1989.
[61] Dan Geiger and Judea Pearl. Logical and algorithmic properties of conditional independence and graphical models. The annals of statistics, (4)):2001-2021 1993.
[62] Robert W Robinson. Counting unlabeled acyclic digraphs. In Combinatorial mathematics V, pages 28-43. Springer, 1977.
[63] Matthew J Vowels, Necati Cihan Camgoz, and Richard Bowden. D'ya like dags? a survey on structure learning and causal discovery. ACM Computing Surveys (CSUR), 2021.
[64] Ignavier Ng, Sébastien Lachapelle, Nan Rosemary Ke, Simon Lacoste-Julien, and Kun Zhang. On the convergence of continuous constrained optimization for structure learning. In Interna- tional Conference on Artificial Intelligence and Statistics, pages 8176-8198. PMLR, 2022.
[65] Dennis Wei, Tian Gao, and Yue Yu. Dags with no fears: A closer look at continuous optimiza- tion for learning bayesian networks. Advances in Neural Information Processing Systems, 33: 3895-3906, 2020.
[66] Xinshi Chen, Haoran Sun, Caleb Ellington, Eric Xing, and Le Song. Multi-task learning of order-consistent causal graphs. Advances in Neural Information Processing Systems, 34: 11083-1 1095, 2021.
[67] Dimitri P. Bertsekas. Nonlinear Programming. Athena Scientific, 3rd edition, 2016. ISBN 978-1-886529-05-2.
[68] Richard H Byrd, Peihuang Lu, Jorge Nocedal, and Ciyou Zhu. A limited memory algorithm for bound constrained optimization. SIAM Journal scientific computing, 1665):1100-1208 1995.
[69] Marcus Kaiser and Maksim Sipos. Unsuitability of NOTEARS for causal graph discovery when dealing with dimensional quantities. Neural Processing Letters, pages 1-9, 2022.
[70] Alexander G Reisach, Christof Seiler, and Sebastian Weichwald. Beware of the simulated dag!
varsortabilityy in additive noise models. arXiv preprint arXiv:2102. 2021.
[71] Michael Irwin Jordan. Learning in graphical models. MIT press, 1999.
[72] Steffen Lauritzen. Graphical models, volume 17. Clarendon Press, 1996.
[73] Alsert-László Barabási and Réka Albert. Emergence of scaling random networks. science, 286(5439):509-33
1999.
[74] Peter Spirtes, Clark N Glymour, Richard Scheines, and David Heckerman. Causation, predic- tion, and search. MIT press, 2000.
[75] Thomas S. Verma and Judea Pearl. Equivalence and synthesis of causal models. In Proceedings of the Sixth Conference on Uncertainty in Artificial Intelligence, 1990.
13

---

Under review as a conference paper at ICLR 2023
[76] Ignavier Ng, AmirEmad Ghassami, and Kun Zhang. On the role of sparsity and dag constraints for learning linear dags. H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 17943-17954. Curran Associates, Inc., 2020. URL.
meurips ccccccc.ppa.. /ppppapp/////
2020// File/d04d42cdF 1tt.oo/42dd145722222222244
[77] Eric Walter. Identifiabiliiyy of parametric models. Elsevier, 2014.
[78] AmirEmad Ghassami, Negar Kiyavash, Biwei Huang, and Kun Zhang. Multi-domain causal structure learning linear systems. Advances neural information processing systems, 31, 2018.
[79] Jonas Peters, Peter Bühlmann, and Nicolai Meinshausen. Causal inference by using invariant prediction:
identification and confidence intervals. Journal of the Royal Statistical Society, Series B (Statistical Methodology), 78(5)9947-0012, 2016. ISSN 13697412, 14679868. URL
http:/ / /www. jstor. org/ httpg//taaeee/44222222
[80] Biwei Huang, Kun Zhang, Jiji Zhang, Joseph D Ramsey, Ruben Sanchez-Romers Clark Gly- mour, and Bernhard Schölkopf. Causal discovery from hederogenmoous/honsattinnnyy data. J. Mach. Learn. Res., (1(89)11-533 2020.
[81] Jiji Zhang and Peter Spirtes. Detection of unfaithfulness and robust causal inference. Minds and Machines, 18(2):239-2) 271, 2008.
[82] Markus Kalisch and Peter Bühlman. Estimating high-dimensional directed acyclic graphs with the pe-algorithm, Journal of Machine Learning Research, 8(3), 2007.
[83] Jiji Zhang and Peter L Spirtes. Strong faithfulness and uniform consistency in causal inference. arXiv preprint arXiv: www.2222222066. 2012.
[84] Caroline Uhler, Garvesh Raskutti, Peter Bühlmann, and Bin Yu. Geometry of the faithfulness assumption in causal inference. The Annals of Statistics, pages 436-463, 2013.
[85] Dan Geiger and David Heckerman. Learning gaussian networks. In Uncertainty Proceedings 1994, pages
Elsevier, 1994.
[86] David Heckerman, Christopher Meek, and Gregory Cooper. A bayesian approach to causal discovery. In Innovations Machine Learning, pages 1-28. Springer, 2006.
14

---

Under review as a conference paper at ICLR 2023
Appendix: D-Struct
Table of Contents
A Additional experiments
15
A.1 Settings and details
15
A.2 Completed results
17
A.3 Other DSFs
18
A.4 Suhsampling datasets
18
DAGs: D-Struct vs NOTEARS
A.6 Gains from enforcing transportability
B Causal interpretation and uniqueness
21
C Transportability in non-overlapping domains
23
D Definitions
24
E
Incorporating prior knowledge on [(P) using L-BFGS-B
25
F Additional details subsampling from different distributions
26
F.1 The general way
26
F2 How it's implemented in D-Struct
26
G ClT-based methods, score-based methods and faithfulness
27
G.1 ClT-based methods
27
(Differentiable) Score-hased methods
28
A ADDITIONAL EXPERIMENTS
Please find our (anonymous) online code repository at:
httpps:/aate //aos//////mmmus 4open. science/r/d-sttuce
Our code is based code provided by Zheng al. [36], and annotated code where used their implementation.
A.1 SETTINGS AND DETAILS
In the interest of space, we left out few details in our main text. Here we discuss hyperparameters (those in addition to the hyperparameters required for the selected DSFs), the evaluation metrics, and how we combine the different parallel DAGs.
Hyperparameters. D-Struct inherits hyperparameters from the chosen underlying DSFs. These hyperparameters act in the same way as they would in their original incarnation. For discussion on these hyperparameters we refer to the relevant literature these methods specifically.
However, D-Struct also adds two additional parameters: K and a. The impact of K already discussed in the main text, recapitulated as: K implicitly determines the sizes of the subsets used to train the parallel DSFs, as such, for high we should have high n. With both increasing, we report better performance (particularly in Scale-Free DAGs).
The impact of a is a bit more subtle, and also function of K. First, consider Fig. 7, displaying the impact each evaluation metric function of different What find is that setting a is mostly
15

---

Under review as a conference paper at ICLR 2023
dependent K as lower tend to work better with higher K, and vice versa for lower K. This makes sense as we sum each LMSE. resulting in a higher value with more K. If a is large in a setting with large K, the regularisation effect would simply be too large. We set our hyperparameters to those which yielded best performance (deduced from Fig. 7 for a, and K = 3 when not varied over as this yielded most stable results overall).
varying
varying
varying d
varying K
20
SHD
10
K
Figure 7: Results showing the effect of a. Depending on the nature of the problem the degree of regularization imposed by a vary. This then changes the amount enforce the similarity between the different D-Struct adjacency. matrices.
Evaluation metrics. The learned graphs from NOTEARS and D-Struct are assessed using four graph metrics namely: (1) Structural Hamming distance (SHD), (2) False discovery rate (FDR), (3) False positive rate (FPR) and (4) True positive rate (TPR). These values standard when evaluating structure learning methods. We provide some insight into these evaluation metrics below.
Structural Hamming distance (SHD) SHD is the total number of edge additions, deletions, and reversals needed to convert the estimated DAG into the true DAG. That means that the worst case SHD is d² - d, as we bound the diagonal to be 0 at all times, As such, the reported SHD with varying is expected to be higher, not due to hardness of the problem, but as property of the SHD (see for example Fig. 6).
False discovery rate (FDR)
Whenever edge is suggested in the estimated DAG, which is incorrect, we add to the falsely discovered edges. As such, the FDR is defined as the number of reversed edges and edges that should not exist, divided by the number of edges in total. Of course, the exception being when no edges are suggested at all (which implies dividing by O), which naturally has an FDR of zero
False positive rate (FPR)
We sum the edges that should have been reversed and those that should not exist, and divide by the total number of non-edges in the ground truth DAG. A non-edge is an edge that does not exist. With a more connected ground truth DAG, we expect this number to be lower automatically (as the numerator of the FPR would be higher). This is the reason why we let be function of d, as increasing the number of expected edges with d would somewhat counter this effect. Note that, in Table we see the FPR increasing proportionate to the factor multiplied with d, which is would expect.
True positive rate (TPR)
This signifies the number of correctly estimated edges, over the number of edges in the true graph. Note that, reversed edges are counted as wrong edges.
Combining graphs. Inference is done by combining the K internal graphs. In our implementation of D-Struct we combine graphs by averaging the adjacency matrices and apply threshold to convert the average graph into a binary matrix. The latter similar strategy to most DSFs' strategies to convert continuous matrix into binary one. This is relatively simple method with promising results, in line with what is currently done in literature.
However, given that D-Struct has multiple graphs, can actually come up with different strategies (a potential topic for future research). Naturally, this would be more relevant with high K, which in turn requires larger sample-size, of ensemble learning. Like D-Struct, ensemble methods need to combine, potentially confficting, outcomes and provide the user with only one outcome.
per our discussion above. Specifically, we enter the domain
16

---

Under review as a conference paper at ICLR 2023
Table 3: Results on Erdos-Renyi (ER) graphs. First block: We sample five different ER random graphs, and accompanying non-linear structural equations using an mode-. From each system we then sample varying number of samples, and evaluate NOTEARS MMP with D-Struct (indicated as "v") and without D-Struct (indicated as "x"). Second block: For each row we sample a new ER graph with varying degree of connectedness (s indicates the expected number of edges). In both cases, we report the average performance in terms of SHD, FPR, TPR, and FDR, with std in scriptsize.
metric
SHD (1)
FPR (L)
TPR (1)
FDR (1)
D-Struct
X
X
varying sample size
200
4.20:10.4 3.60
2.00
4.20:40.44
0.67:10.5
0.64
0.25
0.42:10004
500
3.20
3.94
1.20
3.94
0.66#1022
0.56:10.04
0.18
0.44:1004
1000
2.744
3.67
1.00
2.67:10.3
0.75
0.63
0.18
0.39
varying graph connectedness
0.5d
3.75
1.33
0.50
1.05
0.83
0.88
0.42:10.16
0.73-1001
ld
3.50
7.67=0..5
0.55
1.53:+0.99
0.75=0009
0.46:#0.09
0.40=0.09
0.77=0007
1.5d
3.10
5.67#1135
1.09
1.55
0.89
0.62:
0.32=0.05
0.53++0.44
2d
2.28
10.80
3.67:10.2
1.00.
2.67:10.3
0.67
0.70:10.09
0.11110.0
0.32:10.08
d
varying dimension count
2.28
3.67::0.2
1.00
2.67:10.63
0.67
0.70:10.09
0.11
0.32.10.08
7
8.67=10.56
12.88
0.72
1.07
0.96=0.02
0..0
0.49
0.63
10
19.71
30.82.0.95
0.42
1.18:10.4
0.70
0.00
0.34
0.70+0002
One avenue to not vote on a per-element basis, but on a per-graph basis. Imagine, two graphs in K that are exactly the same aspire more confidence in their accuracy. We could even relax similarity to an SHD across graphs, where we weight each graph's "vote" proportionally their combined SHD. We believe this to be promising area of future research.
Experimental procedure. Here we explain how our experimental setup works, which steps we need to perform before starting an experiment, and which information each model is provided.
There are two main parts to an experimental setup: (i) we need structure, (ii) we need a set of structural equations accompanying the structure of step (i).
(i) The structure. In our setup, structure can only be a DAG. To reduce bias as much as possible, we do not determine structures up front, but sample random structures for each experimental run. Of course, the same random structure is presented for each benchmark. Sampling random structures happens in two ways: either sample random Erdös-Reny. graph, which requires dimension count (d), and an expected number of edges (ds); or we use scale-free graph which is generated using the process described in Barabási and Albert [73] was also done in Zheng et al. [36], which needs a parameter 3 = (the exponent for the preferential attachment process). The expected number of edges in our setup depends on such that s resembles the ratio of edges versus non- edges in the random graph.
(ii) The equations. With sampled structure from (i), we can sample some structural equations. In our paper we use an index model to sample these. In short, an index model is randomly param- eterised as: (X(()))) = Σ Im (Ekepatj) 3mmxX where hı tanh, h2 = cos, hgg = sin, reported in
and each Ojmnk is drawn uniformly from range [-2, 0.5) U [0.5, 2). Exactly Zheng et al. [36].
A.2 COMPLETED RESULTS
Recall from Section 4 that we only reported a subset of the results. In Table we report the remainder for NOT NOTEARS-MA MI.P and the D-Struct implementation scale free graphs.
17

---

Under review as a conference paper at ICLR 2023
Table 4: Results on Scale-Free (SF) graphs. First block: We sample five different SF random graphs, and accompanying non-linear structural equations using an index-model. From each system we then sample varying number of samples, and evaluate NOTEARS- SOB with D-Struct (indicated as "(") and without D-Struct (indicated as "x"). Second block: For each row we sample a new SF graph with varying degree of connectedness (s indicates the expected number of edges). Third block: For each row we vary the feature dimension count (d). Fourth block: For cach row we vary the number of subsets for D-Struct (n).In all cases, report the average performance in terms of SHD, FPR, TPR, and FDR, with std in scriptsize.
metric
SHD (1)
FPR (L)
TPR (1)
FDR (L)
D-Struct
X
x
X
varying sample size
200
6.00
3,8
1.8::0.00
1.27:+0.8
0.49:1.12
0.83
0.63=0008
0.39-10.22
500
3.80
4.60
0.67
1.53.1008
0.57
0.69
0.41
0.48
1000
2.75 10.86
4.33
0.58
1.44
0.61 10.15
0.76.10.05
0.36
0.44
varying graph connectedness
0.5d
14.11=5.40
39.53=0.3
0.31
0.89
0.22:#0.14
0.20=#0.11
0.42=0.17
0.93++001
ld
8.11
39.46-0.38
0.13
0.89
0.30=
0.16:20.09
0.22
0.99+0001
1.5d
15.20
38.41
0.32
1.001
0.58
0.52.10.07
0.40
0.9810001
2d
15.20
38.25
0.32
1.04:1001
0.58:10.23
0.50:10.07
0.40
0.8911.01
d
varying dimension count
5
2.75.10.86
4.33
0.58
1.44
0.61
0.76.+0.05
0.36
0.44
8.25 0.55 15.00
1.0010001
0.96
0.78.10.02
0.49
0.7611001
10
16.80 35.75=0.33
0.36
0.99
0.58
0.67=10.33
0.42
0.85=+001
K
varying subset count
3.00=:0:2
6.00++0.0
0.53
2.00+0.11
0.66#0.04
0.57=00.4
0.21
0.60+0.03
3
2.75 10.86
4.3
1.44.1..17
0.61 10.15
0.76.10.05
0.36
0.44.10.5
2.80
5,25
0.73
1.75
0.74
0.6811..3
0.31
0.53
A.3 OTHER DSFs
We repeat the results above for NOTEARS-SOB which is Sobolev based implementation of NOTEARS, in Table 4. The main difference here with NOTEARS is the nomparametric es- timation the structural equations G. Note that, future implementations DSFs broadly alter the way in which the structural equations are estimated, and much less on how the proposed structure is evaluated to be a DAG (as they are mostly based on eq. (3)). Overall, we find that NOTEARS- - SOB behaves the same as NOTEARS-MLA D-Struct vastly improves performance.
Note that code to reproduce above results is provided in the online code repository linked to above.
A.4 SUBSAMPLING DATASETS
We refer to Table 6 for the full results presented originally in Table 2. While FPR may be a little higher, using D-Struct still outperforms not using D-Struct in terms of the FPR- already shown in Table 1. Furthermore, as the subsampling routine forces D-Struct to leam on different distributions, it is possible that this increase in FPR result of initially more confticting DAG structures. When combined, these structures include more edges which in turn result in more potential for false positive edge discovery. In fact, we observe a lower necessary threshold when using our subsampling routine, necessary to transform the real-values matrix into binary adjacency matrix.
We also report the same metrics as a function of the DAG-finding threshold in Fig. 8, where the threshold is applied to the adjacency matrix to produce binary matrix on which we compute the metrics. Of course, a threshold will be selected in practice; however, we show that for a range of plausible threshold values and all metrics that subsampling with routine is indeed beneficial, compared to randomized subsampling. From this, it seems that the results we find in Table are consistent even with changing thresholds.
18

---

Under review as a conference paper at ICLR 2023
Table 5: Results on Scale-Free (SF) graphs. First block: We sample five different SF random graphs, and accompanying non-linear structural equations using an mnde--ooel. From each system we then sample varying number of samples, and evaluate NOTEARS MMP with D-Struct (indicated as "(") and without D-Struct (indicated as "x"). Second block: For each row we sample a new SF graph with varying degree of connectedness (s indicates the expected number of edges). Third block: For each row we vary the feature dimension count (d). Fourth block: For cach row we vary the number of subsets for D-Struct (s).In all cases, report the average performance in terms of SHD, FPR, TPR, and FDR, with std in scriptsize.
metric
SHD (1)
FPR (L)
TPR (1)
FDR (L)
D-Struct
X
x
X
varying sample size
200
2.80
6.20:10.57
0.73
2.07
0.80
0.54
0.26=0.11
0.62:+0.06
500
20.00
7.20
0.271
2.20
0.77
0.37:10.09
0.14
0.72.10.0
1000
3.25 1.49
5.3310.61
0.75
1.78:10.20
0.68
0.66.10.08
1.2910.18
0.53
varying graph connectedness
0.5d
3.33
8. 8.00=+..7
0.50
1.17#00.0
0.92=0008
0.38
0.41
0.82.+0003
ld
3.33
1.00
0.50
1.17
0.92=2008
0.38
0.41
0.82++007
1.5d
3.25
7.67
0.50
1.17:10.04
0.94
0.42:10006
0.43 0..03
0.80.10.03
2d
2.75
5.00:11.00
0.33:
1.22:1::22
0.64
0.50:10.07
0.14:00.99
0.48:10.12
d
varying dimension count
5
1.29
5.3310.61
0.75
1.78:10.20
0.68
0.66:10.08
0.29
0.53
8.22
15.67
00.09
1.04:1001
0.98
0.83
0.54
0.7611001
10
16.80 35.75=0.33
0.36
0.99
0.58
0.67=10.33
0.42
0.85=+0001
K
varying subset count
2.40::0.24
6.50+10.46
0.53
2.16#0.15
0.83
0.50#+0.06
0.21
0.65++0006
3
2.00 1.044
5.33..06
0.33 10.47
1.78.10.0
0.68 10.14
0.66.10.09
0.14
0.53.10.06
0.75 10.48
5.25
0.25 10.16
2.55
1.00
0.30
0.09
0.76.10.03
SHD (L)
FPR (4)
TPR (t)
FDR (L)
0.9
0.4
0.8
3
0.2
0.6
0.8
0.9
0.8
0.9
threshold
threshold
threshold
threshold
with subsampling (V)
without subssmpling (1)
Figure 8: Subsampling with different DAG-thresholds. The DAG-threshold transforms the real- valued adjacency matrix, to a binary one. As the threshold increases, the amount edges that remain part of the DAG decreases. The above confirms findings from Table in different settings.
A.5 DAGS: D-STRUCT VS NOTEARS
We wish to also highlight that indeed what is recovered by D-Struct is different from NOTEARS. For this we refer to Figs. 9 and 10, each representing an independent run.
A.6 GAINS FROM ENFORCING TRANSPORTABILITT
A key concept of D-Struct is to enforce transportability, which is done using our novel loss function.
C(g (Da) LD + oLLMSE (LLMs)(A(G:)),
19

---

Under review as a conference paper at ICLR 2023
Table 6: Usefulness of our subsampling routine. We sample different ER random graphs, and accompanying non-linear structural equations in Zheng et al. [36]. From each system we then sample n 2000 samples, and evaluate NOTEARS-MLP with our subsampling routine from Section 3.2 (indicated as "V") and withoul the subsampling routine, using random splits instead (indicated as "x"), For each row we repeat our experiment with different K. both cases, we report the average performance in terms of SHD, FPR, TPR, and FDR, with std in scriptsize.
metric
SHD (L)
FPR (L)
TPR (1)
FDR (L)
Subsample
X
X
K
varying amount of splits
2..0
3.40:1..8
2.801.0.3
2.003
0..06
0.71::0.07
10.0s
0.30.1016
3.00
2.0010.51 0.00
1.60
0.73 10.04
0.58:10.06
0.22 10.05
0.24
2.80
1.40
1.40:2..0
0.60
0.71 10.06
0.53
0.18
0.07
M4
14
(a) True DAG.
(b) D-Struct estimated DAG. SHD=2, FDR=0.11, FPR=1.
(c) NOTEARS estimated DAG. SHD=2, FDR=0.2, FPR=2.
Figure 9: First independent run
162
H
(a) True DAG.
(b) D-Struct estimated DAG. SHD=2, FDR=0.11, FPR=1.
(c) NOTEARS estimated DAG. SHD=2, FDR=0.2, FPR=2.
Figure 10: Second independent run
The question is what do gain from the usage of the term which is key to enforcing trans- portability. We conduct an experiment where we set a = O. This not only assesses the importance of this term, but also without LMsE this amounts to assessing K independent versions of vanilla NOTEARS.
Results: When combine the K DAGs by averaging them, the result is NOT DAG.
This highlights that indeed that (1) transportabilityy is key as part this formulation and (2) that simply rumning parallel versions of NOTEARS is not sufficient solution.
We highlight this by showing the independent DAGs discovered without transportability enforced, the average of the DAGs and the true DAG. These results reported in Figs. 11 and 12
20

---

Under review as a conference paper at ICLR 2023
H
(a) True DAG.
59
(b) NOTEARS: Mean. This is not a DAG!
(c) NOTEARS: 1" DAG.
(d) NOTEARS: 2nd DAG.
(c) NOTEARS: 3 DAG.
(I) D-Struct: Mean. This DAG!
g D-Struct: 1 DAG.
(h) D-Struct: 2dd DAG.
(i) D-Struct: 3 DAG.
Figure 11: First independent run. Note the differences between the three DAGS each partition for NOTEARS (Row 1), the average also not a DAG. Whereas, for D-Struct note the similarities by enforcing transportability, the average is also DAG.
B CAUSAL INTERPRETATION AND UNIQUENESS
Causality. Causal relationships between variables are often expressed as DAGs [28]. While Struct is able recover DAGs more reliably, there is actually no guarantee that the found DAG can be interpreted as a causal DAG. There simple reason for this: we do not make any additional identification assumptions the structural equations when learning DAGs, at least not beyond what is already assumed in the used DSFs. Furthermore, should D-Struct be combined with a DSF that able recover a causal DAG the way in which the K internal DAGs are combined may violate these assumptions (recall DAG combination from Appendix A.1).
With D-Struct, we recover Bayesian network (BN), which is directed, yet the included directions are not necessarily meaningful. The only guarantee we have with BNs is that they resemble a distri- bution, which express some conditional distributions (as per the independence sets in Section 2. Order is not accounted for in these independence sets. For more information regarding this, we refer Appendix and Koller and Friedman [26].
However, as is indicated in Koller and Friedman [26, Chapter 21]. "good" BN structure should correspond to causality, where edges X - indicated that causes Y. Koller and Friedman [26] state that BNs with causal structure tend to be sparser. Though, if queries remain probabilistic, it docsn't 't matter whether or not the structure causal, the answers will remain the same. Only when are interested in interventional queries (by using do-calculus) we have to make sure the DAG is a causal one.
Uniqueness.
The above is pragmatic view. To knowledge, there is real proof stating that sparser DAGs are (even more likely to be) causal. However, could offer guidance to try and recover a causal DAG, assuming be sparse [76]. The latter of course assuming that there exists a unique or correct DAG, which is something we implicitly assume to be true. Naturally, when
"We know of that is able
21

---

Under review as a conference paper at ICLR 2023
(a) True DAG.
HHA
H
(b) NOTEARS: Mcan.
NOTEARS: 1 DAG.
(d) NOTEARS: 2nd DAG.
NOTEARS: 3 DAG.
This not a DAG!
(f) D-Struct: Mean. This is DAG!
(g) D-Struct: 1s DAG.
(h) D-Struct: 2d DAG.
(i) D-Struct: 3 DAG.
Figure 12: Second independent run. Note the differences between the three DAGS on each parti- tion for NOTEARS (Row 1), the average is also not DAG. Whereas, for D-Struct note the similar- ities by enforcing transportability, the average is also a DAG.
easy identification
hard identification
non-transportable models
transportable models
identifiable models
causal models
DSFs [35, 36, 51, 52]
CIT [74, 75]
D-Struct
Figure 13: Comparison of methods w.rff. identification and uniqueness. The ultimate goal of structure learning is to come with unique and correct structures. Once we recover the one true DAG, we may interpret the structure as causal model. However, discovering a causal structure using only observational data is not possible. Yet, we can approach it with methods that restrict the possible DAGs. From this illustration, we gather that D-Struct is attempt to restrict the solution space of DSFs, going one step further towards unique solutions.
22

---

Under review as a conference paper at ICLR 2023
aiming make a discovery, aim to recover a tre DAG, where truthful DAG is corresponding with a DAG that can be uniquely recovered.
However, there is a difference between unique DAG, and the unique DAG, Where the former is a matter of identifiability (discussed more below), the latter is one of causality. With the latter we mean: "can method actually recover the unique causal DAG?" From Meek [39] and Meek [39] learn that, from observational data alone, this is impossible and should thus not be a goal if one is not willing make additional assumptions.
We stress that transportability is a weaker goal than identifiability. Enforcing transportability does not guarantee unique or repeatable results. Take CIT-based methods- which we know to be fully transportable. While it is true that the same set of independence statements will always result in the same DAG (i.e. transportabilityy. it is not necessarily true that we will always recover the same independence statements. Depending on which independence test one uses to build the set of in- dependence statements, the resulting DAG may look entirely different. Similar for D-Struct, while D-Struct does encourage similar DAGs (see for example Appendix A), we have guarantee to recover the same DAG over different runs. The latter is a requirement for identifiability [77] as identifiability requires the model to always converge to the same set of parameters.
However, we do believe transportability is a vehicle to bring us closer to unique identification with DSFs. It is clear from our experiments that transportable learners greatly improve edge accuracy. As our synthetic setup is governed by one (and thus unique) graph, having a more accurate learner means learner that discovers DAG that is more like the unique, underlying graphical model. Consider Fig. 13 for an illustration comparing the relevant methods in terms of model identification.
C TRANSPORTABILITY IN NON-OVERLAPPING DOMAINS
Consider the multi-origin setting. where we have at least two datasets, each stemming from a differ- ent source. It is entirely possible that, given the different sources, these datasets are not comparable in terms of recorded features. We can recognise two major manifestations of this phenomenon: either (i) the supports of the datasets do not match, or (ii) the dimensions do not match.
(i) Different support. Recall from Section 2.1 that DAGs encode a set of independence statements. As such, it is mainly independence that governs structure. Transportability in the setting of con- flicting support, thus requires some (mild) assumptions. Specifically, we require that independence holds, regardless of support. The is mostly a pragmatic assumption. If for example, we find that X L X, where each component denotes a dimension in X. we usually don't specify over what support this independence holds. Implicitly, assume that independence holds, regardless of what area in (N, X3} we find ourselves in.
Note that the chosen distributions in P in Section 3.2 govern the entire domain [N], and as a con- sequence X. As such, the problem of conflicting support does not manifest in our solution of single-origin D-Struct. In case one chooses distributions that do not cover [N] equally, we have to assume independence is constant across different supports (i.c. the assumption explained above).
(ii) Different dimensions. more difficult setting of conflicting domains, when we record differ- ent variables in each of the multi-origin datasets. In order for a DAG to be transportable, we require the variable sets to correspond. As such, we are only able to work with overlapping intersections of the oon-ovellapping domains. Doing so requires some additional assumptions on the noise: as- suming we record some noise on each variable, we have to make the additional assumption that the noise is independent of the other variables, or at least the variables outside the intersection between domains. The latter made quite often, and should not limit applicability of D-Struct in this setting too much (recall that applicability of D-Struct is mostly determined by the used DSF). The reason relates to the second assumption, below.
The second assumption is a bit stricter: any variables outside the intersection cannot be confounding variables inside the intersection. two variables have no direct edges, and the nodes part of an in- direct edge fall outside the these two nodes. While this direct edge is wrong, this is actually expected behaviour of most DSFs as the algorithms will find these variables to be correlated (due the third, now unobserved, vari-
have to expect the DSF to find edge between
23

---

Under review as a conference paper at ICLR 2023
able). The only way to overcome these situations, is use DSFs that naturally handle unobserved confounding.
Related work. Some work structure discovery from multiple (non-overlapping) domains has been proposed. For example, Ghassami et al. [78] in the linear setting, Peters et al. [79] for the inter- ventional setting, Huang al. [80] in the temporal setting. While the difference between the first ((78)) is clear (only focusing on linear systems, whereas we on a non- non-parametric setting), the others immediately clear. Some intuition into the difference can be achieved by considering that both the interventional and temporal know where the difference in distribution is coming from. So much so, that the known difference exploited when garnering (causal) structural information. We believe applying our findings on transportability to the settings described earlier can be promising new avenue of research.
D DEFINITIONS
Definition 2 (Markov blanket.). A Markov blanket of random variable X₃ in random set x= {X1, [X.....,, Xa) is any subset X' C. X where, when conditioned upon, results in independence between X \ X' (the other variables) and X
X; L X x*
(6)
We will denote the Markov blanket of X; as (X.)).
In principle, Def. means that X' contains all the information present in X to infer X1. Note that this does not mean that X (X X' contains no information to infer X1, but variables X' are sufficient predict X1.
One step further, is a Markov boundary [53]:
Definition 3 (Markov boundary.). A Markov boundary of a random variable X4 of a random set X = {X,,...,X,, is any subset X C. X which is a Markov blanket (Def. 2) itself, but does not contain any proper subset which itself is a Markov blanket. We will denote the Markov boundary of X; as X- (X,).
We can relate the Markov boundary (Def. 3) to probabilistic graphical modelling, from sim- plified factorisation (in eq. (1)), we can compose a Bayesian network. Specifically, each vari- able X, X (X;) depiet
of three types of relationships: X; is a parent of X,, denoted as Pa(X,) = X,; X, is child of X,, denoted as Ch(X,) = Xz; or X, is a parent of a child of X,, denoted as Pa(Ch(X,))
Xj. Assuming that Px is governed by Markov random field (rather than Bayesian network) simplifies things, as the Markov boundary depicts only directly connected variables.
While the above may suggest that the Markov boundary only implies a vague graphical structure, doing this for ever variable in X' will strongly constraint the possible graphical structures respecting any found independence statements. D-separation (Def. 4) is then used to further limit the set of potential DAGs [28, 34). Relating above definitions those discussed Section 2.1. For more information regarding the above, we refer to Koller and Friedman [26].
Definition 4 (d-separation [34].). In a DAG 9, path between nodes X; and X, is blocked by a set Xa C X (which excludes X; and Xz) whenever there is a node X such that one of two holds:
(1) X E Xa and
Rz-1 +- Xs -- Rx+1,
X X2 X
or Xx-1 + Xx + X
(2) neither Xs nor any of descendants X. and
Xx-1 Xx= Xx+1-
Furthermore, in DAG G, say that disjoint subsets A and B are d-seperated by third (also disjoint) subset Xa if every path between nodes in and blocked by X... We then write
Lg Xa
When Xa d-seperates A and B in G, will denote this as d-sep (A; B)xa)
24

---

Under review as a conference paper at ICLR 2023
Definition 5 (Faithfulness from Peters et al. [34].). Consider distribution Px and DAG g (i) Px is faithful to G if
ALBC A A
for all disjoint sets A, B and C.
(ii) distribution satisfies causal minimality with respect to 9 if is Markovian with respect to G, but not to any proper subgraph of G.
Part (i) posits an implication that the opposite of the global Markov condition
A Lg BIC A l BIC,
for which we refer to Peters et al. [34, Def. 6.21].
Part (ii) actually implied when part is satisfied, when Px Markovian w.r.t. G, as per Peters et al. [34, prop. 6.35]. To have an idea for when faithfulness is not satisfied, we refer to Zhang and Spirtes [81] and Spirtes et al. [74, Theorem 3.2].
E INCORPORATING PRIOR KNOWLEDGE ON I(PP) USING L-BFGS-B
Consider the following, where we wish discover structure between 3 variables: X, Y, Z, where the ground truth satisfies T. Z According to the rules of d-speration (cfr. Def. 4), we are always structure where X and Y are only directly connected to Z, i.c. no direct connection between X and Y exists. Let us further assume that the system is linear (as this is what vanilla NOTEARS assumes, but without loss of generality towards recent NOTEARS extensions), then we have the following,
structural equations
structure
adjacency matrix
X EX, Z 3z,x X EZ:
A
0
Y = By 2 + +
Naturally, using only conditional independence, the direction of the arrows are not identifiable as explained above. However, NOTEARS is unable to narrow it down to the equivalence classes ex- pressed in Def. 4. The reason is simple, NOTEARS' three optimisation components (the h-measure, an Le loss, and an L1 regularizer on A, [76]) are satisfied exactly the same with the following sys- tem:
structural equations
structure
adjacency matrix
EX2 Z = 3z,x X +,,,
Z
0
A'
0
0
0
0
0 0
Y By.x. X -
where By.x 33.2 and éy dy,2c2+ resulting in Y being determined again by simple linear equation. Both systems allow the same data be generated, however under the constraint that X 1. 2 only the former is possible.
We argue that NOTEARS (and extensions) are unable to differentiate between them. Consider the components optimised by NOTEARS: both solutions propose DAG (i.e. h(A) h(A') 0); each DAG has an equal amount of arrows, leading to the same Llsss across A and A': and each equation is linear so NOTEARS is able perfectly converge to each solution using its L2 loss. Given that each component scores exactly the same, NOTEARS is unable to differentiate between these two results. Crucially however, in the latter system X is always dependent of Y. resulting in X Y Z|Z (and even X 1. eliminating v-structures) which completely opposite to the former system.
25

---

Under review as a conference paper at ICLR 2023
Prior Markov independencies. We can however force known independence statements into DSFs a priori, using the L-BFGS-B B optimizer. For example, consider the following / = X, 1 X, If I is known priori, then also know there cannot (under any cireumstance) exist direct link between X, and X, as this would immediately contradict which in turn would invalidate a structure proposing such link.
As such, we propose to fix these directed edges to 0 + (g), and exclude them from gradient calculation. This will not only constraint each DSL in step 2 above resulting in easier convergence, but it will also enforce any known x(Pxx) ) to be taken into account. Setting Ax,y Ay.x would immediately restrict NOTEARS from converging to this false solution as the solution would require Ax,Y to be 1. The same approach is currently used in NOTEARS (and consequentially D-Structs parallel DSFs), by setting bounds of each diagonal element in (0, 0).
Setting some elements to O using the L-BFGS-B bounds, we effectively limit the set of possible solutions. In fact, when applied to the above problems, the second solution would sit outside the set of possible solutions, ensuring that NOTEARS cannot converge it.
F ADDITIONAL DETAILS ON SUBSAMPLING FROM DIFFERENT DISTRIBUTIONS
In Section 3.2 introduced method to sample subsets from singlc-origin dataset such that the subsets correspond to distinct user-defined distributions. To provide some additional detail, we shall first discuss the general case, and then move discuss how implemented this in D-Struct.
F.1 THE GENERAL WAY
high-level view on our subsampling routine provided in Fig. 3. From Fig. 3 we learn that we need two ingredients for our subroutine work:
We need dataset that spans some domain We can retrieve this domain simply by caleulating the maximum and minimum value of each dimension in X. We have illustrated a simple dataset in Fig. 3a.
2. We need set of K distinet distributions that span X. In principle there is no constraint on these, besides them being different from one another, and each region in X' having a non-zero probability of being sampled. This is illustrated in Fig. 3b.
Using the above two ingredients. we create K empty subsets. For each subset, we then define distribution, illustrated in Fig. 3b. In Fig. we used a Gaussian for each subset they span the domain, and are simple to evaluate. Using these distributions, we will fill each subset using data from Fig. 3a. Each data point in our dataset is evaluated K times: using the user-defined distributions in Fig. 3b, we either include the sample in the corresponding subset, or not. When the probability of being sampled is high enough, enough could be determined by something simple as a threshold, or something less parametric as Bernoulli experiment. When finished, the subsamples look like Fig. 3c.
included, when not high enough, it is excluded. High
Alas, Gaussian distributions become more difficult to handle with increasing dimensionality as data spread sparser in high dimensions. The provided high-level example may serve well as a (visual) explanation of our subroutine, it does not work well in practice. As such, we used different implementation for D-Struct, which we explain in Section 3.2, and in more detail below.
F.2 HOw IT'S IMPLEMENTED IN D-STRUCT
Recall that the main issue with the simple Gaussian implementation above is that it does not scale well to high-dimensions. As such, we need a different implementation that scales to high- dimensions.
Defining the distributions. We do this using a very simple idea: rather than sampling in covariate- space, we sample the dataset's indices, which correspond to sample's covariates. However, before we do this, we need to make sure that the indices are in some way correlated with the covariates, which is not the case for standard dataset they are sampled i.i.d.
26

---

Under review as a conference paper at ICLR 2023
To provide some correlation between index and covariates, we first sort the covariates and reindex the dataset. This way, a smaller set of covariates now corresponds with a smaller index index-value. Note that it is unimportant whether we sort descending ascending, the only thing that matters is that there is some logical ordering.
Having index that is correlated with the covariates allows define distribution the indices (which are one-dimensional) rather than over the covariates (which are d-dimensional), We chose the beta distribution our user-specified distribution, where each of the K distributions is given different parameters. The advantage a beta distribution has, is their flexibility to move its density over the entire domain (contrasting Gaussian distributions which are symmetrical). This point is illustrated in Fig. 4.
Sampling data. Once we have defined our distributions, we can use them sample data. As with our high-level idea in Appendix F.1, we will evaluate each data point K times to determine whether or not should be included in each subset. However, rather than evaluating the chosen distributions using the covariates directly, we now use the index instead. Regardless of the number of dimensions we have, the index remains one-dimensional.
Evaluating a sample in D-Struct is done using Bernoulli experiment: with the beta distributions we query the probability of being sampled and provide it to a Bernoulli experiment, the outcome determines inclusion exclusion.
G CIT-BASED METHODS, SCORE-BASED METHODS AND FAITHFULNESS
G.1 CIT-BASED METHODS
CIT-based methods such as the well known PC-algorithm, the SGS algorithm, or the inductive cau- sation (IC) algorithm all require faithfulness as per Def. 5. The reason is such that they render the Markov equivalence class identifiable. As have explained in Section 3.1, using d- d-separation have a one-to-one correspondence to this class of DAGs. Any query of d-separation statement can therefore be answered by checking the corresponding conditional independence test [15].
Most ClT-based methods have main phases, based on set conditional independence state- ments. Assuming the latter is correct set (that is, have correctly inferred all the independence statements present in Px ), we first infer a skeleton graph, and then orient the edges. After these two phases, we have either fully identified DAG, Markov equivalence graphs in case there edges we were not able to orient.
Phase 1: inferring skeleton. Based lem. (below) introduced in Verma and Pearl [75], the SGS and IC algorithm build a skeleton from a completely unconnected graph.
Lemma 1.
The following two statements hold:
(i) Two nodes X and Y in a DAG E) are adjacent iff they cannot be d- separated by any subset S C X X (X.Y).
(ii) If nodes X and Y in DAG (X,E) are not adjacent, then they are d-separated by either Pax or Pay.
Clearly, by using above lemma, SGS [74] and IC [28] require faithfulness. Contrasting methods that build from an unconnected graph, is the PC-algorithm which does the reverse: PC starts with fully connected graph and step-by-step removes edges when they violate (ii) in lem. 1. While a different approach, both require d-separation, i.e. this too requires faithfulness to hold!
Phase 2: orienting the edges. As per Meek [39] there exists set of graphical rules that is shown complete to correctly orient the edges based only d-separation. Of course, this requires com- plete set of correct independence statements which is arguably a much stricter assumption than faithfulness.
Essentially, we can relax the assumption of a complete set of independencies, but we'l have to replace it with other assumptions. One such example is assuming a Fx be Gaussian (which is also quite strict, but it serves our example). With the latter assumption, we can test for partial correlation [34, Appendices A.1 and A.2], which allows identify the underlying Markov equivalence class
27

---

Under review as a conference paper at ICLR 2023
[82]. Furthermore, by additionally assuming condition called strong faithfulness [83, 84], we have uniform consistency [82]. We refer to Peters et al. [34, Ex. 7.9] for an example.
G.2 (DIFFERENTIABLE) SCORE-BASED BASED METHODS
Contrasting CIT-based methods, are score-based methods. Score-based methods generalise our dif- ferentiable score based methods and non-differentiable methods. Contrasting CIT-based methods, which directly encode the independence statements governing Fx into G, score-based method will evaluate 9 on how well it fits the observed data. The rationale behind these score-based methods is that wrongly encoded independence statements will yield poor model fits [85, 86).
We can formalise score-based based method as function S which is to be optimised over candidate DAGs:
arg S(D,9).
G DAG over DEX
As such, there two elements that comprise score-based method: (i) the function S, and (ii) the way we optimise In our case, that is:
(i) S corresponds eq. (5), which in large part determined by the underlying DSF through DDM
(ii) S is optimised using gradient-optimisation, which has proven very efficient in this problem setting
Importantly, that rationale behind these methods does not require the faithfulness assumption for them to work. The latter may lead to violations against d-separation case faithfulness does hold. However, in Appendix E show how
combat against this, by also incorporating any known independencies into our graph (which does require the faithfulness assumption to hold for those independence statements) using the L-BFGS-B optimisation algorithm.
28


Reviews:
[{"review_id": "coZxMYvgT0", "reviewer": "Reviewer_nxCL", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper develops an extension of Notears, a recent approach for structure learning with continuous optimization, to learn transportable structures (transportable in the sense that the learned structures are the same across multiple datasets with potentially different distributions). This is achieved by incorporating a regularization term in the optimization procedure to encourage the structure parameters over different dataset to be equal to one another. Empirical results are provided to validate the proposed method.", "summary_of_the_review": "- A number of highly related works were not discussed and compared to. Specifically, the proposed method is similar to [1, 2].\n- The proposed method adds a regularization term w.r.t. both structure and model parameters, which may lead to the wrong structure in practice especially when the data distributions are different for different datasets.\n- The paper does not consider experiments of different data distributions for different datasets, which may be more common in practice and is the key motivation of transportability/invariance.", "strength_and_weaknesses": "### Strengths\n- The problem studied is highly relevant, since learning transportable structures may be an importance topic especially because of data heterogeneity across different datasets.\n- The paper is well written and easy to follow.\n\n### Weaknesses\n- The setting studied is very similar to (and seems to be a special case of) differentiable federated structure learning [1, 2], which should be discussed. E.g., [1] considered data heterogeneity difference (i.e. different data distribution across multiple clients, but the underlying DAG is the same), which is essentially the same as transportability in this paper. If I understand correctly, the only key difference appears to be that [1, 2] additionally considered privacy issue.\n- Therefore, the resulting approach is also similar to [1, 2]. Specifically, [1] compute the average adjacency matrix in each step and enforces different clients to use the same average matrix, while [2] enforces the adjacency of different clients to be the same via ADMM. The only key difference appears to be that the proposed approach uses a soft regularization scheme to *encourage* the same structure, while [1, 2] used a hard scheme to *enforce* the same structure.\n- Following the comment above, although adding a soft regularization scheme encourages the same structure, it does not guarantee/enforce it, which I think might be a limitation (in practice the participating clients may want to learn a shared structure that is the same). This is also acknowledged by the experiment in Sec. 4.\n- Adding a soft regularization directly over the parameters $A$ (which correspond to both structure and model parameters) may lead to the wrong structure. I.e., this regularization may conflict with the original least squares loss of the specific dataset when the data distributions are different. In my opinion, the backbone method Mcsl used by [1] might seem like a better choice (as compared to Notears-MLP) because it decomposes the structure and model parameters, and so the proposed regularization can be applied w.r.t. the structure parameters only.\n- I did not manage to find experiments of different datasets having different data distributions in Sec. 4, which I think are much more relevant (considered by [1]), since that's the key motivation of using transportability (Definition 1) or invariance.\n- The considered setting is highly similar to (invariant) causal discovery from multiple domains or heterogenous data, e.g., [3, 4, 5], which should be discussed and compared.\n\n### Minor/Other Comments\n- Sec 2.2: $\\rho$ and $\\lambda_2$ are not \"hyperparamers\", but rather the parameters of augmented Lagrangian method that will be updated during the optimization process. \n- Sec. 2.2 \"One can traverse G_X smartly to arrive at a DAG much faster\": There was no guarantee that DSF arrives at a DAG in (Zheng et al, 2018); this was proved by [6, 7] and should probably be mentioned. Also, to the best of my knowledge, Zheng et al. (2018) did not claim/demonstrate that Notears runs \"much faster\" than CIT-based methods like PC, and I would encourage the authors to add a reference regarding this.\n- For Notears-MLP, is the regularization scheme applied to the weighted adjacency matrix constructed w.r.t. the MLP weights, or all weights of the MLPs? Alg. 2 seems to indicate the latter, but the former sounds more intuitive to me.\n- The efficiency gain in Fig. 5 is interesting and surprising. Is it because the for loop in fifth line of Alg. 1 can be run in parallel?\n\n1. Federated causal discovery, 2022.\n2. Towards Federated Bayesian Network Structure Learning with Continuous Optimization, 2022.\n3. Multi-domain Causal Structure Learning in Linear Systems, 2018.\n4. Causal inference using invariant prediction: identification and confidence intervals, 2015.\n5. Causal Discovery from Heterogeneous/Nonstationary Data, 2020.\n6. On the Convergence of Continuous Constrained Optimization for Structure Learning, 2022.\n7. DAGs with No Fears: A Closer Look at Continuous Optimization for Learning Bayesian Networks, 2020.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Rm26ulXgUqz", "reviewer": "Reviewer_auHu", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors consider the problem of learning DAG from heterogeneous datasets that share the same DAG. This problem statement is quite restricted. From Eq(4) we can see that the authors not only encourage the DAG to have the same structure, but also to have the same parameter. While in Figure 1, the authors assume that multiple datasets to have different distribution. In this case, there is only one possibility, that is the distribution of noise variable may be different. In the NOTEARS framework, the only possibility is that the Gaussian noise may have different mean and variance in different datasets. This setting is in fact quite restricted. ", "summary_of_the_review": "The paper proposed a method to learn DAG from heterogeneous datasets, but the setting in the paper is quite restricted and the experiments are also weak.", "strength_and_weaknesses": "Weakness: \n\n1. The problem setting is quite restricted.\n2. There is no theoretic guarantee that the proposed algorithm will find a set of consistency DAG.\n3. The experiment is quite weak. Instead of only using nonlinear models, the authors may also add some linear experiments, where the DAG is known to be identifiable. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "9AHrqsqLdn", "reviewer": "Reviewer_CF11", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes transportability in DAG structure learning problem, which can be seen as a multi-task DAG learning problem or a cross validation method from a single task. The changes from NOTEARS methods is the addition from an average graph loss term additional to the differential scoring function. The method is compared with one baseline NOTEARS to show its superior performance in accuracy.", "summary_of_the_review": "the paper addressed a new problem in DAG learning but with straightforward technical contribution. ", "strength_and_weaknesses": "Strength:\nIt seems this is a first work to learn one single DAG from multiple domain.\nSubset construction is interesting, as it may improve the statistical property of the estimation. \n\nWeakness:\nThe technical contribution is rather limited, with a MSE terms on graphs. It would be interesting for authors to discuss the choice of such a regularization against potential other alternatives. \nOnly one baseline is compared. Understandably this may be first work, but other works could and should be adapted as a baseline. For example, the multi-task DAG learning from [62].\n\nComments:\n- subset construction: it is not fully clear how much the performance gain is from d-struct formulation, compared against an ensembled approach on different subset of data (maybe taking an average to obtain the final result with measure to ensure acyclicity).  This could also serve as a baseline.\n- While transportability definition is clear, authors did not discuss much about it with respect to other standard problem regimes. For example, how is transportability problem different from a typical multi-task learning or transfer learning for DAGs?\n- another way to improve technical contribution is to study feature differences or absences in different domains, and how to integrate them into one single DAG. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "523jpitxPm6", "reviewer": "Reviewer_pcus", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper studies the problem of learning DAGs from observational data. In contrast to previous work under the continuous framework, the authors aim to learn a DAG that is transportable to other distributions over the same observables that respect a set of conditional independences. In a few words, the approach consists of learning K different DAGs where each DAG is penalized by how far away it is from the mean of the K DAGs, in this way, the approach remains fully differentiable and the resulting DAGs are somehow forced to be similar or equal for the K different datasets. The authors apply a preprocessing sampling step to make the method work even for the case of observing a single dataset. Some experiments are provided to demonstrate the efficacy of the proposed method against the nonparametric NOTEARS algorithm.", "summary_of_the_review": "My main concerns are listed above which currently make me inclined toward rejection.", "strength_and_weaknesses": "### Strengths\n\nThe core ideas of the paper are well-written and well-motivated. The authors do a good job of stating the problem under study and the contributions of their work. The method is simple to grasp and simple to adapt to existing differentiable approaches to learning DAGs.\n\n### Weaknesses\n\nThe main weakness is the experimental section. \n\n* Lack of proper comparison to other methods. First, in my opinion, the correct baseline should not be plain NOTEARS but the version of NOTEARS with the same exact sampling mechanism for Dstruct and taking the average or median as the final output, i.e., the only difference to Dstruct should be the L_MSE step. I stress this point because we need to make sure that (part of) the improvements are not due to the ensemble mechanism (which would be a very incremental contribution) but to the addition of the L_MSE regularization. Second, there really should be comparisons to other methods, especially Bayesian methods such as DiBS [1]. This is because in those methods we can sample DAGs from a posterior distribution and, again, perhaps taking the mode or mean would result in a transportable DAG. For these Bayesian methods, I would feed all the data---even from different distributions---and see if they are robust to these changes in distributions. \n\n* The speed-up claims are rather disappointing, the graphs have a very small number of variables, $d=5$. To make a more precise claim about computational speed-ups there must be experiments for different values of d and not just n. This is because increasing $n$ really just makes the training of the neural nets longer but does not increase the complexity of enforcing acyclicity, for which $d$ does. Recent work [2] for instance ran the NOTEARS-MLP for up to 100 nodes leveraging a new acyclicity function. \n\n[1] Lorch et al. (2021), \"DiBS: Differentiable Bayesian Structure Learning\"\n[2] Bello et al. (2022), “DAGMA: Learning DAGs via M-matrices and a Log-Determinant Acyclicity Characterization”\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject