Learning Large DAGs is Harder than you Think: Many Losses are Minimal for the Wrong DAG
Authors: Jonas Seng
Matej Zečević
Devendra Singh Dhami
Kristian Kersting
Venue: ICLR 2024 poster (2024)
Keywords: Directed Acyclic Graphs
Structure Learning
Primary area: general machine learning (i.e., none of the above)

Abstract: Structure learning is a crucial task in science, especially in fields such as medicine and biology, where the wrong identification of (in)dependencies among random variables can have significant implications. The primary objective of structure learning is to learn a Directed Acyclic Graph (DAG) that represents the underlying probability distribution of the data. Many prominent DAG learners rely on least square losses or log-likelihood losses for optimization. It is well-known from regression models that least square losses are heavily influenced by the scale of the variables. Recently it has been demonstrated that the scale of data also affects performance of structure learning algorithms, though with a strong focus on linear 2-node systems and simulated data. Moving beyond these results, we provide conditions under which square-based losses are minimal for wrong DAGs in $d$-dimensional cases. Furthermore, we also show that scale can impair performance of structure learners if relations among variables are non-linear for both square based and log-likelihood based losses. We confirm our theoretical findings through extensive experiments on synthetic and real-world data.

Published as a conference paper at ICLR 2024
LEARNING LARGE DAGS IS HARDER THAN YOU THINK: MANY LOSSES ARE MINIMAL FOR THE WRONG DAG
Jonas Seng
Matej Zečević
Technical University of Darmstadt jonas seng@tu-da www..at...tt.....mmm.mmm de
Technical University of Darmstadt matej. wwc.vi www.wdtammttat..mmmmm de
Devendra Singh Dhami
Kristian Kersting
Eindhoven University of Technology d. S. chami@tus nl
Technical University of Darmstadt
Centre for Cognitive Science of TU Darmstadt
Hessian Center for AI (hessian.Al)
German Center for Artificial Intelligence .d
kersting@cs,
ABSTRACT
Structure learning is crucial task science, especially in fields such as medicine and biology, where the wrong identification of (in)dependencies among random variables can have significant implications. The primary objective of structure learning is learn Directed Acyclic Graph (DAG) that represents the underlying probability distribution of the data. Many prominent DAG learners rely on least square losses or log-likelinee liglliielhood losses for optimization. regression models that least square losses are heavily influenced by the scale of the variables. Recently has been demonstrated that the scale of data also affects performance of structure learning algorithms, though with strong focus on linear 2-node systems and simulated data. Moving beyond these results, we provide conditions under which square-based losses are minimal for wrong DAGs in d- dimensional cases. Furthermore, we also show that scale can impair performance of structure learners if relations among variables based and log-likelihood based losses. We confirm our theoretical findings through extensive experiments synthetic and real-world data.
well-known from
non-liness for both square
INTRODUCTION
Given finite data sample from unknown probability distribution, structure learning algorithms aim to recover the graphical structure underlying the data generating process that lead to said unknown probability distribution (for introduction to probabilistic graphical models see (Koller & Friedman, 2009)). The use of a Directed Acyelic Graph (DAG) as a representation of choice is prevalent in many applications, particularly in Bayesian Networks (BN; see (Pearl & Russell, 2000)). The directedness and acyclicity of DAGs provide significant advantages proving foundational results, including research in causality, (Pearl, 2009). In DAGs, node corresponds and each edge marks a direct statistical dependence between two random variables. The absence of edge encodes (in)direct independencies between random variables. Since DAGs compactly encode statistical (in)dependencies, structure learning algorithms learning DAGs are widely deployed throughout Machine Learning applications. Different approaches have been proposed solve the non-trivial task of recovering the independence structure from finite data sample. Some use statistical independence-tess
random variable
to infer graph, others use score-functions which are being optimized during learning (Mooij et al., 2020; Peters et al., 2017). In recent years, score based methods became appealing with the development of NOTEARS (NT; (Zheng et al., 2018)) a novel score-based structure learning algorithm with a continuous and differentiable DAG-constraint which avoids the combinatorial nature of constructive constraints. Often the score for these methods is chosen s.t. maximizes the (log-)ll )likelihood of the data given the model. Under the frequently made Gaussian noise assumption, this reduces to minimizing the mean squared error (MSE). To highlight that the MSE is taken w.r.t. the entire graphical model, we call this score model squared error (MMSE).

---

Published as a conference paper at ICLR 2024
Loh & Bühlmann (2014) showed that least square based losses unsuited for structure identification of DAGs when the variances of exogenous noise variables remain unknown, Reisach et al. (2021) recently had analyzed specifically the behaviour of NT and similar methods and found evidence that they might exploit these variances during structure learning, thus confirming the results of Loh & Bühhlann (2014) in low-dimensional linear cases. However, both works only considered linear dependencies among the variables in low dimensions and square based losses. Also, real world problems usually involve dozens or more variables and many structure learners optimize log- likelihood or similar losses (e.g. BIC, ELBO). Thus investigating conditions under which optimizing MMSE leads to wrong DAGs d-dimensional cases as well as analyzing whether losses like BIC and ELBO are more appropriate choices for structure learning is of high interest for many practitioners Consequently our contribution will (1) generalize the above mentioned results to d-dimensional and non-linear cases, (2) provide exact conditions under which MMSE fails to identify the correct structure and (3) show that all log-likelihood based losses are susceptible to scaling under appropriate assumptions, both theoretically and empirically.
Medical Example. To highlight the importance of identifying the correct DAG given data, provide the following example: Assume a dis- tribution pa, which is defined as in App. A and factorizes following the Bayesian Network G, shown Fig. Further assume we sample data X from Pc., The variables iz, iy denote whether a patient got infected with disease or y respectively, s denotes if patient shows symptoms typical for x and y, e denotes an exposure to virus and i denotes if we con- duct a treatment. Given two graphs G1 and G2 s.t. G1 G. (i.e., G1 is without red edge in Fig. and G2 with red edge), fitting both
1x
e
iy
Figure 1: Network Structure can flip decisions. Fitting two DAGs G1 (without the red edge) and G2 (with the red edge) on the same data sampled from Pc, changes the probability of assigning a treatment (t=11 1) given the exact same evidence. graphs to X yields different probabilities whether the same patient should receive treatment:
Pc, (t = j2 = 1, ig = 0) = 0.49 < 0.51 = PG, (t = 1/1 1, in 0). It is common to choose a threshold value of 0.5 decide which action should be taken for variables with binary domain, thus using G2 instead of G₁ would flip the decision from not giving a treatment to giving a treatment based on the same evidence. Thus recovering the wrong graph structure can have severe impact on decision making.
We proceed by stating assumptions made throughout our paper and briefly revisit widely used structure learning algorithms which we use in our experiments. Then, we establish novel theoretical findings on how scale influences MMSE and likeliillhood based scores in structure learning contexts. Finally, we conduct experiments and empirically support our theoretical results before giving a conclusion.
2 BACKGROUND AND RELATED WORK
To the best of our knowledge, Loh & Bühlmann (2014) were the first raising the awareness that MSE-like losses are not suitable for the identification of graph structures from data when the variances of exogenous noise variables are unknown. Reisach et al. (2021) recently has shown that structure learning algorithms like NT suffer from performance degradation when data is standardized. Kaiser & Sipos (2021) 1) independently reported similar results. Other important works discussing identifiability and its relation variance include Peters & Bühlmann (2014); Park (2020); Weichwald et al. (2020).
2.1 ASSUMPTIONS
We consider distribution pl X X....., Xa) defined over random variables XI,..., Xa associated with a DAG G representing the independencies of variables XI,..., Xa in The following assumptions are made: (A1) Immiscible Structures: G is either single chain, fork or collider with d variables. We also make the common assumption (A2) Additive Noise Model with Invertable Functions: Each
variable X, can be written as X, = f(PAÇ X, N,,N (N( o)) where f is a linear or non-linear function that is invertable w.r.t. the noise, PA G X, refers the parent variables of X; and (O a)) denotes
2

---

Published as a conference paper at ICLR 2024
zero- mean Gaussian with standard deviation We assume to obtain ii.d. samples from p. denoted as X E Raxd
Remark Note that each DAG is composed of chains, forks and colliders. is possible that node X; of a DAG is e.g. both fork and collider. If we do not allow for such structures, each DAG can be decomposed into subgraphs fulfiling (A1) Immiscible Structures. This decomposition allows us to reason about far more complex graphs than chains. forks and colliders.
2.2 DETAILS ON DAG LEARNERS INVESTIGATED IN THIS WORK
We briefly present four prominent score based structure learning algorithms which often optimize square based or log-like ligllike based loss. We consider these methods throughout our work as they are widely used and/or constitute the foundation of many more recent algorithms (Lee al., 2020; Lorch et al., 2021; Wei et al., 2020; Zhang et al., 2022).
GES (Chickering, 2003). Greedy Equivalence Search (GES) is a score based structure learner which can identify the correct structure of a Bayesian Network up to Markov equivalence. It consists of two phases: Starting from an empty graph, edges are added to optimize some score/loss C. A second phase aims remove as many edges possible until L cannot be improved anymore the first phase might add to many edges. Often L. is the Bayesian Information Criterion (BIC). Note aims find Markov Equivalence Classes (MECs).
that GES does not necessarily learn a DAG However, having variables on different scales assigns higher scores to certain DAGs in a MEC. We consider the highest scoring DAGs as the solution of GES.
NT (Zheng et al., 2018). Given data X E Raxd from a distribution p(.,,..., induced by an unknown DAG G. NT aims to recover G's structure by solving a constraint optimization problem where G is represented by weighted adjacency matrix G == W E R and the optimization problem is defined as arg minlw gRaxa adjacency representing G, A is regularization parameter favour sparser graphs, h continuous and differentiable DAG-constraint and a the Lagrange multiplier. The continuous nature of the optimization problem allows efficient gradient based optimization.
X,))
3 + >/| + ah (W). Here, W is the learnt |X XWI
DAG-GNN (Yu et al., 2019). DAG-GNN (DG) extends NT to learn structures even if relation- ships non-linear long they are invertible by stating structure learning as an encoding- decoding problem. The encoder aims to learn the inverse function describing the noise variables Z (2 Za} function of the observed variables X [X Xa) while the decoder aims to learn the Encoder- and decoder parammerrss as well as the adjacency W are learned during ELBO optimization with acyclicity-constraint, resulting in an augmented Lagrangian. For more details refer to App. D.1.
describing the relationship between noise Z and variables X.
GraN-DAG (Lachapelle et al., 2020) Given data X from random variables X Xj,..., "y....,X. X₂ GraN- DAG (GND) models relationships among variables via dedicated neural network (NN) for each X; given X-, where X-, refers to all variables except X,, By following the so called neural network paths through the NNs, the authors derive adjacency W based on the parameters of all NNs. Then, inspired by NT, W and hence the NN parameters is constrained s.t. W is a DAG, ie. h hWW)= which is formulated a Lagrangian augmented optimization problem, solved via gradient descent.
Note that we only consider score based structure learners in this work since constrained based algorithms are based on independence tests which typically do not depend variable scales, See App. C.9 for details.
3 INFLUENCE OF SCALE ON DAG LEARNERS
In this section we will prove conditions under which structure learners optimizing least square and log-likelihood losses are guaranteed to predict a DAG that differs from the ground truth DAG, sometimes even with different d-separation statements. Note that there are exactly three structures that any DAG is composed of: (I) chain structures consist of d single-link struetures where one single-link structure is followed by another, i.c. X, + X2 + ... + X, (2) fork structures share the same d-separation statements chains do, however their graph structure slightly different and is
3

---

Published as a conference paper at ICLR 2024
defined X1 x- ... t X1-1 X, + X1+1 + - Xa and finally (3) colliders are defined as X1 + X2 7 X3 f X,-1 t Xd.
3.1 DATA GENERATING PROCESS
For our analysis we make the following assumptions about the data generating process: (1) The data X contains samples from each variable in DAG G, (2) X is sampled from a distribution pG which Markovian w.r.t. G and (3) each dependence among variables in G can be represented
function I: PAx. + X; that is a linear or non-linear function with additive Gaussian noise where PAx, x, are the parents of a variable X,, Also we assume that the variance of each variable X only depends on the unit of measurement. For example, when measuring distances, the scale is determined by whether metres (m) or kilometres (km) are measured (e.g. Ikm vs 1000m). As a specific unit (e.g. km) often can be expressed in terms of another unit scaled by constant factor (e.g. km 1000 m), we represent measuring on different scales as a linear scaling operation done on data X: For data samples X of variable X,, different measuring units can be expressed as X!" := X?". Note that the struciure of the data generating process is invariant w.r.t. the measurement scale.
3.2 EXTENDING VARIANCE SENSITIVITY RESULTS TO DIMENSIONS
For our theoretical analysis let us define the Model MSE (MMSE) of model fw. parameterized by i and respecting (in)dependence structure imposed by adjacency W representing DAG as:
Definition 1. Given an adjacency matrix W E Raxd
d of a DAG of d nodes, fw.9 Rd + Rd and 2n fw.o
data X E Raxd we define the Model MSE (MMSE) as MMSE(X. (.MSE(X.fw.))>>
Remark 2. Note that structure learners use constraints such as that the learned graph is a DAG. Often these constraints are expressed regularization terms. Throughout this section we assume that all such construints are fulfilled and only consider the space of valid solutions.
Note that due to (A2) Additive Noise Model with Invertable Functions fw,e be written vector of functions fw 9 (f1.0....,, fa.o) where X (X,o(Xr,) holds for index set Pj Pax, selecting all parents of X, according to W. We will use this property in our subsequent propositions.
3.2.1 SQUARE BASED LOSSES
We start our theoretical analysis by showing that MMSE is scale dependent given (A2) holds:
Proposition 1. Given samples X E Ruxd from a distribution p(X1,..., Xa) over real-valued random variables X1...., X an adjacency W E ddddd representing a DAG and a function Jw respecting (A2) Additive Noise Model with Invertable Functions, the MMSE is proportional the sum of MSEs of each node in the graph:
MMSE(X, W)
1 2n
|X
fw.o(X)
(x)
Σ Var(X)) + Σ MSE(X (x)(Xp))) (1)
sEZ
IEN
In the above proposition, Z denotes the set of node-indices without parents, N denotes the node- indices of nodes with parents and MSE(X." fr,e(()))) )) denotes the mean squared сттоr between X. and free (X p. ). A proof can be found in App. B. Since the MMSE contains variance terms, it susceptible to the scale of variables. Knowing conditions when minimal MMSE corresponds to wrong DAG would be crucial to understand the behavior of structure learners. Therefore, let us consider only linear dependencies among variables in G, i.e. (X) = XW where W is a weighted adjacency. Our first key result states that, given chain graph and data from its associated distribution, the MMSE of a reversed version of said chain is smaller than the ground truth variables can be sorted by its scale:
Proposition 2. Consider a chain graph G = +... Xa and G = t .... + Xá. distribution p( X(1,..., Xa) over zero-mean random variables XI,..., Xa induced by G and i.i.d. data X E Rxd from p. If ((Vaa((()> ... Var(X?) holds, then MMSE(X, G') MMSE(X,G) G).
Proof (Sketch). The main observation leading to the proof is that (1) the MMSE of a graph with d variables is the sum of d MSEs and (2) when writing the MSEs in terms of the variances and
4

---

Published as a conference paper at ICLR 2024
covariances of the variables only two things change in the MMSE for reversed chain: (a) instead of Var(X() only Var(X/) is added to the sum of MSEs and (b) the regression coefficients between variables X; and X1+1 are computed as Cov(X, X instead of Cov(X, X rendering them Var(X
Var(X
smaller due to the symmetry of covariance. Since the weight's magnitude and the variance of the exogenous variable decide whether the original DAG the reversed chain receives smaller MMSE,
the reversed graph will receive a lower loss. The full proof is given in App. B.
Also, if there is no strict order of the variables in terms of their scale, optimizing MMSE leads to wrong graph as the following proposition shows:
X a distribution factorizing according G and data X sampled from p induced by G. Then Var(X3 Var(X")
Proposition 3. Consider a d-dimensional chain graph G =, - x +
E11 Cov(X, X Cov(X, X
suffices to make MMSE smaller for the reverse chain.
Var(X/+1)
Var(X
Proof (Sketch). Due to the symmetry of the covariance and the replacement of Var( Var(X]) with Var (X) MMSE will be minimized if the condition Var(X?) > Var(X - d-- Cov(X) X
Var(X,,))
Cov(X? x holds. Note that the strong statement of the reverse chain being preferred over all
Var(X?)
other possible DAGs is not made, i.. only the reverse chain is preferred over the ground truth chain. Still, a flip on e.g. the last edge could yield an even lower MMSE. We give empirical evidence for this conjecture in Sec. 4. A full proof is given in App. B.
Similar to how scale dictates that MMSE is minimal for the reversed chain, MMSE gets minimal for a fork structure instead of chain structure if the conditions derived in the following proposition hold: Xa and data X E naxd sampled
Proposition 4. Given chain graph G X1 - from a distribution pl p(X1,... Xa) induced by G, a fork originating at node Xj receives a lower MMSE if the scale of the variables is such that Var(X?) > Var(X Vi E {l...., - 1} and
Var < Var(X ViE Vie li+,,...,, d-... 1}.
Proof (Sketch). The key observation is that one can consider the outgoing paths of X, as chains. In this case, the right-going path from X, will receive the same MMSE as in a chain graph if we ignore the rest of the graph. For the left- eeft-going path from X, the same reasoning applies as for reversing chain-graph then. See App. B for a full proof.
Prop. 2-4 can be generalized a Markov Equivalence Classes (MEC):
Proposition 5. Consider a DAG G with nodes X1...., X MEC M(G) and data X E Raad sampled from distribution p(X1,..... Xa) induced by G. Then the graph G' E MM(G) will receive a lower MMSE than all other graphs in the same MEC if the edges of G' are oriented s.t. they point from
variables towards strictly higher-variance variables.
The proof follows the same reasoning as the more special cases in Prop. 2, 4. see App. B.
Proposition 6. Given a chain X1 -... - X& and data X from distribution D(X1.....X.. DAG having Xj as a collider and an edge between Xj-1 and Xj+1 41 gets smaller MMSE than DAG without additional edge if Var( < ... < Var(X?) and (XT.).)) >...> Var(X
Proof (Sketch). The path X1 +.... X5-1 is the same as in the original graph, thus the MSEs are the same. By Prop. a chain X. - ......... X, will receive a lower MSE than vice versa. The additional edge can be explained by the fact that X; and X are dependent in X, the collider however would induce an independence between these variables. Thus, to further minimize the MMSE, an edge between X,-1 and X,+1 is required. See App. B for the full proof.
The above proposition shows that DAG learners using optimizing MMSE predict a collider and an additional edge if variables are scaled appropriately, effectively encoding different independencies.
5

---

Published as a conference paper at ICLR 2024
3.2.2 LOG-LIKELIHOOD BASED LOSSES
As Prop. 2-6 only apply to the MMSE-loss under the assumption of linear dependencies among variables, we will now generalize our results to non-linear dependencies in two ways: First, we will show that the log- logeleiilhood is as susceptible to scaling as MMSE under Gaussian noise assumption. Second, we generalize this result to log-likelihodd based losses and show that popular losses such as the Bayesian Information Criterion (BIC) and Evidence Lower Bound (ELBO) are also susceptible to scaling under Gaussian noise assumption:
Proposition 7. Assuming (1) distribution p over random variables XI....,X, which factorizes according to some DAG 9 which represented by an adjacency W, (2) Gaussian, additive noise Cj for each X; in the graph, (3) possibly non-linear transformations fj s.l. X;= 1===a (f(aax,)) + Cj. (4) data X Buxd sampled from p and (5) a solution space of DAG adjacency matrices W, then optimizing the MMSE is equivalent to optimizing the log-likelihool.
Proof (Sketch). Given data X, the log-likelihood w.r.t. DAG and parameterized functions ʃ3.0 for each Xj predicting parameters for distribution over Xj given Pax, can be recursively written as EL= Ej=1 log fee refers to model parameterized by 0 predicting parameters for the distribution over X, given its parents. Assuming P is Gaussian, the log-likelihood reduces the MMSE from Definition and reads E/ (X (X,))) Note that f5,0 can be any function, thus also non-linear. O
where Pj- Pax, refers to the parents of Xj according to W,
We note, however, that in contrast to the linear case, it is not trivial derive conditions under which the optimum of the log-likelihood render a wrong DAG without additional assumptions about functions fz.e. We leave the derivation of such conditions for future work.
As the log- likeliood term appears in many scores/losses used for structure learning, we extend Prop. 7 to a more general case. To that end, we define the family of aigliiklihood based losses as:
Definition 2. The family of higeeiied losses for distribution p. parameters and data vector x E R is defined as L(x, 0) = >i=1 log p(x.(0) + h(-) where fu(-) is some arbitrary function. Proposition 8. Under the same assumptions of Prop. 7, the family of log-likelihood losses reduces to square based losses and thus is susceptible to variable scale.
The proof for Prop. 8 immediately follows from Prop. 7 as it extends the regular log-likelihood by an additive term. We now show that BIC and ELBO as two widely used instantiations of the log-likelihood loss family are susceptible to variable scaling.
Proposition 9. We assume the assumptions of Prop. 7 and further assume an encoder (( mapping observed random variables to latents Z, decoder pl p(XiZ) mapping latents observation spuce X and prior Z) Further, we assume the encoder and decoder to be defined as in Yu et al. (2019). Then, ELBO can be written in terms of the MMSE in Def. 1 and thus is susceptible to scaling.
Proof (Sketch). The ELBO can be written as Z is multivariate latent and X multivariate random variable over the input space. Since Ex(21x) llog(((X (ee(p(X]p) [Z)]] is an approximation of the log-likelihood of the seen data given the la- tents and the log-likelihood w.r.t. DAG model with adjacency W can be written the sum of log-likelshoods of each variable X3 in the DAG, the reconstruction loss of the ELBO reduces to the MMSE due the Gaussian noise assumption.
[lo](p(( Here,
Proposition 10. Under the same assumptions of Prop. 7, BIC can be written terms of the MMSE in Def. I and thus is susceptible to scaling.
Proof. The proof directly follows from Prop. 7 and Def. 2 as BIC w.r.t. an adjacency W and models f,, can be defined as the sum of the lieeellldoodd and additive regularization preferring simpler models: BIC(X) k log(n) - 2( [ Et_ log(p(X,,, (og(x(x.,,1130(xx:))) Here, & refers to the number of model parameters, the number of samples, the rest as above and P, Pax. is an index set selecting the parents of Xy. It is easy to see that the BIC an instantiation of the log-likelihood loss family and hence is susceptible to scaling under Gaussian noise assumption. n

---

Published as a conference paper at ICLR 2024
3.3 REDUCING VARIANCE SENSITIVITY
As shown above, numerous losses, assuming Gaussian noise, exhibit high sensitivity to the scaling of the measured variables. Free variance terms in losses proportional MMSE are major source of misleading information: Nodes lacking parental connections are treated as "not explainable" and their variance is unjustly included in the loss. Hence the graph minimizing free variance terms as far as possible will be rated good candidate. Also, it is well known that variance influences MSE. However, we argue that a good score should only measure how well structural dependencies among variables in dataset are captured by some DAG and nodes without parents should be treated as unmodelled noise. Consequently, enhancing the resilience of structure learning algorithms can be achieved by: (1) Scaling all variables to have equal variance and (2) excluding free variance terms from the loss, leading to Scale Robust Loss (SRL) defined as L. (X; W) L(X; W) Σisz Var(X;) Here, Z is the set of nodes without parents above and a square based or log- likeliihoodd based loss with fixed variance Gaussian noise assumption. Normalization makes MSE independent of scale. By excluding free variances, we ensure that the loss exclusively represents the quality of dependencies encoded by a DAG. Note, however, that SRL can only be used for discrete structure learners as it requires us to know which nodes have no parents. As continuous structure learners operate a continuous parameter space representing graphs and usually W + 0 for all i, in early stages of optimization, each node has at least one parent. The task of devising effective strategies to bolster the resilience of such structure learners against scaling is matter we defer to future research endeavors.
4 EXPERIMENTAL EVALUATION
Table I: Predictions of structure learners deter- mined by scale. We generated 10,000 data points from 10-variable ground truth DAGs (ch=chain, fo=fork or co=collider) with additive Gaussian noise. Variable de- pendencies were linear (lin) or non-linear (cos=cosine function used). We simulated different measurement scales by multiplying each variable with a different scale, experiments were repeated 30 times with dif- ferent data from the same distribution to account for stochastic effects. GESR means GES with SRL.
After laying the theoretical foundations, we aim to empirically answer the following questions: (Q1) Can we empirically confirm our theoretical findings?
(Q2) How severe
the effects of variable scales if only subset of variables is measured on differ- ent scale? (Q3) What happens if assumption (A1)) Immiscible Structures does not hold? (Q4) Do scales affect the prediction of struc- ture learners on real world data?
Confirming theoretical findings (Q1): To validate our theoretical results, we artificially created data from (3. 10)-node chains, forks and colliders with (non)-linear relations be- tween the variables and additive noise from standard normal distribution (see App. C). We applied scaling operations on each vari- able in the data to simulate measuring on dif- ferent scales. Scaling was done to match our assumptions, i.c. the order of variables w.r.t. their variance coincides with a certain DAG we expect the structure learners to predict. For example, in Fig. 2(b)/Fig. 2(c) we sam- pled data from chain X .... X10 and expected a collider/fork at X6 to be pre- dicted due scaling the variables s.t. the nodes can be sorted by variance according to their topological order with Xg having the largest/smallest variance. Given data from each structure (chain, fork, collider), we tested each possible structure to be predicted by the structure learners due to scaling, For each pair of ground truth, expected prediction and linear/non- - cases, we used 3 different scales. Each of them was tested 10 times to account for stochastic effects. We
Predicted Graph
ch
fo
co
lin (NT)
100%
100%
100%
lin (GES)
100%
100%
0%
ch
lin (GESR)
50%
50%
0%
cos (DG)
100%
100%
100%
Ground Truth Graph
cos (GND)
100%
100%
100%
lin (NT)
100%
100%
100%
lin (GES)
0%
100%
100%
fo
lin (GESR)
50%
0%
0%
cos (DG)
100%
100%
100%
cos (GND)
100%
100%
100%
lin (NT)
100%
100%
100%
lin (GES)
0%
0%
100%
co
lin (GESR)
0%
0%
0%
cos (DG)
100%
100%
100%
cos (GND)
100%
100%
100%
We did not relax (A2) Additive Noise Model with Invertable Functions as it is common assumption in the literature (Yu et al., 2019).
7

---

Published as a conference paper at ICLR 2024
X
X9
X8
X7
X10
X9
X8
X7
X
X
X
X2
X3
X4
X
X1
X₂
X
X
X
(a) GND prediction, non- -onearn dependencies
(b) NT prediction, linear dependencies
X 10
X9
X8
X7
X 110
X9
XN
X7
X
X
X1
22
X3
X4
X5
X
X
X
X5.
(c) DG prediction, non-lincar dependencies
(d) GES prediction, lincar dependencies
Figure 2: Predictions of structure learners impaired by measurement scales, Given a chain as ground truth, NT/GND/DG/GESS predict the wrong graph if certain variables are measured on different scales (marked blue). This holds in both, linear (a, c) and non-linear cases (b). The following color coding was used: ..... means edge in ground truth, means predicted edge, means edge appears in ground truth and prediction. (Best viewed in color.)
observed that our expectations were fulfilled in 100% of the cases for NT/DG/GND, Due to its greedy strategy, the prediction of GES did not match our expectation in some cases. However, GES still predicted a graph different to ground truth due to variable scales. SRI. improves robustness of GES against scale even further. Thus we empirically confirm our theoretical findings and conclude that scale has severe effects structure learners optimizing square based loss (see Tab. and App. C).
Subset of Variables (Q2): Usually only some variables are measured on different scales. For example, an internet provider could measure the provided download speed in GBit/s and upload speed in MBit/s as often download speed is more important to customers. To investigate how severe the effect of different scales is if just subset of variables is scaled, reuse the experimental setup from Q1. However, this time scaling was applied to only a subset of variables: Let X E TR Rxd denote a dataset with n samples from d variables and ((A) E |n*|A| denote a dataset where a subset measured different scale. We conducted experiments with
A C {X1,..., Xa) of variables e (3, 10) for linear and non- non-linear relationships among the variables. We found that it is enough to measure single variable on different scale which has at least two neighbor nodes in the ground truth DAG to provoke severe effects. For example, in Fig. 2(b) we expected a collider in Xe to be predicted instead of the ground truth chain only by scaling data from X6 by a factor of 12. We empirically confirmed that all structure learners except GES predict fork/collider in 100% of the cases if one variable is measured on different scale, for both linear and non- linear dependencies. GES introduces forks/colliders in ~ 40% of the cases, however often on different nodes.
Ablation of (A1) Immiscible Structures (Q3): To see if assumption (A1) is strictly required for scale to have severe effects on predictive performance, we generated 20 random DAGs with 10 nodes and artificially generated data as above. None of the random DAGs constituted a single chain, fork or collider, thus violating assumption (A1). We then identified 3-node substructures in each DAG, each making up a chain, fork or collider. We simulated measurements on different scales in these sub-graphs s.t. we expected structure learners predict a chainn/fork/collider--usttttitt this substructure is not present in the ground truth. This was done by scaling all variables of the identified substructure (Case 1) by only scaling one variable (Case 2). Again, each simulation was tested 10 times on data from the same distribution to account for possible stochastic effects. In both cases we found that the identified substructures are severely affected by scale for NT, DG and GND. This holds in 100% of the cases for both linear and non-linear dependencies. For example, in Fig. 3(a) we expected structure learners to predict a collider in X, instead of the fork structure in the ground truth only because X, was measured on scale with factor of 4 larger than the original data. We also discovered that in both cases, many substructures were wrongly recovered without direct connection to the substructure measured on different scale. We suspect that this is due to the more complex dependence-strrutuue without SRL, we only encountered changes in the graph's independence statements in ~ 20% of the cases. The conjecture is that GES benefits from the complex structure and its greedy strategy to find graphs: By greedily searching, graphs being sensitive w.r.t. scale might already be excluded from the candidate set in early stages. We conclude that different scales have severe effects even if (A1) does not hold, however GES seems to be rather robust. For more detailed results refer to App. C.
although
high dimensional cases. GES was surprisingly robust against scale
8

---

Published as a conference paper at ICLR 2024
X
P38
PKC
Akt
X7
Erk
X1
X6
Rof
X2
X3
X4
X55
Mak
Flcg
PIP2
(a)
(b)
Figure 3: Prediction of NT severley impaired by measurement scale complex graphs. Even in cases with complex dependencies among variables, NT predicts a collider in X1 (Fig. (a)) and Akı (Fig, (b)) when data is measured on different scales, (marked blue). Color coding is in Fig. 2.
Real World Data (Q4): Since generated data might not accurately resemble real-world data, also conduct our experiments with real world dataset provided in Sachs et al. (2005). As the ground truth graph of this dataset unknown, the following protocol used: (1) run Structure Learning (SL. algorithm on the original data and obtain a graph G predicted by the SL-algorithm, (2) identify substructure (chain, fork collider) to alter measurement scale and construct expected graph G', (3) run the same SL-algorithm on the scaled dataset and (4) assess the similarity between the predicted graph G after rescaling and G' as well as the similarity between G and the expected graph G'. For evaluation: If the substructure we expected to obtain is present in G. we consider the scale to have severe effect. If this is not the case, the scale is considered non-decisive for the considered substructure. As we know from Q2 that different measurement scales have unintentional side effects in complex systems, we also measure the similarity s(G, G") and s(, G') where is the Structural Hamming Distance (SHD). With that we aim to capture the severity of side effects imposed by measurement scale. We again observed that in 100% of our simulations severe effects were present due to scale except for GES, as shown App. C. Fig. 3(b) illustrates measurement on different scales s.t. Akt was turned into collider. GES, again, is more robust against scale and did predict a chain/fork substructure in G' if said substructure was a collider in G. In Tab. 8, in App. we provide measures for the severity of side effects using SHD for different structure learners: Graphs predicted data with different measurement scales have higher SHD w.r.t. G' although our expectations were fulfilled, i.e. G and G' share at least one substructure. Hence other substructures must contribute to the higher SHD, i.e. different measurement scales also lead to worse predictions w.r.t SHD. To sum up, we have shown that in real world scenarios where we have no control about assumptions predictions of structure learners can be severely impaired by measurements on different scales, the same time leading to a decrease performance w.r.t. SHD.
CONCLUSION
We formally proved that measurement scale can decide which DAGs will minimize the MMSE or log-likelihood based losses such as ELBO and BIC. Hence, predictions of structure learning algorithms which make use of these loss functions should be treated with care. We empirically confirmed our theoretical results in extensive experiments using state of the art structure learners and have shown that discrete structure learners are robust against scale when data normalized and variances are excluded from the loss.
Limitations and Future Work. Although our work provides proofs that measurement scale can determine the DAG minimizing MMSE, ELBO and BIC for d-dimensional chain-, fork- and collider- structures, a further theoretical analysis of more complex structures with less assumptions such as in our experiments would be interesting. Additionally, we did not incorporate other dynamics in the optimization process our proofs, such as regularization or other hyperparameters controlling the learning algorithm. An interesting next direction can be to check the theoretical properties for other families of losses. Also, deriving scale-independent score functions is of high importance for reliable score based structure learning.

---

Published as a conference paper at ICLR 2024
REPRODUCIBILITY
We acknowledge the significance of reproducibility in scientific research and have taken multiple steps to ensure the strength and replicability of our work.
com/ JOnas Seng/ FooLS. We have used publicly available software and libraries to guarantee accessibility and have comprehensively described the architecture, software, versions, and hyperparameters Our code is deterministic, incorporating seeds for all random number generators to guarantee the replicability of results. We attempted to include most of the code used to create the result tables and figures in this manuscript.
Code: Our implementation accessible on GitHub at https://giieee
App. C.
Datasets:
This study only utilizes publicly available datasets which have been correctly cited. Furthermore, the authors contribute open source repository containing all the datasets used in this work, which will be made available upon acceptance.
We have provided thorough descriptions and formulations of our architecture in the main text, supplemented by additional clarifications, and implementation details in Sec. C, ensuring clear understanding of our contributions and facilitating reproduction. This documentation intended to provide researchers with all the necessary information for an accurate replication of experiments.
Algorithm Details:
ACKNOWLEDGEMENTS
This work supported from the National Segh-Performance Computing project for Computational Engineering Sciences (NHR4CES). Furthermore, it benefited from the Hessian Ministry of Higher Education Research, Science and the Arts (HMWK) via the DEPTH group CAUSE of the Hessian Center for AI (hessian.ai) and the cluster project "The Third Wave of Al". The Eindhoven University of Technology authors received support from their Department of Mathematics and Computer Science and the Eindhoven Artificial Intelligence Systems Institute.
REFERENCES
David Maxwell Chickering. Optimal structure identification with greedy search. 3:507-554, 2003.
Marcus Kaiser and Maksim Sipos. Unsuitability notears for causal graph discovery. arXiv preprint (rXiv:2/04005441 2021.
Daphne Koller and Nir Friedman. Probabilistic graphical models: principles and techniques. MIT press, 2009.
Sébastien Lachapelle, Philippe Brouillard, Tristan Deleu, and Simon Lacoste-Julien. Gradient-based neural dag learning. In International Conference Learning Representations, 2020.
Hao-Chih Lee, Matteo Danieletto, Riccardo Miotto, Sarah T. Cherng, and Joel T. Dudley. Scaling structural learning with NO-BEARS to infer causal franscriptome networks, pp. 391-402. 2020.
Po-Ling Loh and Peter Bühlmann. Hligh-dimensione learning of linear causal networks via inverse covariance estimation. Journal of Machine Learning Research, 15(88):3065-3105 2014.
Lars Lorch, Jonas Rothfuss, Bernhard Schölkopf. and Andreas Krause. Dibs: Differentiable bayesian structure learning. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 24111-24123, Curran Associates, Inc., 2021.
Joris M Mooij. Sara Magliacane, and Tom Claassen. Joint causal inference from multiple contexts. 2020.
Gunwoong Park. Identifiability of additive noise models using conditional variances. J. Mach. Learn. Res., 21(75):1-34, 2020.
Judea Pearl. Causality. Cambridge university press, 2009.
10

---

Published as a conference paper at ICLR 2024
Judea Pearl and Stuart Russell. Bayesian networks. 2000.
Jonas Peters and Peter Bühlmann. Identifiability of gaussian structural equation models with equal eπror variances. Biometrika, 101(1):219-228 2014.
Jonas Peters, Dominik Janzing, and Bernhard Schölkopf. Elements of causal inference. The MIT Press, 2017.
Alexander Reisach, Christof Seiler, and Sebastian Weichwald. Beware of the simulated dag! varsortability in additive noise models. arXiv preprint arXiv:2102.136477 2021.
Karen Sachs, Omar Perez, Dana Pe'er, Douglas A. Lauffenburger, and Garry P. Nolan. Causal protein-signaling networks derived from multiparameter single-cell data. Science, 308(5721): 523-529, 2005.
Dennis Wei, Tian Gao, and Yue Yu. Dags with no fears: closer look continuous optimization for learning bayesian networks. Advances in Neural Information Processing Systems, 33:3895-39066 2020.
Sebastian Weichwald, Martin E Jakobsen, Phillip Mogensen, Lasse Petersen, Nikolaj Thams, and Gherardo Varando. Causal structure learning from time series: Large regression coefficients may predict causal links better in practice than small p-values. In NeurlPS 2019 Competition and Demonstration Track, pp. 27-36. PMLR, 2020.
Yue Yu, Jie Chen, Tian Gao, and Mo Yu. Dag-gnn: Dag structure learning with graph neural networks. In International Conference on Machine Learning, pp. 7154-7163, PMLR, 2019.
Zhen Zhang, Ignavier Ng. Dong Gong, Yuhang Liu, Ehsan Abbasnejad, Mingming Gong, Kun Zhang, and Javen Qinfeng Shi. Truncated matrix power iteration for differentiable dag learning. ArXiv, abs/2208. 14571, 2022.
Xun Zheng, Bryon Aragam, Pradeep K Ravikumar, and Eric Xing. Dags with tears: Continuous optimization for structure learning. Advances in Newral Information Processing Systems, 31, 2018.


Reviews:
[{"review_id": "Q2YlL46BqZ", "reviewer": "Reviewer_HNsv", "summary": "This work examines the behavior of score based and continuous optimization approaches to causal discovery when mean squared error is used as the objective measure. Building off of the work of Loh & Buhlmann who studied the behavior, and inapplicability, of the MSE under unknown variances, the work examines behavior under non-linear dependence and characterize the failure conditions. The authors then show that under the Gaussian noise assumption all log-likelihood based scoring functions are susceptible to scaling failure conditions. Empirical results are presented on synthetic data which coincide with the theoretical results.", "questions": "Not necessarily a question, but it would be useful if the authors can add a discussion and / or empirical comparison of constraint based discovery methods.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This work provides an interesting and important extension in analyzing the issues with applying mean squared error as an objective in structure learning. Given the recent rise in continuous optimization approaches for structure learning I believe this is particularly important. The inclusion of both optimization and scored based methods is also nice. The authors do a good job of clearly defining the problem, and presenting the analysis in a clean manner, along with the implications of results.", "weaknesses": "I think the biggest issue here is a lack of discussion of / contextualization with constraint based approaches. This seems particularly important since constraint based discovery still constitutes a large portion of commonly applied discovery algorithms. It would also have been good to see comparisons to constraint based discovery in the experiments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VMbFdysyw5", "reviewer": "Reviewer_kwyp", "summary": "In this paper, the authors provide conditions under which using square-based losses will lead a wrong DAG result in structure learning in d-dimensional cases. They also show that scale influences the performance of structure learning if relations among variables are non-linear for both square based and log-likelihood based losses.", "questions": "1.\tIn this article, MMSE seems to be used for prediction, rather than for learning causal graphs. The definition of MMSE is the sum of squares of predicted residuals, which characterizes the statistical correlation of variables rather than causal relationships. It is an obvious conclusion that only using statistical correlation for causal graph learning will lead to errors.\n2.\tIn structure learning algorithms, the score function usually contains penalty terms, and the variable that needs to be optimized in the score function is the adjacency matrix W, rather than data X. In addition, the adjacency matrix W will also be included in the penalty term. So, why does the score function directly degenerate into a square based loss function? Or, why is optimizing the score function equivalent to optimizing MMSE?\n3.\tIn practical applications, most of the score functions we choose are not affected by data normalization, just like the SRL function mentioned in this article. Thus, this article does not have a clear critical objective, or rather, the critical target seems to be rare in practical applications, which greatly reduces the significance of this study.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "the authors provide conditions under which using square-based losses will lead a wrong DAG result in structure learning. The conclusions drawn in the paper are reasonable.", "weaknesses": "I think that the findings in the paper aren't unexpected. Learning causal structures should go beyond just predictive effects (MMSE). I don't see the paper as particularly significant.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WmOMtD9qcK", "reviewer": "Reviewer_FNo4", "summary": "The paper points out the issue that structural learning approaches that are based on a log likelihood or squared error loss, such as NOTEARS, are typically not scale invariant. This is, these approaches are typically sensitive to the scaling of the data, where rescaling can arbitrarily change the inferred graph structure. The authors provide multiple theoretical statements connecting the variance of the data with the squared loss of a structure learning approach under specific assumptions and evaluate the statements experimentally.", "questions": "The paper points out some very important issues with L2 and log likelihood based approaches. However, many points need more clarification. Some of my questions and remarks, which I hope the authors can address:\n\n- Consider introducing the graph G more formally as a collection of nodes/vertices and edges.\n- It would be very insightful to comment more on the assumption A1, since it seems to be very strong. For instance, the formulation of A1 implies that you only consider a chain or a fork or a collider, but none of the combinations. If this is true, this needs a particular remark and justification. Otherwise, it does not seem to be useful for any realistic graph structure beyond these trivial ones. \n- In this regard, at the end of paragraph 3, you say that with A1 you consider \"all possible substructures\", but A1 is clearly defined as \"either ..\" and not a combination.\n- Assumption A2 is a bit confusing, since typically, one only requires invertibility with respect to the noise and not the functional relationships (as mentioned in the 'Weakness' section). For instance in DAG-GNN, the \"invertibility\" only refers to the invertibility of Y = f(X, N) with respect to N. Similar to the previous point,  the need for A2 requires more justification and explanation, since it appears to be a very strong assumption in the current form.\n- The unit measurement assumption is a bit unclear. Why is the unit of a node relevant, since, as you also write it, the \"rescaling\" is part of the functional relationships. In particular, the current paragraph even reads as that these measurements need to be comparable, but looking at the theoretical statements, I don't see why. For instance, one could model \"Latency in ms -> Click Rates on a website as discrete number -> Revenue in dollars\", where all nodes have a completely different unit.\n- While I understand the focus on the MMSE, it should be noted that in an optimization task, some regularization is required. Otherwise, the best solution for ||X - f(X)||_2 is f(X) = X, the identity. Here, a brief remark on this would be insightful to avoid confusion.  \n- One of my concerns is that the theoretical statements are rather commonly known points. For instance, there is a clear connection between the least squares error and the conditional variance, which is clearly not scale invariant. However, as mentioned before, I still see the value in bringing all these points together in a single paper in the context of structure learning.\n- In Section 3.3, the exclusion of \"free variance terms\" needs more clarification. It currently reads as that one needs to know the graph structure to identify these terms, while the whole task is to infer the graph structure in the first place. I assume there is a misunderstanding where the authors can maybe comment on.\n- With regards to the previous point, maybe looking at the conditional variance instead of the marginal variance could help in focusing on the structural connection.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper addresses a very important problem and common misconception when log likelihood or squared loss based approaches, like NOTEARS, are used for inferring graph structures.\n- Great introduction to the problem.", "weaknesses": "- While the implications of the statements are definitely important, they appear rather obvious, since, for instance, it is well known that the L2 loss is closely connected with the variance. In that sense, the issue with the scaling is already discussed in other works. However, that being said, I did not find a paper that particularly focuses on this issue in a self-contained manner.\n- The need for A2 remains unclear. In particular, the \"invertibility\" of the \"function\" in the related literature typically refers to the invertibility with respect to the noise, not with respect to the input. This is, in a general SEM Y = f(parents, noise), the f needs to be invertible with respect to the noise, but it doesn't matter for the parents. Otherwise, this would be an extremely strong assumption. In the case of additive noise models, this invertibility is always given by definition, since N = Y - f(parents), i.e., it is not a restriction on f. However, I might have also missed something here and maybe the authors can clarify this.\n- While you reference great related literature commenting on the same issue, it remains unclear where they lack and where you are filling the gap.\n- The proposed approach in Section 3.3 seems rather trivial. Here, I think the main focus of the paper should rely on the theoretical statements.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9ighHmLQRn", "reviewer": "Reviewer_NBUG", "summary": "The paper considers many common score/loss functions used in learning DAGs and shows thay can lead to identifying a DAG different than the ground truth. More precisely, the authors show that the (non-)linear score functions are sensitive to the units of measurement used for representing the data, which can at worst lead to the discovered graph encoding an incorrect set of d-separations. This then could lead to bad results in inference. The authors also propose an approach for mitigating the sensitivity to the variance. The theoretical results are complemented by multiple empirical experiments on both synthetic and real-world data to provide evidence of the correctness of the claims.", "questions": "Could the authors explain how these results relate to the identifiability results of Peters et al. [1, Cor. 31]? They state that the ground truth graph is identifiable assuming that each f is three times differentiable and non-linear, so what is then the motivation of considering MMSE in the non-linear case if it can lead to an incorrect graph as the authors describe?\n\n[1] Jonas Peters, Joris M. Mooij, Dominik Janzing, Bernhard Schölkopf:\nCausal discovery with continuous additive noise models. J. Mach. Learn. Res. 15(1): 2009-2053 (2014)", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 2, "strengths": "While perhaps not surprising due to structure learning being a notoriously hard problem, the paper presents interesting and to my knowledge novel results on how susceptible many score functions are to scaling and the used units of measurement. One crucial observation the authors make is that poorly chosen units of measurement can lead to the discovered DAG encoding different conditional independencies than which the ground truth DAG encodes. I find this especially important since even finding a DAG from the equivalence class of the ground truth would be sufficient for many problems, but the paper demonstrates that the algorithms may fail to find such a DAG.\n\nThe empirical experiments demonstrate well the claims of the paper, i.e., model mean squared error and log-likelihood based losses leading to wrong predictions about the structure of the DAG.", "weaknesses": "My main concern is the practicality of the results for the non-linear case; see Questions. I'd be happy to consider increasing my score if the authors offer compelling arguments against my concerns.\n\nMinor + typos:\n- Line 5: Add space between \".\" and \"The\"\n- Table 1 is a bit hard to understand for now. I would add a short explanation in the caption about what the reported probabilities represent (i.e., did the outputted prediction match your expectation).\n- Sec. 3.1: The domain of the function $f$ should probably not be the set of parents but instead the power set of the nodes\n- End of Sec. 2.2: Missing period after \"GraN-DAG (Lachapelle et al., 2020)\"\n- I'd emphasize that the DAG encodes conditional independencies in Sec. 2, not just independencies", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)