Beware of the Simulated DAG! Causal Discovery Benchmarks May Be Easy to Game

1 Introduction . Causal structure learning aims to infer a causal model from data . Academic disciplines anywhere from biology , medicine , finance , to machine learning are interested in causal models [ Rothman et al. , 2008 , Imbens and Rubin , 2015 , Sanford and Moosa , 2012 , Schölkopf , 2019 ] . Causal models not only describe the observational joint distribution of variables but also formalize predictions under interventions and counterfactuals [ Spirtes et al. , 2000 , Pearl , 2009 , Peters et al. , 2017 ] . Directed acyclic graphs ( DAGs ) are common to represent causal structure : nodes represent variables and directed edges point from cause to effect representing the causal relationships . This graphical representation rests on assumptions which have been critically questioned , for example by Dawid [ 2010 ] . Inferring causal structure from observational data is difficult : Often we can only identify the DAG up to its Markov equivalence class ( MEC ) and finding high-scoring DAGs is NP-hard [ Chickering , 1996 , Chickering et al. , 2004 ] . Here , we focus on learning the DAG of linear additive noise models ( ANM ) . Data scale and marginal variance may carry information about the data generating process . This information can dominate benchmarking results , such as , for example , the outcome of the NeurIPS Causality 4 Climate competition [ Runge et al. , 2020 ] . Here , the magnitude of regression coefficients was informative about the existence of causal links such that ordinary regression-based methods on raw data outperformed causal discovery algorithms [ Weichwald et al. , 2020 ] . Multiple prior works 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . state the importance of data scale for structure learning either implicitly or explicitly . Structure identification by ICA-LiNGAM [ Shimizu et al. , 2006 ] , for example , is susceptible to rescaling of the variables . This motivated the development of DirectLiNGAM [ Shimizu et al. , 2011 ] , a scale-invariant causal discovery algorithm for linear non-Gaussian models . The causal structure of ANMs is proven to be identifiable given the noise scale ( cf . Section 2.2 ) . Yet , such identifiability results require knowledge about the ground-truth data scale . Simulated DAGs may be identifiable from marginal variances under generic parameter distributions . An instructive example is the causal graph A → B with structural equations A = NA and B = wA+NB with w 6= 0 and independent zero-centered noise variables NA , NB . The mean squared error ( MSE ) of a model X → Y is given by MSE ( X → Y ) = Var ( X ) + Var ( Y |X ) . It holds that MSE ( A→ B ) < MSE ( B → A ) ⇐⇒ Var ( A ) < Var ( B ) ⇐⇒ ( 1−w2 ) Var ( NA ) < Var ( NB ) ( see Appendix A ) . Deciding the directionality of the edge between A and B based on the MSE amounts to inferring an edge from the lower-variance variable to the higher-variance variable . For error variances Var ( NA ) ≤ Var ( NB ) and any non-zero edge weight w , the MSE-based inference is correct . This resembles known scale-based identifiability results based on equal or monotonically increasing error variances [ Peters and Bühlmann , 2014 , Park , 2020 ] . However , if the observations ofA were multiplied by a sufficiently large constant , the MSE-based inference would wrongfully conclude that A← B . This is problematic since simply choosing our units of measurement differently may change the scale and variance of A. Arguably , this is often the case for observations from real-world systems : There is no canonical choice as to whether we should pick meters or yards for distances , gram or kilogram for weights , or yuan or dollar as currency . A researcher can not rely on obtaining the same results for different measurement scales or after re-scaling the data when applying any method that leverages the data scale ( examples include Peters and Bühlmann [ 2014 ] , Park [ 2020 ] , or Zheng et al . [ 2018 ] , who employ the least squares loss studied by Loh and Bühlmann [ 2014 ] ) . Continuous causal structure learning algorithms optimize model fit under a differentiable acyclicity constraint [ Zheng et al. , 2018 ] . This allows for the use of continuous optimization and avoids the explicit combinatorial traversal of possible causal structures . This idea has found numerous applications and extensions [ Lachapelle et al. , 2019 , Lee et al. , 2019 , Ng et al. , 2020 , Yu et al. , 2019 , Brouillard et al. , 2020 , Pamfil et al. , 2020 , Wei et al. , 2020 , Zheng et al. , 2020 , Bhattacharya et al. , 2021 ] ; Vowels et al . [ 2021 ] provide a review . NOTEARS [ Zheng et al. , 2018 ] uses the MSE with reference to Loh and Bühlmann [ 2014 ] , while GOLEM [ Ng et al. , 2020 ] assesses model fit by the penalized likelihood assuming a jointly Gaussian model . On simulated data and across noise distributions , both methods recover graphs that are remarkably close to the ground-truth causal graph in structural intervention distance ( SID ) and structural hamming distance ( SHD ) . We agree with the original authors that these empirical findings , especially under model misspecification and given the non-convex loss landscape , may seem surprising at first . Here , we investigate the performance under data standardization and explain how the causal order is ( partially ) identifiable from the raw data scale alone in common generically simulated benchmarking data . Contribution . We show that causal structure drives the marginal variances of nodes in an ANM and can lead to ( partial ) identifiability . The pattern in marginal variances is dominant in ANM benchmark simulations with edge coefficients drawn identically and independently . We introduce varsortability as a measure of the information the data scale carries about the causal structure . We argue that high varsortability affects the optimization procedures of continuous structure learning algorithms . Our experiments demonstrate that varsortability dominates the optimization and helps achieve state-ofthe-art performance provided the ground-truth data scale . Data standardization or an unknown data scale remove this information and the same algorithms fail to recover the ground-truth DAG . Even methods using a score-equivalent likelihood criterion ( GOLEM ) recover neither ground-truth DAG nor its MEC on standardized data . To illustrate that recent benchmark results depend heavily on high varsortability , we provide a simple baseline method that exploits increasing marginal variances to achieve state-of-the-art results on these benchmarks . We thereby provide an explanation for the unexpected performance of recent continuous structure learning algorithms in identifying the true DAG . Neither algorithm dominates on raw or standardized observations of the analyzed real-world data . We show how , even if data is standardized and even in non-linear ANMs , a causal discovery benchmark may be gamed due to covariance patterns . Consequently , recent benchmark results may not transfer to ( real-world ) settings where the correct data scale is unknown or where edge weights are not drawn independent and identically distributed ( iid ) . We conclude that structure learning benchmarks on ANMs with generically sampled parameters may be distorted due to unexpected and perhaps unintended regularity patterns in the data . 2 Background . 2.1 Model Class . We consider acyclic linear additive noise models . Single observations are denoted by x ( i ) ∈ Rd where x ( i ) j denotes the j th dimension of the ith iid observation of random vector X = [ X1 , ... , Xd ] > . All observations are stacked as X = [ x ( 1 ) , ... , x ( n ) ] > ∈ Rn×d and xj ∈ Rn refers to the jth column of X. Analogously , n ( i ) denotes the corresponding ith iid observation of the random noise variable N = [ N1 , ... , Nd ] > with independent zero-centred components . The linear effect of variable Xk on Xj is denoted by wk→j = wkj . The causal structure corresponding to the adjacency matrix W = [ wkj ] k , j=1 , ... , d with columns wj = [ wk→j ] k=1 , ... , d ∈ Rd can be represented by a directed acyclic graph G = ( VG , EG ) with vertices VG = { 1 , ... , d } and edges EG = { ( k , j ) : wk→j 6= 0 } . Edges can be represented by an adjacency matrix E such that the ( k , j ) th entry of El is non-zero if and only if a directed path of length l from k to j exists in G. For a given graph , the parents of j are denoted by PA ( j ) . The structural causal model is X =W > X +N . 2.2 Identifiability of Additive Noise Models . Identifiability of the causal structure or its MEC requires causal assumptions . Under causal faithfulness and Markov assumptions , the causal graph can be recovered up to its MEC [ Chickering , 1995 , Spirtes et al. , 2000 ] . Faithfulness , however , is untestable [ Zhang and Spirtes , 2008 ] . Shimizu et al . [ 2006 ] show that under the assumptions of no unobserved confounders , faithfulness , linearity , and non-Gaussian additive noise , the causal graph can be recovered from data . Hoyer et al . [ 2009 ] show that this holds for any noise distribution under the assumption of strict non-linearity . This finding is generalized to post-nonlinear functions by Zhang and Hyvarinen [ 2009 ] . Peters and Bühlmann [ 2014 ] prove that the causal structure of a linear causal model with Gaussian noise is identifiable if the error variances are equal or known . Any unknown re-scaling of the data breaks this condition . For the case of linear structural causal models , Loh and Bühlmann [ 2014 ] provide a framework for DAG estimation based on a noise variance-weighted least squares score function . For ANMs , they give conditions under which the general Gaussian case can be identified via approximating it by the equal noise-variance case given knowledge of the ( approximate ) noise scale . Finally , subsuming further prior results on ( linear ) ANMs [ Hoyer et al. , 2009 , Ghoshal and Honorio , 2017 , 2018 , Chen et al. , 2019 ] , Park [ 2020 ] shows that the causal structure is identifiable under regularity conditions on the conditional variances along the causal order . In particular , identifiability holds if the error variances of nodes are weakly monotonically increasing along the causal order . 2.3 Structure Learning Algorithms . Combinatorial structure learning algorithms ( such as PC , FGES , DirectLiNGAM ) separately solve the combinatorial problem of searching over structures and finding the optimal parameters for each structure . To remain computationally feasible , the search space of potential structures is often restricted or traversed according to a heuristic . One can , for example , carefully choose which conditional independence statements to evaluate in constraint-based algorithms , or employ greedy ( equivalence class ) search in score-based algorithms . In our experiments , we consider PC [ Spirtes and Glymour , 1991 ] , FGES [ Meek , 1997 , Chickering , 2002b ] , DirectLiNGAM [ Shimizu et al. , 2011 ] , and a greedy DAG search ( GDS ) algorithm MSE-GDS that greedily includes those edges that reduce the MSE the most . For details see Appendix D. Continuous structure learning algorithms ( such as NOTEARS and GOLEM ) employ continuous optimization to simultaneously optimize over structures and parameters . As a first step towards expressing causal structure learning as a continuous optimization problem , Aragam and Zhou [ 2015 ] propose l1-regularization instead of the conventional l0-penalty for model selection . Zheng et al . [ 2018 ] propose a differentiable acyclicity constraint , allowing for end-to-end optimization of score functions over graph adjacency matrices . We examine and compare the continuous structure learning algorithms NOTEARS [ Zheng et al. , 2018 ] and GOLEM [ Ng et al. , 2020 ] . For details see Appendix D. NOTEARS [ Zheng et al. , 2018 ] minimizes the MSE between observations and model predictions subject to a hard acyclicity constraint . The MSE with respect to W on observations X is defined as MSEX ( W ) = 1 n‖X−XW‖ 2 2 where ‖ · ‖2 = ‖ · ‖F denotes the Frobenius norm . GOLEM [ Ng et al. , 2020 ] performs maximum likelihood estimation ( MLE ) under the assumption of a Gaussian distribution with equal ( EV ) or non-equal ( NV ) noise variances . There are soft acyclicity and sparsity constraints . The unnormalized negative likelihood-parts of the objective function are LEV ( W , X ) = log ( MSEX ( W ) ) and LNV ( W , X ) = ∑d j=1 log ( 1 n‖xj − Xwj‖ 2 2 ) , respectively , omitting a − log ( |det ( I −W ) | ) term that vanishes when W represents a DAG [ Ng et al. , 2020 ] . To ease notation , we sometimes drop the explicit reference to X when referring to MSE , LEV , LNV .

Summary: This submission is about causal structure learning and how some simulated directed acyclic graphs (that are used for benchmarking structure learning approaches, and that are generally used for causal representations) can have properties (in particular, marginal variances and also covariance patterns) that could render some structure learning algorithms more efficient, even though that might be not necessarily replicable for real-world data; that is highlighted a concern.  The authors describe a measure, varsortability, that relates to a relationship between marginal variances and the causal structure in a causal graph. They discuss how identifiability might be affected by that. The submission shows how, and explains potentially why, the varsortability of causal graphs and the data simulated/generated from them might interplay with the performance of some structure learning algorithms (in particular, continuous structure learning methods), and how that performance might be explained and depend (potentially, inadvertently) up to some significant extent by the varsortability (in particular, for simulated DAGs). The submission covers that through some theoretical analysis and empirical studies. Because marginal variances and varsortability depend on the choice of the scales of variables, the submission raises concerns about the usefulness/interpretation of existing benchmarks (particularly if no experiments with data standardisation are performed) and their results for existing structure learning algorithms. The submission also presents a baseline structure learning method that relies on marginal variances; that method can be used as a baseline for benchmarking. The theoretical results in the submission is for linear additive noise models; the empirical experiments cover both some linear and non-linear settings. The empirical experiments (and results) cover simulated and also some real-world data. In addition to discussing the points that relate to marginal variances, the submission also has a part that relates to covariance patterns and how those might influence the benchmarking in a similar way. The paper finishes with a conclusion and discussion, including some recommendations/comments as part of that.