Standardizing Structural Causal Models
Authors: Weronika Ormaniec
Scott Sussex
Lars Lorch
Bernhard Schölkopf
Andreas Krause
Venue: ICLR 2025 Poster (2025)
Keywords: Causality
Causal Discovery
Structural Causal Model
Standardized Structural Causal Model
Variance Artifact
Covariance Artifact
Simulation
Benchmark
Primary area: causal reasoning
TL;DR: Model for benchmarking causal structure learning methods that does not induce variance and covariance artifacts in datasets.

Abstract: Synthetic datasets generated by structural causal models (SCMs) are commonly used for benchmarking causal structure learning algorithms. However, the variances and pairwise correlations in SCM data tend to increase along the causal ordering. Several popular algorithms exploit these artifacts, possibly leading to conclusions that do not generalize to real-world settings. Existing metrics like $\operatorname{Var}$-sortability and $\operatorname{R^2}$-sortability quantify these patterns, but they do not provide tools to remedy them. To address this, we propose internally-standardized structural causal models (iSCMs), a modification of SCMs that introduces a standardization operation at each variable during the generative process. By construction, iSCMs are not $\operatorname{Var}$-sortable. We also find empirical evidence that they are mostly not $\operatorname{R^2}$-sortable for commonly-used graph families. Moreover, contrary to the post-hoc standardization of data generated by standard SCMs, we prove that linear iSCMs are less identifiable from prior knowledge on the weights and do not collapse to deterministic relationships in large systems, which may make iSCMs a useful model in causal inference beyond the benchmarking problem studied here. Our code is publicly available at: https://github.com/werkaaa/iscm.

Published as a conference paper at ICLR 2025
STANDARDIZING STRUCTURAL CAUSAL MODELS
Weronika Ormaniee* ETH Zürich, Switzerland wormaniec@ethz.co
Scott Sussex" ETH Zürich, Switzerland s ssussex@ethz.c
Lars Lorch" ETH Zürich, Switzerland 11 11orch@ethz.cr ch
Bernhard Schölkopf
Andreas Krause ETH Zürich, Switzerland krauseaßethz. ch
MPI for Intelligent Systems, Tübingen, Germany bs bs@tuebingen.nue wwp.pinnnnn.gg.......mmmm. .e..de
ABSTRACT
Synthetic datasets generated by structural causal models (SCMs) are commonly used for benchmarking causal structure learning algorithms. However, the vari- ances and pairwise correlations in SCM data tend to increase along the causal ordering. Several popular algorithms exploit these artifacts, possibly leading conclusions that do not generalize to real-world settings. Existing metrics like Var- sortability and -o-sortayility quantify these patterns, but they do not provide tools to remedy them. To address this, we propose causal models (iSCMs), a modification of SCMs that introduces standardization operation at each variable during the generative process. By construction, iSCMs are not Var-sortable. We also find empirical evidence that they are mostly not Re-sortable for commonly-usee graph families. Moreover, contrary to the post-hoc standardization of data generated by standard SCMs, we prove that linear iSCMs are less identifiable from prior knowledge on the weights and do not collapse to deterministic relationships in large systems, which may make iSCMs a useful model in causal inference beyond the benchmarking problem studied here. Our code is publicly available at: https://github. cwwwwwrk/atteeeeeeoe com/werkaas/isem.
structural
INTRODUCTION
Predicting the effects of interventions and policy decisions requires reasoning about causality. Con- sequently, scientific fields ranging from biology and earth sciences to economics and statistics are interested in modeling causal structure (Pearl, 2009; Maathuis et al., 2010; Imbens and Rubin, 2015; Runge et al., 2019). A wide array of causal discovery algorithms has been proposed with the goal of inferring causal structure from data (c.g.. Squires and Uhler, 2022; Vowels et al., 2022). How- ever, benchmarking these algorithms is challenging, since real-world datasets with agreed-upon, ground- -tuut truth causal structure are rare (e.g.. Sachs et al., 2005; see Mooij et al., 2020). The community predominantly relies synthetic data for evaluating structure learning algorithms, where obser- vations are generated according to a predetermined causal structure and system mechanisms. The inferred causal structures then be directly compared the ground truth. To generate synthetic data, it is common practice to sample from structural causal models with additive noise (SCMs) (Reisach al., 2021). Unless stated otherwise, this work considers SCMs which the variance scale of the additive noise is the same for all variables, a typical simplification made in benchmarking.
Under common benchmarking practices, synthetic datasets generated by SCMs contain patterns that are directly exploitable to make structure discovery easier. We will refer to such patterns as artifacts. In SCMs, the pairwise correlations between variables tend increase along the causal ordering, since variance builds up downstream and, result, the proportion of the variance driven by the additive noise vanishes (Figure Ta). Reisach et al. (2024) characterize this phenomenon through an increase of the coefficients of determination (R²) of the variables regressed on all others. Crucially, this artifact occurs both in the raw data and when shifting and scaling (standardizing) the variables to have zero mean and unit variance. One of the implications is that downstream causal dependencies in SCMs become effectively deterministic, especially in large-scale systems. As Reisach et al. (2024) demonstrate, simple causal discovery baselines can perform competitively on benchmarks of this kind by directly exploiting this phenomenon. This makes SCMs alone in their general definition
Equull contribution.

---

Published as a conference paper at ICLR 2025
222
10
J2
10
1
2
Z
10
F2
Iy
T10
lp|:
0.75
0.86
0.98
0.98
lpl: 0.75
0.75
0.75
0.75
(a) Standardized SCM
(b) iSCM
Standardizing SCMs two ways. Generative process for a chain graph of (a) standard SCMs, with data x standardized post-hoc, and (b) SCMs with standardization performed during the generative process (iSCMs). Dashed arrows indicate z-standardization. Solid arrows indicate linear functions with weights from Unif (0.5, 2.0] and additive noise from (O,,, 1). We report absolute correlations |e| of two consecutive observed variables, (a) z and -5+1 or (b) Tj and Zj+1, averaged over 100,000 models. standard SCMs (a), correlations tend to increase along the causal ordering.
Figure 1:
possibly insufficient for benchmarking. Ultimately, evaluating on synthetic data with these pattems could lead to conclusions that do not generalize as expected to real-world scenarios.
In this work, propose a simple modification of SCMs that stabilizes the data gata-generating process and thereby removes exploitable covariance artifacts. Our models, denoted internally-standardiaa standardized SCMs (iSCMs), introduce standardization operation at each variable during the generative process (Figure 1b). In Section 4, we provide a theoretical motivation for this idea by studying linear iSCMs. We prove that, contrary SCMs, the causal dependencies of iSCMs under mild assumptions never collapse to deterministic mechanisms as the graph size becomes large. Moreover, we formalize the correlation artifact commonly observed in benchmarks by proving that linear SCM structures in Markov equivalence class (MEC) are partially identifiable for certain graph classes, given weak prior knowledge on the weight distribution of the ground-truth SCM. Most importantly. we show that this is not the case for the corresponding iSCMs. In Section 5, we empirically demonstrate that the baselines proposed in Reisach al. (2021; 2024) are unable to exploit covariance artifacts in iSCMs, while practical classes of causal discovery algorithms are still able to learn causal structures in both linear and nonlinear systems. Our findings reveal that SCM artifacts affect structure learning both positively and negatively, making iSCMs practical tool, alongside SCMs, for disentangling the drivers causal discovery performance of different algorithms in practice.
BACKGROUND AND RELATED WORK
We begin by introducing structural causal models and the problem of causal structure learning, before discussing how synthetic data is often generated for evaluating structure learning algorithms. We then review existing works that study identifiability and patterns frequently present synthetic data.
Structural causal models A structural causal model (SCM) (Peters et al., 2017) of d variables x (Ii,...,,A .a} consists of a collection of structural assignments, each given by
(xpa())) Es),
(SCM)
where Xpa(s) C { are called the parents of z4. Here, fo are arbitrary functions, and e; are independent random variables that model exogenous noise (or unexplained variation). Together, they entail a joint probability distribution p(x) over the variables x. It is common to consider SCMs with additive noise, e.g., with linear functions fo, as given by
[x e.) w. xpa(t) + Ei,
where Wij E R denotes the weight from E pa(i) to i. The structural assignments in (SCM) induce a causal graph g (V, E. over the variables I;, which assumed to be acyclic. Specifically, the directed acyclic graph (DAG) G has vertices v; E V for every æ; and directed edge (i, j) & xr E Xpa(j). We will explicitly distinguish this DAG & and its vertices V from the variables x. The skeleton of G denotes G with all edges undirected. If the skeleton of is acyclic, we call g a forest.
Structure learning and benchmarking Given a set of i.i.d. observations from the probability distribution p(x) induced by unknown SCM, causal structure learning aims to infer the causal
2

---

Published as a conference paper at ICLR 2025
graph G underlying the SCM. In this work, we focus structure learning from observational data and only consider SCMs with no latent confounders. Because it is difficult to obtain the true ç for many real-world datasets, it is common to evaluate structure learning algorithms on synthetic data where G is known. A ubiquitous approach is to sample a DAG g, then SCM functions defined over G, and finally dataset from this SCM, with the goal of later recovering 9 from the data. It is common to consider E; with mean O and fixed variance (often 1), and for linear systems, to sample each Way uniformly and i.i.d. with support bounded away from (Shimizu et al., 2011; Peters and Bühlmann, 2014; Zheng et al., 2018; Yu et al., 2019; Lachapelle et al., 2020; Zheng et al., 2020; Ng et al., 2020; Reisach et al., 2021; Lorch et al., 2022: Reisach et al., 2024). There exist alternative benchmarking strategies with domain-specifer simulators (Schaffter et al., 2011; Dibaeinia and Sinha, 2020).
Data standardization and artifacts of SCMs Previous work shows that generating data as de- scribed above can lead to strong artifacts. Reisach et al. (2021) observe that the variance of variables tends to increase along the topological ordering of G. This leads to the Var-SSORN REGRESS baseline, which sorts variables based on their empirical variance and then performs sparse regression to infer g. Seng et al. (2024) show that structure learning algorithms minimizing an MSF-based loss (e.g.. Zheng al., 2018) can identify 9 under similar conditions. Therefore, Reisach et al. (2021) propose using standardization (Figure la) to remove this variance artifact from benchmarks. Specifically, they first sample all x; according standard SCM and then post-hoc transform the variables as
22 E
(Standardized SCM)
Var [.
such that our observations correspond to samples from p(x*). Standardization, however, only removes the variance artifact. Even in standardized SCMs, the fraction of variable's variance that is explained by all others, measured by the coefficient of determination R2, tends to increase along the topological ordering (Reisach et al., 2024). RR-SSORTNRGRESS exploits this correlation artifact analogously to SEr-SORTNREGGRES
Existing heuristics aiming to avoid the variance accumulation adjust the sampling process of fü. but they ultimately limit the causal dependencies that can be modeled, e.g.. to certain levels of correlations among the observed x (Mooij et al., 2020) or a constant proportion of variance explained by the parents Xpa(i) (Squires et al., 2022) and fail to induce data free from both artifacts (Appendix D.I.. To our knowledge, there are currently no general methods for generating SCM data without strong correlation artifacts or significant limitations on the functions fi and noise E,.
Identifiability Given a class of SCMs, there may be several SCMs with different causal graphs G that entail the same distribution p(x) (Peters et al., 2017). Thus, even with infinite observations from p(x), we may be unable to identify the causal graph G that generated the observations. However, some identifiability results are known depending on the class of functions and noise distributions of the SCMs considered. For example, among all linear SCMs (1) with Gaussian noise E ~ NO, of), the graph G can only be uniquely identified up to its MEC (Verma and Pearl, 2013). However, if the noise is Gaussian with equal variances of = o² (Peters and Bühlmann, 2014) or the noise is non non-Gaussian (Shimizu et al., 2006), 9 can be uniquely identified given p(x).
In this work, we present, to our knowledge, the first (partial) identifiability result for standardized SCMs in the linear Gaussian case. Since standardization affects the implied noise scales, existing linear Gaussian identification results, which rely on of a², longer hold when observing p(x*). Other identifiability results, e.g., based on non-Gaussian noise, do continue to hold for standardized SCMs (c.g., Shimizu et al., 2006). Our result concerns a setting with prior knowledge on the magnitudes of w in Equation (1), an assumption underlying common benchmarking practices. Under this setup, we show a stark difference in the identifiability of standardized SCMs and the iSCMs we propose, which provides novel explanation for what we empirically observe in benchmarks.
SCMS WITH INTERNAL STANDARDIZATION
3.1 DEFINITION
We propose internally-standardiee SCMs (iSCMs) as a modification to the standard data-generating process of SCMs. An iSCM (S, Pz) consists of d pairs of assignments, where for each i E {1...., d).
ft (a))))) Ex) and 5
T;-
B 2
(iSCM)
Var
3

---

Published as a conference paper at ICLR 2025
Algorithm 1 Sampling from iSCM
Input: DAG 9, noise distribution Ps.
functions {f,,-., Se}
Fi
π < topoloical ordering of g
for i 1 to d do
Tk
Ex. ~ Pe..
In: fn, (Xpa()r))) Ems))
Tr.- t Jz. Bl Blaxz]
/ /arr [TT,]
Figure 2: Causal mechanisms in iSCMs. The function fi modeling z; depends on the standard- ized Xpa(t)- Dashing indicates z-standardization.
return [+ ( Fd
D E Rd
with parents ppa(i) C { of ãे in the underlying DAG. In the above, fi are general functions, and the exogenous noise variables = [=1.... Ea] ~ Pe are jointly independent, as for SCMs. The variables I; are latent, and the variables 2; are observed. Figure illustrates the generative process. Algorithm 1 summarizes how to sample from (iSCM). If computing the population expectations and variances of x; intractable, the empirical statistics obtained from n samples can be used for standardization at each loop iteration of Algorithm 1.
Motivation By construction, iSCMs model observed variables with zero mean and unit marginal variance. Contrary standard SCMs, iSCMs avoid the accumulation of variance downstream in the causal ordering that can occur in standard SCMs (see Figure 1) through the standardization operation. Because each variable x; only depends on the standardized variables Xpu(i), the relative scales of the noise distribution Pe., and the causal mechanisms fi are the same everywhere in the system and do not change, for example, downstream in the causal ordering. The causal mechansims of iSCMs are thus scale-free, in that the local interaction of mechanism f: and noise occurs at scale independent of the position of x; in the global ordering. This property makes iSCMs particularly useful for benchmarking, where random ground-truth models are commonly generated from fixed distribution over functions fi and noise E,. Contrary to existing heuristics (Section 2), iSCMs model arbitrarily strong weak causal dependencies and levels of cause-cxplained variance.
Interventions Analogous to standard SCMs, interventions in iSCMs can be defined as modifications of the structural assignments f, in (iSCM) (Figure 2). while keeping the standardization operation based on the observational distribution. When the population statistics for standardization are intractable, we first sample observational data to obtain empirical statistics. Since we do not study interventions in this work, we defer further discussion of interventions in iSCMs to Appendix B.
Units When modeling a physical system, the functional mechanisms in standard SCMs have to account for the difference in units between the variables for the model to be unit-covariant (see Villar et al., 2023). A side-effect of internal standardization is that variables of iSCMs become unit-less, so iSCMs obey the passive symmetry of unit covariance by construction. Therefore, iSCMs naturally model both unit-less quantities and variables measured in different units, which can make them useful beyond benchmarking. Learned iSCMs would be invariant the units chosen by the experimenter, similar to the physical world being independent of the mathematical models chosen to describe it.
3.2 IMPLIED SCMS
It is natural to investigate whether SCMs can generate the same observations as standardized SCMs or iSCMs, given the same causal graph G and exogenous variables e. In other words, can standardized SCMs and iSCMs be written as SCMs? For both models, the answer is yes. Specifically, we can express the generative process of x* in (Standardized SCM) and x in (iSCM) as
x 9 (xpa(t) + +
and
T g. (pa(t)) + 0.5..
(2)
respectively, by moving the standardization operations into the causal mechanisms of the observables but leaving the DAG G and the variables e unchanged. Appendix A describes how to construct these implied causal mechanisms 9f and 9. and implied noise scales 0 and 0,. We refer to the above SCM form of a standardized SCM or an iSCM with additive noise as their implied (SCM) model. Corresooniigly, the implied SCMs have zero mean and unit variance. The notion of implied
4

---

Published as a conference paper at ICLR 2025
SCMs is powerful, because it enables to analyze standardized SCMs and iSCMs SCMs, and it sheds light on the performance of structure learning algorithms that assume unstandardized SCMs to underlie the generative process of the data (e.g.. Shimizu et al., 2011; Zheng et al., 2018; Yu et al., 2019; Lachapelle et al., 2020; Zheng et al., 2020).
To provide first characterization of standardized SCMs and iSCMs, our theoretical analyses focus on systems where f; are linear functions with additive, zero-mean noise as given by Equation (1). As stepping stone for this analysis, we use an analytical expression for the covariance of linear SCMs, whose variables have unit variance by construction, without any form of standardization:
Lemma (Covariance in linear SCMs with unit marginal variances). Let x be modeled by a linear SCM defined by (1) with DAG that satisfies Var[az]] 1. Then, the covariance Cov (ra, syyl is the sum of products of the weights along all unblocked paths between the nodes of ay and x in G. Specifically, for any i E {I.... d3 such that + j. holds that
Cov(a,, [.)]
ΣE II
Pr E::ss (l.m)Epset
where Pjesi all unblocked paths from 3. x4 in G, and (1,m) Pjesi indicates that the directed edge (1, m) is part of the path Pjesi-
This lemma, also called the trek rule, is originally due to Wright (1934). We give a proof in Appendix C.2. Since the implied SCMs of linear standardized SCMs and iSCMs are linear SCMs, the setting of Lemma 1 applies precisely to the SCM forms of both models. Thus, Lemma 1 enables us to study the covariances in standardized SCMs and iSCMs, and we show next, derive conditions for the (non)identifiabilityy of their DAGs 9 from the observational distribution.
4 ANALYSIS
In this section, we give two theoretical results that support the suitability of iSCMs over standard SCMs for causal discovery benchmarking. First, we prove the general case of Figure Contrary to standardized SCMs, iSCMs do not degenerate towards deterministic implied SCM mechanisms in deep graphs. Moreover, we prove that the DAGs of linear iSCMs cannot be identified beyond their MEC, assuming the DAG is forest, even if the support of w is known. Crucially, also show that this is not generally true for standardized SCMs. This suggests that algorithms can less casily game benchmarks based on linear iSCMs when knowing the data-generating process. For all results, we consider linear SCMs (1) with zero-mean additive noise and equal noise variances. All results are at the population level, so assume we know p(x*) p(x). Proofs given in Appendix C.
4.1 BEHAVIOR WITH INCREASING GRAPH DEPTH
Standardized SCMs tend towards increasing correlations between adjacent nodes down the topological ordering. This correlation artifact makes standardized SCMs problematic for benchmarking, because it may not be property assumptions on w, that the dependencies in standardized SCMs become deterministic with increasing graph depth. This implies that any exogenous variation e; vanishes lower down the system. Unless prior domain knowledge leads us to assume this holds in applications of interest, it may not be desirable to implicitly bias structure learning benchmarks towards such systems. For example, if the causal ordering represents time (Pamfil et al., 2020), the mechanisms of standardized SCMs are unable model or characterize time-invariant or stable processes. Moreover, we expect causal mechanisms to be independent (Schölkopf, 2022), the qualitative behavior of causal mechanism should not provide information about its position in the topological ordering relative to other mechanisms, as it would in SCMs. Reisach al. (2024) show that basclines like ----SRRTNNGGRRSS can perform competitively on benchmarks by exploiting this artifact (Section 2).
expect to underlie real data. Reisach et al. (2024) show, under some
iSCMs do not tend towards determinism with increasing graph depth (Figure lb). In standardized SCMs, the correlations increase downstream, because the marginal variances of the underlying SCM increase with node depth, while the variance scale is fixed (Reisach al., 2021). Thus, for large i, the variance scale of x;-1 becomes large relative to the scale of E,, and the correlation of x; and x;-1 tends towards 1. Since xa and 2ن_1 are just standardized versions of these variables, they maintain
5

---

Published as a conference paper at ICLR 2025
the same correlation. iSCMs avoid this by standardizing internally, which scales the variance of any parents in a mechanism fo to 1, modulating the relative variance of E, and Xpa(i). In the following, we formalize this result for general graphs by bounding the fraction of cause explained variance (CEV). The fraction of CEV for xy is the proportion of Var [ax] explained by its causal parents and given by
Var [a; - E| xpacoll
CEV =
(4)
Var[z;)
The following results shows that we can bound the fraction of CEV for any variable in a linear iSCM:
Theorem 2 (Bound on CEV, in linear iSCMs). Let x be modeled by a linear iSCM (1) with DAG g and additive noise of equal variances Var(ez] o². Suppose any node in G has at most m parents and w = ma (I.....) Wiys Then, for any E {1,.... d). the fraction of CEV Ty is bounded as
CEV 1
m 222 02
Since the fraction of CEV bounded, iSCMs are guaranteed not to collapse to determinism in large systems, alleviating several of the concerns with (standardized) SCMs discussed above.
4.2 IDENTIFABILLTTY
Figure la illustrates that the pairwise correlations SCMs over chain graphs depend on the position in the topological ordering. This can allow algorithms like R RERRRNRREEEESSS to infer the graph. By contrast, Figure 1b shows that iSCMs do no exhibit this pattern, with correlations between variables not increasing the identifiability of any part of the system.
B
3/3
B
Q
iii
11
322
333
In the following, we formalize this phenomenon for forests, that is, all DAGS with acyclic skeletons (Section 2). Specif- ically, we prove two results concerning the identifiability of the DAG G from the observational distribution, for stan- dardized SCMs and iSCMs. This makes our finding the first identifiability result for standardized SCMs. While not ev- ery DAG is forest, DAGs have forests as subgraphs and resemble forests as sparsity increases, thus providing us with intuition for generally sparse systems (e.g., Alon and Spencer, 2016, Chapter 11).
iiii)
1
x2
F3
(a) DAGs with edge weights and /
1.0
1.00
0.71
0.63
011 0.71
1.00
0.89
0.8
0.63
0.89
1.00
3
Our first result leverages the observation that, for standardized SCMs, many DAGs in an MEC are infcasible given p(x*) when their edge directions tion of increasing absolute covariance. To illustrate this idea, suppose our goal is to distinguish between the DAGs in the MEC Č = {i), (ii). (iii) in Figure 3a. We overload notation and denote the weights of the edges a and | regardless of orientation. For standardized SCMs, we can apply Lemma to the implied SCM of graph (i) to obtain the covariances
(b) Cov. matrix of linear iSCMs
not consistent with the direc-
Figure 3: iSCMs with the same covariance matrix. (a) DAGs in an MEC with the same edge weights (b) Covariance matrix for all linear iSCMs in (a) when = 1, 3 2.
Va-+i and See Appendix C.4.1. Together, both expressions imply that standardized SCMs with DAG (i) satisfy
Cov[zf -
Cov[zj. = BV +l(0++1)+1
|Cov[2f, xلحه < [Cor[2g, xgll
a=+1 <
(5)
If |8 2 1, then the right-hand side of Equation (5) is always true. In this case, the absolute covariance increases from J to I3 in all standardized SCMs with DAG (i). By symmetry, the covariance in SCMs with DAG (ii)) increases from I3 to 2 when |a| 2 1. Therefore, if both weights are greater than 1, the absolute covariance increases downstream in all SCMs of (i) and (ii). This implies that, among (i) and (ii), only the DAG whose edges align with the covariance ordering in p(x*) can induce p(x*). Irrespectively, the DAG (ii) remains plausible. We can extend the intuition of this 3-variable example to identify almost all edges in any forest MEC:

---

Published as a conference paper at ICLR 2025
Theorem 3 (Partial identifiability of standardized linear SCMs with forest DAGs), Let be modeled by a standardized linear SCM (1) with forest DAG G, additive noise of equal variances Varle,] = o². and [wa.g] for all i E pa(j). Then, given p(x²) and the partially directed graph & representing the MEC of we can identify all but at most one edge of the true DAG G in each undirected connected component of the MEC g.
Our proof of Theorem 3 considers each undirected component separately from the rest of the MEC Č. Hence, the identifiability result extends undirected tree components of arbitrary, non-forest MECs as well. Theorem 3 shows that, when using standardized SCM data for benchmarking, algorithms can use pairwise correlations to orient additional edges correctly. The weights assumption of Theorem is relevant to causal discovery benchmarking, because weights are often sampled i..dd from intervals bounded away from 0 (Section 2). Hence, empirical evaluations may render standardized linear SCMs identifiable only through the design of their weights distribution. In the following, we show that, under similar conditions, iSCMs are more difficult to identify from their MEC. In the 3-variable example above, we can show that the observational distribution of iSCMs is the same for all DAGs (i). (ii), and (iii when the weights a and / are shared over the corresponding edges in the MEC (Figure 3b; see Appendix C.4.1). This result generalizes to forests:
Theorem 4 (Nonidentifiability of linear Gaussian iSCMs with forest DAGs). Let & be modeled by a linear iSCM (1) with forest DAG G and additive Gaussian noise of equal variances Var(e:]- Then, for every DAG G' in the MEC of G. there exists a linear iSCM with DAG G' that has the same observational distribution as X, the same noise variances, and the same weights on the corresponding edges in the MEC.
Our proof consists of showing that the covariance matrices of these systems are equal. For linear Gaussian iSCMs, this then implies that their observational distributions are identical. Theorem thus shows that additional knowledge of the weight distribution in a benchmark does not allow identifying any additional edges beyond the MEC. By contrast, Theorem 3 shows that, for standardized SCMs, dower-bounding the weight magnitudes is sufficient for identifying most of the graph from its MEC. Without standardization, G is fully identified from its observational distribution under even weaker assumptions (Peters and Bühlmann, 2014). Importantly, Theorem 4 does not generalize arbitrary graphs beyond forests. Appendix C.4.2 provides counterexample involving a 3-node skeleton. As we study in the next section, this implies that causal structure can still be learned from nontrivial iSCMs. However, DAGs in benchmarks are often sparse, so we expect the implications of our identifiability results to capture relevant parts of empirical phenomena in benchmarking settings.
5 EXPERIMENTAL RESULTS
Our analyses suggest that iSCMs address shortcomings of naive standardization, in particular, when sampling each fo and E; from the same distribution, as common in benchmarking. In this section, we now also provide empirical evidence that iSCMs do not contain the covariance artifacts of SCMs. This makes iSCMs a useful tool for disentangling, alongside SCMs, which data patterns drive causal dis- covery in practice. To show this, we benchmark the SORTNR structure learning algorithms to gain insights into how their performance varies when benchmarked on standardized SCMs and iSCMs. Appendix E provides complete details of the experimental setup.
baselines and suite of popular
5.1 -2-SORTABIL
Reisach et al. (2024) introduce the -R-sortabil metric to evaluate the correlation artifact underlying dataset. R sortabiilyyy measures (rescaled [0, 1)) the association between the variables' causal ordering and the R2 coefficients obtained from regressing each variable onto all others (Appendix D.2). An Ro-sortability close to 0.5 suggests that the R coefficients from regression contain no information about the causal ordering. Conversely, an R²'sortabilitty of 0 or implics that the causal ordering can be completely identified from this information. The metric gives rise to the R² R2-SORTNREGRER baseline described in Section 2. Reisach et al. (2024) show that R² -sottabbiyy in SCMs is driven by an interplay of graph connectivity and the weight distribution of ft.
Figure 4 summarizes the Rossorabbility statisties for linear SCM and iSCM data. We write ER(d, k) and SF(d, k) to denote Erdös- Erdös-Rényi and scale-free graphs of size d and (expected) degree k respec- tively (see Appendix E.2 for details). We find that iSCMs generate datasets that not R² Rorttalb
7

---

Published as a conference paper at ICLR 2025
Standardizel SCM
ISCM
0.8)
=
+11.3,3.0 3.0
100,5, 2.0]
#11.3,3.0|
R2 (>-sorabbility
11.00
R22 R22.ortabilltty
1.000
0.75-
0.75
0.50
0.50
100
200
100
200
100
200
100
200
100
200
100
200
d
d
(a) ER(d, 2)
(b) SF 2)
Figure 4: -ssortaability for different graph sizes. Linear standardized SCMs and iSCMs with Er ~ (O,, 1) and weights drawn from uniform distributions with supports given above each plot. For every model, we evaluate 100 systems and n = =1000 samples each. Lines and shaded regions denote mean and standard deviation. Datasets that satisfy - 0.5 (dashed) not Re-sortable.
sorttiibyyyy = 0.5) and thus artifact-free while sampling over common graph structures (e.g.. Zheng et al., 2018; Yu al., 2019; Reisach et al., 2021). Conversely, standardized SCMs generate datasets that are strongly Re-sortable (R -orttbiryyyyy - 0.5| >> 0). Since R can be ex- ploited for causal discovery. iSCM data serves as test for evaluating whether algorithms utilize any data properties beyond the association between R.² and the causal ordering in SCMs. Our results do not exclude the possibility of iSCM configurations that still produce R²-sortable datasets. However, we show empirically that, for commonly-used G. Pe. and w, iSCM datasets are not R Rorttbee with high probability. Appendix D. I reports the sortability metries of the existing heuristics in Section 2, show- ing that neither mitigate both Var- and Rtssortability, Appendix F provides results for denser graphs.
5.2 STRUCTURE LEARNING
Under the same weight and noise distributions, standardized SCMs and iSCMs have different implied SCMs and generate qualitatively different datasets. Here, we study how this affects causal structure learning in practice. For this, we evaluate Var- and R² RE-ERERRNEGRESS (SR) (Reisach et al., 2021; 2024) as well as an SR variant that uses random ordering (Random SR). In addition, we evaluate representative algorithms from several approaches to learning structure from (co)variance information. NOTEARS by Zheng et al. (2018) leverages continuous optimization minimize MSE loss, which is affected by noise scaling (Loh and Bühlmann, 2014; Seng et al., 2024). GOLEM (Ng et al., 2020), similar NOTEARS, formulates causal discovery as continuous optimization problem. Its EV and NV versions assume equal and potentially unequal noise scales, respectively. CAM (Bühlmann et al., 2014) searches over causal orderings and performs sparse nonlinear regression find the parents, while also estimating the noise scales. PC (Spirtes and Glymour, 1991) and GES (Chickering, 2002) approaches based on statistical independence testing and greedy search, respectively. Finally, AVICI by Lorch et al. (2022) predicts graphs using a model pretrained on simulated data and is thus optimized to exploit any artifacts that improve predictive accuracy. To investigate its susceptibility to artifacts, we evaluate the public model checkpoints trained on standardized SCMs.
Figure summarizes the results for linear and nonlinear systems with Gaussian noise (see Figure 18, Appendix F for non-Gaussian systems). The nonlinear mechanisms fi are samples from Gaussian process with squared exponential kernel. As expected, Var-SORIN performs better SCMs
performs
best when SCMs are not standardized. Likewise, R²=SOORTN and standardized SCMs, as iSCMs have Ro-sortability close to 0.5 (Section 5.1). AVICI shows the same trend, suggesting it may indeed be exploiting the correlation artifacts present in its training distribution. Like Reisach et al. (2021), we find that NOTEARS performs best on unstandardized data. However, and more interestingly. NOTEARS also performs better iSCMs than standardized SCMs, especially in linear and larger systems. As we investigate later on, this gap may be explained by the fact that the implied models of standardized SCMs violate the assumptions of NOTEARS more strongly than iSCMs. Overall, we find GOLEM-EV shows the same patterns as NOTEARS, severely underperforming on standardized SCMs and slightly improving the predictive accuracy on iSCMs. CAM and GOLEM-NV, which do not assume equal noise scales, perform equally well or better on standardized data, respectively, and generally better on iSCMs. The poor performance of GOLEM-NV
8

---

Published as a conference paper at ICLR 2025
SCM
Standardized SCM
iSCM
F1
LINEAR
NONLINEAR
DR 2)
0.50-
0.25
0.00
ER(100, 2)
0.75
0.50
0.25-
0.00
NOTBARS
GOLEM
GOLEX
AVICE
Pc
Ghs
CAM
Var SR
Re-sa
Rand -8n
NOTEARS
GOLUM
GOLRM
AViCE
GDS
CAM
Var-
R²-sa
Rand -nn
EV
NV
DV
NV
Figure 5: Structure learning performance on SCM and iSCM data. Fl scores for recovering the edges of the true graph. Box plots show median and interquartile range (IQR). Whiskers extend the largest value inside 1.5x IIQ from the boxes. Left (right) column shows results for linear (nonlinear) causal mechanisms with additive noise E; NN 1) and Wi, Unif=40.5.2000 (Appendix E). For every model, we evaluate 20 systems each using 1000 data points.
on unstandardized SCMs was also observed by Reisach et al. (2021). In addition, for approaches based on discrete search, we find that, in particular on large systems, the PC and GES algorithms perform better on iSCMs. Overall, performance differences tend to be more pronounced for linear systems, where the downstream variance accumulation in SCMs is unbounded. Appendix F reports the results for the structural Hamming distance (SHD) and different weight ranges.
NOTEARS (SCMs with impl. noise scales)
1.0
0.8
0.4
0.2
Properties of the implied SCMs data, the implied SCM corresponds the SCM that could have generated the observations. Therefore, algorithms as- suming that unstandardized SCMs generated the data will be susceptible to any assumption violations of the implied SCM, such as assumptions about the exogenous noise. Figure (bot- tom) shows the distribution of inverse implied noise scales 1/03 for the variables of the implied models (see Equation 2). Since Varle:] = 1 in our experiments, these inverse squared noise scales are equal to the inverse variances of the full ad- ditive noise terms. We find that standardized SCMs induce inverse noise scales that are orders of magnitude greater than those iSCMs. This distribution essentially the footprint of the determinism in the depth limit discussed in Section 4.1. This observation also provides empirical support for our earlier explanation for the improved performance of the PC algorithm on iSCM data. The modes at 1 1/06 and 1/0f > 1 in the iSCM plot correspond to root and non-root nodes, respectively.
When standardizing SCM
1.0
0.8
PDF
0.5
0.2
0.0
100
10
102
10a
1/82
Original SCM
Impl. noise scules of std. SCM Impl. noise scales ISCM
Figure 6: Implied Noise. Bottom panel shows the distribution over in- verse implied noise scales in the im- plied SCMs for ER(100, 2) graphs (kernel density estimate). Lines and shading denote mean and standard deviation. Top panel shows the per- formance of NOTEARS on systems with these noise scale statistics but the same Var-sortability SCMs (see Appendix E.2 and E.5).
Figure 6 (top) shows the performance of NOTEARS when isolating the noise properties the implied models from the fact that standardized SCMs and iSCMs are not Var-sortable. For this, we construct SCMs that have the marginal variances (and Var-sortability, here 0.99 average) of unstandardized

---

Published as a conference paper at ICLR 2025
SCMs but the noise variances of the implied models by correcting their weights (see Appendix E.5). NOTEARS performs better in such systems, suggesting that (i) the noise statistics may indeed explain the performance difference on iSCM data, and (ii) Var-sortability may not be the only reason why NOTEARS performs significantly worse on standardized data (Reisach et al., 2021). Conversely, when the weight ranges of (standardized) SCMs are smaller, the phenomenon of exploding marginal variances is less pronounced (Figure 19 in Appendix F.3). In this case, we indeed find that NOTEARS performs similarly on standardized SCMs and iSCMs (Figure 15, left, in Appendix F.I).
This sheds light on previous benchmarking results, where MSE-based algorithms perform below expectations despite perhaps not intending to evaluate the algorithms under model mismatch (e.g.. Reisach et al., 2021: Kaiser and Sipos, 2021). For the MSE loss, Loh and Bühlmann (2014) and Seng et al. (2024) show that smaller ratios of noise variances increase the magnitude of weights required for the true DAG to be the unique minimizer. The MSE loss ultimately does not account for the inverse variance factor in the Gaussian noise likelihood. Overall, the statistics of the implied models of standardized SCMs are empirically further from SCMs with equal noise variances than their iSCM counterparts.
CONCLUSIONS
We deseribe the iSCM, one-line modification of the SCM that modulates the scale of interaction between the causal mechanism fi and noise e; at each variable Z;. Through several theoretical and experimental results, we study its properties in relation standard SCMs and its ramifications for benchmarking causal discovery algorithms. To conclude, we highlight the following key takeaways:
Standardizing during the generative process removes sortability artifacts. When the functions fi and the noise E are, for example, sampled i.i.d. for each variable z;, SCMs exhibit artifacts that are not removed when shifting and scaling the generated data. Our results in Section 5 show that iSCMs are effective removing Var- and R -o-sottability. This makes iSCMs useful complement to structure learning benchmarks with SCMs, enabling specific evaluation of the ability of algorithms to transfer to real-world settings that do not exhibit R2 artifacts. Despite the removed sortability artifacts, causal discovery algorithms are able to infer nontrivial structure from iSCM data (Figure 5).
Standardizing post-hoc can lead to partial identifiability and degenerate implied SCMs. Scal- ing the units of SCM data not innocuous. Theorem 3 shows that mild knowledge on the distribution of fi can identifyy edges in standardized SCMs that are typically not identifiable from observational data. To our knowledge, our result is the first concerning the identifiability of G from the standardized observational distribution of linear SCMs. This may make benchmarks, where similar assumptions on fi often hold, trivial under standardized SCMs. Moreover, Figure 6 shows that standard SCMs can collapse to modeling near-zero exogenous noise. Theorems and 4 demonstrate that neither property appears in the analogous iSCMs. Ultimately, (non)identifiabiliiy may be either a feature or bug, depending on whether assumptions are verifiable in practice or priori known during evaluation.
iSCMs are stable and scale-free, making them useful models beyond benchmarking. Beyond data generation, the stable generative process of iSCMs might also provide insights for modeling, e.g., large, temporal (Kilian, 2013; Pamfil et al., 2020) or physical systems. In iSCMs, the scale of a causal mechanism fs and its unexplained variation are both unit-less and independent from its position in the causal ordering (Section 3). If we think of each structural assignment as a physical mechanism, energy conservation must be respected, since mechanism can only output as much energy as receives from its inputs (including unexplained noise). Standardization may thus not be completely unrealistic, since it naturally bounds the output scale of every mechanism.
Since each iSCM implies a standard SCM, iSCMs can also be viewed as reparameterization of SCMs that facilitates modeling and learning the functions fi on the same scale, e.g., under shared prior or level of regularization. Conceptually, iSCMs are related to batch normalization (loffe and Szegedy, 2015). a technique used to stabilize the optimization of neural networks, which compose sequences of functions like SCMs, by adding internal standardization. Overall, these properties may make the iSCM a useful structural equation model beyond the benchmarking problem studied here.
10

---

Published as a conference paper at ICLR 2025
REPRODUCIBILITY STATEMENT
To facilitate reproducibility, we provide code, configuration files, and the commands used to obtain all the experimental results in this manuscript supplementary material. They also available at: http:/gihhhb.oowwwrraaaaaa
In Appendix E, we describe the experimental setup, including the computational resources and wall time used to produce the results. Finally, we provide detailed proofs of our theoretical results in Appendix C.
ACKNOWLEDGEMENT
This research was supported by the European Research Council (ERC) under the European Union's Horizon 2020 research and innovation program grant agreement 815943 and the Swiss National Science Foundation under NCCR Automation, grant agreement 5INF40 180545. This work was also supported by the German Federal Ministry of Education and Research (BMBF): Tübingen AI Center, FKZ: 01 01188803B, and by the Machine Learning Cluster Excellence, EXC number 2064/1, project number 390727645.
REFERENCES
Alon, N. and Spencer, J. H. (2016). The probabilistic method. John Wiley & Sons. 6
Andersson, S. A., Madigan, D., and Perlman, M. D. (1997). A characterization of markov equivalence classes for acyclic digraphs. The Annals of Statistics, 25(2):505-541.
Barabási, A.-L. and Albert, R. (1999). Emergence of scaling in random networks. science, 286(5439):509-512. 31
Bühlmann, P., Peters, J., and Ernest, J. (2014). CAM: Causal additive models, high-dimensional order search and penalized regression. 8
Chickering, D. M. (2002). Optimal structure identification with greedy search. Journal of machine learning research, 3(Nov):507-554. 8
Dibaeinia, and Sinha, S. (2020). SERGIO: a single-cell expression simulator guided by gene regulatory networks. Cell systems, 11(3)222--2713
Erdós, P. and Rényi, A. (1959). On random graphs. Publicationes Mathematicae, 6:290-297. 31
Imbens, G. W. and Rubin, D. B. (2015). Causal inference in statistics, social, and biomedical sciences. Cambridge university press.
Ioffe, S. and Szegedy, C. (2015). Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pages 448-456. pmlr. 10
Kaiser, M. and Sipos, M. (2021). Unsuitability of NOTEARS for causal graph discovery. arXiv preprint 1rXiv:2104005444 10
Kalainathan, D., Goudet, O., and Dutta, R. (2020). Causal discovery toolbox: Uncovering causal
relationships in python. Journal of Machine Learning Research, 21(37):1-5. 32
Kilian, L. (2013). Structural vector autoregressions. In Handbook of research methods and applica- lions in empirical macroeconomics, pages 515-554, Edward Elgar Publishing.
Lachapelle, S., Brouillard, P., Deleu, T., and Lacoste-Julien, (2020). Gradient-based neural DAG learning. In International Conference on Learning Representations. 3,
Loh, P.-.. and Bühlmann, P. (2014). High-dimensional learning of linear causal networks via inverse
covariance estimation. The Journal of Machine Learning Research, 15(1):306531155. 8, 10
Lorch, L., Sussex, S., Rothfuss, J., Krause, A., and Schölkopf, B. (2022). Amortized inference for
causal structure learning. Advances in Neural Information Processing Systems, 15:13104--3111
3, 8, 30, 31, 32
11

---

Published as a conference paper at ICLR 2025
Maathuis, M. H., Colombo, D., Kalisch, M., and Bühlmann, P. (2010). Predicting causal effects in large-scale systems from observational data. Nature methods, 7(4):247-24)
Meek, C. (1995). Causal inference and causal explanation with background knowledge. In Pro- ceedings of the Eleventh conference on Uncertainty in artificial intelligence, pages 403-410. 25
Mooij, J. M., Magliacane, S., and Claassen, T. (2020). Joint causal inference from multiple contexts. The Journal of Machine Learning Research, 21(1):3919-403 1, 3, 29, 30
Ng, I., Ghassami, A., and Zhang, K. (2020). On the role of sparsity and DAG constraints for learning linear DAGs, Advances in Neural Information Processing Systems, 33: 33.17943-1799 3, 8,
Pamfil, R., Sriwattanaworrchai, N., Desai, S., Pilgerstorfer. P., Georgatzis, K., Beaumont, P., and Aragam, B. (2020). DYNOTEARS: Structure learning from time-series data. In International Conference on Artificial Intelligence and Statistics, pages 1595-1605. Pmlr. 5,
Pearl, (2009). Causality. Cambridge university press.
Peters, J. and Bühlmann, P. (2014). Identifiability of Gaussian structural equation models with equal error variances. Biometrika, 101(1):219-228. 3,
Peters, J., Janzing, D., and Schölkopf, B. (2017). Elements of causal inference: foundations and learning algorithms. The MIT Press. 2, 3, 17
Rahimi, A. and Recht, B. (2007). Random features for large-scale kernel machines. Advances in neural information processing systems, 20.
Reisach, A., Seiler, C., and Weichwald, S. (2021). Beware of the simulated DAG! Causal discovery benchmarks may be easy to game. Advances in Neural Information Processing Systems, 34:27772- 27784. 1, 2, 3, 5, 8, 9, 10, 30, 32, 33
Reisach, A., Tami, M., Seiler, C., Chambaz, A., and Weichwald, S. (2024). A scale-invariant sorting criterion to find causal order in additive noise models. Advances Neural Information Processing Systems, 36. 1, 2, 3, 7, 8, 29, 30, 32
Runge, J., Bathiany, S., Bollt, E.,, Camps-Vall G., Coumou, D., Deyle, E.,, Glymour, C., Kretschmer, M., Mahecha, M. D., Muñoz-Mari, J., et al. (2019). Inferring causation from time series in earth system sciences. Nature communications, 10(1):2553.
Sachs, K., Perez, O., Pe'er, D., Lauffenburger, D. A., and Nolan, G. (2005). Causal protein- signaling networks derived from multiparameter single-cell data. Science, 308(5721):533-52)
Schaffter, T., Marbach, D.,, and Floreano, D. (2011). GeneNetWeaver: in silico benchmark generation and performance profiling of network inference methods. Bioinformatics, 27(16):2263-222 22222222220.2. 3
Schölkopf. B. (2022). Causality for machine learning, In Probabilistic and causal inference: The works of Judea Pearl, pages 765-804. 5
Seng, J., Zečević, M., Dhami, D. S., and Kersting, K. (2024). Learning large DAGs is harder than you think: Many losses are minimal for the wrong DAG. In The Twelfth International Conference on Learning Representations. 3, 8, 10
Shimizu, S., Hoyer, P. O., Hyvärinen, A., Kerminen, A., and Jordan, M. (2006). A linear Gaussin acyclic model for causal discovery. Journal of Machine Learning Research, 7(10). 3, 39
Shimizu, S., Inazumi, T., Sogawa, Y., Hyvarinen, A., Kawahara, Y., Washio, T., Hoyer, P. O., Bollen, K., and Hoyer, P. (2011). DirectLiGGAN A direct method for learning linear con-Gaussian structural equation model. Journal of Machine Learning Research-JMI.R.
12288. 3,
Spirtes, P. and Glymour, C. (1991). An algorithm for fast recovery of sparse causal graphs. Social science computer review, 911:62--72 8
12

---

Published as a conference paper at ICLR 2025
Squires, C. and Uhler, C. (2022). Causal structure learning: A combinatorial perspective. Foundations of Computational Mathematics, 23()):171-1185.
Squires, C., Yun, A., Nichani, E., Agrawal, R., and Uhler, C. (2022). Causal structure discovery between clusters of nodes induced by latent factors. In Conference on Causal Learning and Reasoning, pages 669-687, PMLR. 3, 29, 30
Verma, T. S. and Pearl, J. (2013). On the equivalence of causal models. 3
Villar, S., Hogg, D. W., Yao, W., Kevrekidis, A. and Schölkopf. B. (2023). Towards fully covariant machine learning. arXiv preprint arX14:2301.3324.
Vowels, M. J., Camgoz, N. C., and Bowden, R. (2022). D'ya like DAGs? A survey on structure learning and causal discovery. ACM Computing Surveys, 55(4):1-36.
Wienöbst, M., Luttermann, M., Bannach, M., and Liskiewicz. M. (2023). Efficient enumeration of markov equivalent dags. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 12313-12320.
Wright, S. (1934). The method of path coefficients. The Annals of Mathematical Statistics, 5(3):611- 215. 5
Yu, Y., Chen, J., Gao, T., and Yu, M. (2019). DAG-GNN: DAG structure learning with graph neural networks. In International Conference on Machine Learning, pages 7154-7163, PMLR. 3, 5, 8
Zheng. X., Aragam, B., Ravikumar, P. K., and Xing, E. P. (2018). DAGs with NO TEARS: Continuous
optimization for structure learning. Advances in neural information processing systems, 31. 3, 30, 32
Zheng. X., Dan, C., Aragam, B., Ravikumar, P., and Xing. E. (2020). Learning sparse nonparametric
DAGs. International Conference on Artificial Intelligence and Statistics, pages 3414-3425 Pmlr. 3,5
13

---

Published as a conference paper at ICLR 2025
CONTENTS
A Implied Models
15
A.1 Implied Model of a Standardized SCM
15
A.2 Implied Model of an iSCM
15
A.3 Weights of the Implied Model Linear iSCM
16
B Interventions in iSCMs
17
Proofs
17
C.I Definitions
17
C.2 Explicit Covariance in Linear SCMs with Unit Marginal Variances
17
C.3 Bound the Fraction of CEV
19
C,4 Identifiability
20
Background Related Work
28
D.I Heuristics for Mitigating Variance Accumulation
28
D.2 Sortability Metries
29
D.3 Structure Learning Algorithms
30
Experimental Setup
31
E.1 Data
31
E.2 Experiment Configurations
E.3
Methods
32
E,4
Hyperparameter Selection
33
E.5
Transferring Noise Variances While Keeping Vortaiityyyl Unchanged
33
E.6
Compute Resources
39
Additional Experimental Results
39
F.1
Structure Learning
39
F2
RR-Sortabiyyty
40
F3
Implied Noise Scales
40
F4
Covariance Matrices for Figure
40
14

---

if is non-root variable, and Published as a conference paper at ICLR 2025
A IMPLIED MODELS
In this section, we describe how to express the assignments of the observed variables of standardized SCMs and iSCMs with general additive noise mechanism
fi 8:) fx(x) + Ex,
(6)
in the form of (SCM), while sharing the same causal graph G and exogenous noise variables e. We obtain the SCM form by moving the standardization steps into the causal mechanisms by linearly rescaling fi and Ei, such that each observed variable is only a function of observed variables and the noise E;. Throughout this work, the implied (SCM) model denotes the specific construction given in the following two subsections. For this, we assume that we can express the first two moments of the system closed form. Similar the main text, we overload notation for both standardized SCMs and iSCMs and write
/ :::=F(aa
and 8; V Var(a.]
We also derive analytic expressions for the weights of the implied models of linear iSCMs defined by Equation (1), which we later use in our proofs.
A.1 IMPLIED MODEL OF A STANDARDIZED SCM
Let x modeled by (Standardized SCM) with causal mechanisms defined by Equation (6). We recall that x are the observations obtained after standardizing Thus, we can rearrange 2. as
I; 8 + f4i
and substitute every unstandardized variable z; by function of its standardized parents x pa(s) as fx(x O spa(f) + Mpa(t))
f((Xpa(()) 14
E
where denotes elementwise multiplication, and Mpati) and spa(i) the vectors of the parent means and standard deviations before standardization. Thus, the assignments of x" in a standardized SCM can be written the SCM given by
x = of (xpa(t)) + OfE,
with implied noise scales 0- 1/8 and implied causal mechanisms
Ji( (xx(((()) Spa(1) ppp(() ji
of (xfpa())
8;
f:
if is root variable.
A.2 IMPLIED MODEL OF AN ISCM
Let x be modeled by (iSCM) with causal mechanisms defined by Equation (6). In an iSCM, x are the observed variables and the latent variables. We can express every observation I; in terms of its observed parents Xpa(1) as
(x((((()) + u4 f((Rpa(t))
M4i
33
8;
8;
8;
8i
Thus, the assignments of x in a iSCM can be written as the SCM given by
Z ğ (xpa(t) + 0,E,,
with implied noise scales 0, = 1/8; and implied causal mechanisms
f. fx(xxa(t) - P4
if i is non-root variable, and
ge (Kpa(t))
81
f3 14i
if é is root variable.
8i
15

---

Published as a conference paper at ICLR 2025
A.3 WEIGHTS OF THE IMPLIED MODEL OF A LINEAR ISCM
Here, we derive the analytical form for the mechanisms of the implied model of linear iSCM with zero-centered, additive noise eq. This iSCM is given by
w xpa(t) + E
and
Zi
Var(z.
where E, satisfies Ef(e;] = 0 and Var(e:] = o..- We can write the above as
W W.XXpate pa(i))
+
Σjepa(t) Wj. + [arl(a
Σ
24
Varr[z. Ei.
Var(aa
Varlzz
jEpa(s)
It follows that the implied SCM of linear iSCM is also linear, with weights and noise variances given by
W3.
3
Var [a)
and
Var(z;)
(7)
In the above, we can write the variance of explicitly as
Var[æ;] = Var
WHIT + t = Var
Σ
Wy,stg
+ to,
jEpa(1)
jEpa(i)
Σ
Covlur Wontryl + o,
(8)
kEpa(f) jEpa(1)
Σ
W, Cov F,1 +
kεPa(t) jEpa(1)
where ① follows from Bienaymé's identity and ② from covariance being bilinear. Substituting the variance into the expressions for the weights and noise variances, we obtain
Wj,i
(9)
v Ekepu(i) Ejepa(s) wky Cov [2
T5 o²
(10)
EkEpu(i) E EjEpa(s) Wk, Cov [Tk F.g to²
Finally, by construction, the variables ă an iSCM have unit marginal variances. Thus, when the parents of 7; pairwise independent, Equation (10) simplifies
Wji
(11)
/Ejepatl
w + o²
This independence condition always holds when the DAG g is forest.
Efficient computation We can efficiently compute the implied model weights using bottom-up
dynamic programming approach. This allows sampling data directly from the exact implied model of an iSCM without resorting empirical standardization statistics. Algorithm 2 describes the procedure. We iteratively compute the weights and noise variances of the implied model following Equations (9) and (10). At each iteration, update the covariance matrix according Lemma 1. The algorithm processes the nodes in topological order, mirroring the proof by induction of Lemma 1.
16

---

Published as a conference paper at ICLR 2025
Algorithm 2 Computing the Implied Model Parameters of Linear iSCMs
Input: DAG G, weight matrix [Wks == Wi,j, noise variances o² E IR
W Oaxd
- Ia
topoloical ordering of g for i = to d do
w f- W.,:
Edge weights ingoing to m
xx + w Σw + o
>> Equation (8)
W.,:: - w/ [=
Equation (9)
x2 n: on Var[ax]]
> Equation (10)
for j = 1 to do Eay,m, + (Eay.) W.,.. 1, Em.,. Dm
return implied weights W. implied noise variances /²
B INTERVENTIONS IN ISCMS
For an iSCM (S, Pe), we can formalize interventions as changes to its causal mechanisms fi, analogous to the common definition for SCMs (Peters al., 2017). Specifically, let μ4 E[ and = / /arrizz] be the mean and standard deviation of the latent variable wt. We define an intervention as replacing one (or several) of the assignments to the latent variables
h; p Ez);
for function h.. Importantly, the statistics f4 and 8; used for the standardization operation
remain unchanged. Thus, we intervene on mechanisms of iSCMs, the variables x may no longer have zero and unit variance, and the perturbations of x, propagate downstream through the causal mechanisms. We note that, under the above definition, intervening on an iSCM through a new mechanism h; is equivalent to intervening the implied SCM of iSCM with the mechanism
(x(x,e) 14(x, e) pa
8;
Appendix A.2 provides details on the implied models of iSCMs.
C PROOFS
C.1 DEFINITIONS
We define the key concepts used throughout our analysis. A path Pjesi between and vj is a set of directed edges that allows reaching v; from by (and vice versa), not taking into account edge directionality, and that joins unique vertices. We call a node collider in path if the node has two ingoing directed edges in the path. We say that path between v; and vy is unblocked and only there is no node Us. that is collider in the path (see Figure 10a). Finally, we use the term undirected connected component to refer to any maximal subgraph of which any two nodes are connected by path containing only undirected edges (Wienöbst et al., 2023).
C.2 EXPLICIT COVARIANCE LINEAR SCMs WITH UNIT MARGINAL VARIANCES
Lemma (Covariance in linear SCMs with unit marginal variances). Let x be modeled by a linear SCM defined by (1) with DAG G that satisfies Var[x;] 1. Then, the covariance Cov [z,, 2z] is the sum of products of the weights along all unblocked paths between the nodes of and 2; in g.
17

---

Published as a conference paper at ICLR 2025
Bj
Us
Figure 7: Lemma I inductive step. If ny is before v; in the topological ordering, then all unblocked paths from V v; must contain parent of o; as the second to last node. To this, suppose an unblocked path from 1; to would instead contain a child of U; as the last node. Then, there either exists collider on the path vy. contradicting that the path is unblocked, or all edges in the path point away from 0ç implying that 2y is descendant of U; and contradicting the topological ordering. Dotted lines represent unblocked paths (which may have common nodes). Solid lines represent edges. by may or may not be parent of which we illustrate with blue arrow.
Specifically, for any i E {I, ..... d3 such that + j, holds that
Cov[a,
Σ II
W1,m
Pa EPo (l.))
where Pless all unblocked paths from zj 24 in G, and (1,m) Pj+ti indicates that the directed edge (1, m) is part of the path Pjevi-
Proof. We will give a proof by induction on the number of vertices | in the DAG Without loss of generality, we assume that the indices of the nodes are ordered according to some fixed topological ordering æ, so T (j) < n (() jf < By the unit marginal variance assumption,
Covl2,, x.} Var[z;] = 1.
(12)
From now on and without loss of generality, we consider two arbitrary indices j < v. The covariance between z; and zz is symmetric.
Base case (d = 2) If Uy is not an ancestor of e; in graph G, they both must be root nodes, because edge 0; + v, the only possible edge when π (j) < (i). Since F, and xy are root nodes, they are independent and Cov(zs, =ry] = 0. Since path of one edge cannot contain a collider, there are no unblocked paths between U; and Vy, so the RHS of Equation (3) is also 0.
Conversely, if vy is ancestor of v; in graph G, 2y is the only parent and ancestor of us. This implies that
Cov[z,, xz,] []] = Cov(u,, + El xy]
=, Cov[zy,
= =Wjji
where the last equality follows from Equation (12). This is exactly Equation (3) for two-node graph.
Induction step (d > 2) Let us assume that Equation (3) holds for all graphs of size d 1, and let have d nodes. We will apply the inductive hypothesis the subgraph of the first 1 nodes in and show that the full DAG G including the d-th vertex still satisfies Equation (3). First, we note that, since the d-th vertex is last in the topological ordering, it has no outgoing edges. Because the node has no outgoing edges, it is not visited on any unblocked paths between vy and for i, d, as va must be a collider in any path. Second, adding the node Uy to a subsystem containing 21, 2i....,22 Id- results in change the joint distribution of 2;, Zz-- Therefore, it has no effect on the covariance between Fi, Zy. Hence, both sides of Equation (3) are unchanged by the presence node va for all j.j < d and the equation still holds for all i d.
We want to show that Equation (3) also holds for = d and any j < i. For this, we first construct all unblocked paths from vy U,. First, note that any unblocked path must go through the parents
18

---

Published as a conference paper at ICLR 2025
k pa(i), because j < i in the topological ordering (see Figure 7). Moreover, for any k pa(i), appending & + to an unblocked path Pj+*k between by and Uk. creates a new unblocked path between 2 and v4. Hence, for i d and any it holds that
Cov (1s, Covl
Σ kEpa(1)
Wk.t + sg,
Σ k (Epa(1)
Whyt (ov(2R,23))
W Cov Coyle,,
Σ kEpa(()
Wket Cov(2g.
E
II
Wiit +
Whit
Wi,m
(
PookEPoo (lm))EjjR
II (lm))Ejjok
Whi +
Wket
Wi,
kEpa(i)
1[k = flw3 + #
Wky
II (lm))Ejeek
Wt,m
EEaa(i)
II
W1,m-
PjosEP (l.m)
For step O. consider two cases. If pa(i), then wy, 0 and the equality trivially holds. If j E pa(i), then it holds by pulling the term for j out of the sum in the previous line. In ④ we apply the inductive hypothesis to express the covariances in terms of sum of products of weights. In 3 we rearrange terms to pull the Wj.s term into the sum over parents. In Q we use the fact that the set of unblocked paths from vy is corresponds all paths from vy any parent of v;, which is us here, with an extra edge k + i appended, and a possible single-edge path directly connecting vj with es (if j pa(i)).
This completes the induction step and the proof.
C.3 BOUND ON THE FRACTION OF CEV
Theorem 2 (Bound on CEV, in linear iSCMs). Let be modeled by a linear iSCM (1) with DAG g and additive noise of equal variances Var[ei] 0², Suppose any node in G has at most parents and w max,,je(1....d W..g Then, for any E [l,..., d}, the fraction of CEV for ãs is bounded as
CEV <
2 02
Proof. We begin by bounding the variance of the latent variables x; in iSCMs. Starting from Equation (8), we can bound the covariances with a product of unit variances as
Var(zy]
Σ
Σ
wk Cov Fx] +
kEpa(t) jEpa(s)
Σ (i jEpa(t)
WkiW + oo2
Σ Wjss
2
jEpa(1)
m 2 +0?
19

---

Published as a conference paper at ICLR 2025
where ① uses Cov Fyg S since Var[ž]] and Var[žg] 1, and 2 applies the Cauchy-
Schwartz inequality. Since we obtain 2; from 2; just by shifting and scaling the latter, we observe that CEV; [ [2V Using the upper bound the variance of æ; and the definition of the
fraction of cause-explained variance in Equation (4)), we get
Varlz, Elz, Xpa(i)]
Varla; w (p((i))
CEV = CEVr[2]] =
1
Var [x]]]
Var[z:]
Var[e:]
o²
σ²
=
1
Var[r:]
Var [x,]
S
u + +
C.4 IDENTIFIABILITY
In this section, we prove Theorems 3 and 4. We begin by deriving the covariances for the 3-node example in Section 4.2 and then give the general proofs for forests. The proofs of both theorems share the same underlying argument. We first derive the SCM forms of the original models, i.c., standardized SCMs in Theorem 3 and iSCMs in Theorem 4. By showing that the standardized SCMs and iSCMs are SCMs with the same causal graphs and observational distributions p(x), we can leverage Lemma 1 to obtain the covariances between the observed variables in both model classes. Ultimately, these covariances allow us to derive (non)identifaabiiity conditions for the DAGS G in an MEC underlying the original models.
Theorems and assume that the exogenous noise is sampled from a zero-centered distribution with equal variance across variables. Since the results are based on the analysis of covariances, they also hold with the assumption that [[[,]] * O, but the zero-mean assumption simplifies notation. To derive the results for iSCMs, we additionally assume that the noise is Gaussian (see Theorem 4). When referring to an undirected edge between nodes ne, Vy for example, in an MEC, we still denote the edge with (v, vy), but the ordering of the nodes is arbitrary.
C.4.1 3-NODE CASE
We begin by studying the 3-node example of Figure 3 in Section 4.2. Let 0s, 3r Yi X E IR be linear function weights, and consider the following three causal graphs 9 belonging to the same MEC, along with their corresponding SCMs and iSCMs.
SCM
iSCM
T1 = E
T1 := E
U1
U2
V3
22 01 E2
(13)
22 31 E2
(14)
x3 = 31 22 + Eg
33 = \ 52 + Eg
31 = 0232 + E1
3 = 72 + 11
11
12
13
= &2
(15)
= E2
(16)
33 = 3212 + E3
33 = 22 + E3
31 = 0z32 + E1
11 := 7332 E1
f1
U2
U3
22 = 850z + E2
(17)
42 == AzI3 + E2
(18)
F3 == E3
F3 : E3
In the following subsections, we derive the covariance matrices of each of the three systems, respec- tively. This leads to the equivalence presented in Equation (5) for standardized SCMs. Moreover, we show that, for iSCMs, all three systems induce exactly the same observational distribution if and only if X X2 X and ?1 = 72 73. These are the --do special cases of Theorems 3 and 4.
20

---

Published as a conference paper at ICLR 2025
STANDARDIZED SCM
To obtain the covariances between the observed variables in the standardized SCMs of Equations (13), (15), and (17), we first show that the assignments to the observed variables in standardized SCMs can be written in the form of linear SCMs over the same causal graph, which allows to use Lemma 1. In all three systems, every vertex has at most one parent. When the node Vj is the only parent of ei under our assumptions on the noise, we have zj / /arr(z so the assignment of rt can be written in the form of an SCM over x as
24
Wy + Var(z;) r3:
WjaV Varlz,] t
Var[z,] Varla,]
27
2
(19)
Var [i
Varl [
Var [2
To Equation (19), first need to compute the marginal variances of the unstandardized observa- tions Tj. For the standardized SCMs, these marginal variances are, respectively:
for Equation (13):
for Equation (15):
for Equation (17):
Var[z1] o²
Var[ri] = (aă + 1)o²
Var[rn] (a3(03 + 1) + 1)o²
Var[2uu = (af + 1)o²
Var[æz] =
Var[au/ (33 + 1)o²
Var[za]= (a) + 1)a²
Var(za] (35 1)0²
Var[za] a²
Given Equation (19) and the marginal variances, we know the weights of all three implied SCMs explicitly. Since all implied SCMs are linear, have unit marginal variances, and share the same causal graph, we can apply Lemma and obtain the covariances of the observational distributions in the original models:
for Equation (13):
for Equation (15):
for Equation (17):
Cov[ui, =5] = Vaf+i
Cov[ui,2u =5]] =
og
Cov[ui,.5) = 03
Važ+1
agiPs +1)+I
Cov [3]
+ 81
Cov(ai,
8g
Cov(ai,
(až+ 1)(0f+1)
Vagis3 1)++
Cov [o2,,, 31
af+1
Cov[rj,
Cov(u2,
+
In the standardized SCM (13), the causal graph is 02 E3. Hence, the edge directions of the DAG G are consistent with the direction of increasing absolute covariance and only
[Cov[zi, < Cov [x x311
a1
B1
aj+1 piaaf
Vat+I
87
ai+1
af+1
+f(o++1)+1
(of +1 < (a} +
(20)
20 + fof L0 + +8300 +
+1)
< <
In the above equivalences, we always multiply or divide by quantities greater than 0, so the direction of the inequality does not change, and transformations equivalent. For the standardized SCM (17) with causal graph v + U2 t- V3, we get an analogous condition for the edges to be aligned with the order of increasing absolute covariance when following the same algebraic manipulations:
[Covlag, x3ح1
|oov [2) xil
+++
ag.
2
21

---

Published as a conference paper at ICLR 2025
We make use of both of these conditions in Section 4. Since 2/(2 + 1) 1 for any 2 > 0, the right- hand sides of both conditions are true if all weights are greater than 1. In this case, the absolute covariance increases downstream in all SCMs of Equations (13) and (17). Hence, among these two systems, only the DAG 9 whose edges aligns with the covariance ordering in the observed p(x*) can induce p(x*), and we can conclude that the other DAG is not the true causal graph.
ISCM
To derive the observational distributions of the iSCMs in Equations (14), (16), and (18), we proceed in the same way as we did for standardized SCMs. We first show that the iSCM is an SCM with a specific set of mechanisms and then apply Lemma to obtain the covariances between the observed variables. To see this, we write the assignment of 2; as
D4
" +
Ej
(21)
Var[a.
Var[ma
Var(a
/ar[ass]
As before, using Equation (21) requires first computing the marginal variances of the latent variables Jy. For the iSCMs defined by Equations (14), (16), and (18), they given by
for Equation (14):
for Equation (16):
for Equation (18);
Var [21] = o²
Var[zi] 72 0
Var(æi] = 73 + o²
Varl Var[za] = 7/ + o²
Var[z2] = o²
Var[æz] = > + o²
Var[zs) >Ι + o²
Var[zs) A 0²
Var[ar) o²
Given Equation (21) and the marginal variances, we obtain an explicit form for the weights of all three implied SCMs. Since the implied SCMs are linear, have unit marginal variances, and share the same causal graph, we can apply Lemma 1 and obtain the covariances of the observational distributions in the original models. It turns out that the observational distribution of all three ground-trui truth systems (zi, T2, Tz3 in Equations (14), (16), and (18) is multivariate Gaussian with the same covariance matrix, with the diagonal elements equal to and the off-diagonal elements given by
7
Cov[ăi,
VTF + 0²
7id
Cov[ž1,
(22)
v (A - + +
A
Cov Cov[T2,55)
+ o
Since the observational distribution of all three SCMs is zero-centered multivariate Gaussian, the distributions are equal and only if their their covariance matrices are identical. The covariances equal if and only if \₁ X2 A and 71 722 Ys. because the function f(2) = 222 + o² appearing in Cov [ن] and Cov [T2, Ty]] of Equation (22) is injective for any o > 0, which means that distinct weights are mapped distinct covariances. Therefore, the three node linear iSCMs in the above MEC share the same observational distribution if and only if they also share the same weights for each edge. regardless of edge orientation.
This implies that the three DAGs G in the MEC of Equations (14), (16), and (18) are not identifiable from p(x): given p(x) induced by an iSCM with DAG in this 3-node MEC, the two other DAGs with the same linear function weights induce the same distribution p(x).
C.4.2 FORESTS
In this section, we generalize the above partial identifiability result for standardized SCMs to arbitrary forest DAGs (Theorem 3). After that, similarly generalize the nonidentifaability of iSCMs to forests (Theorem 4). Our results concern the identification edge directions in an MEC represented by its partially directed graph 9 (V, E), where & contains both directed and undirected edges.
22

---

Published as a conference paper at ICLR 2025
Ui+1
Ui+2
1/3
0++1
1i++
ci+1
0i+2
(a) Subsystem
(b) Subsystem
(c) Subsystem
Figure 8: Proof subcases of Lemma 5, Three possible subgraphs in a chain without a collider.
STANDARDIZED SCM
Before proving the main theorem, we extend the 3-node example to chains of arbitrary length. We show that all but at most one edge in the MEC can be correctly oriented from observational data using the assumption on the support the weights. Analogous to the three node case, we then use this to prove similar result for forest graphs.
Lemma (Orientation edges in undirected chains of standardized SCMs). Let xa be modeled by standardized linear SCM (1) with chain DAG G = (V.E), where Varle,] = for non-root nodes and p for all pa(j). Additionally, suppose 9 contains no colliders. Then, given p(x*) and the partially directed graph G representing the MEC of G, we can identify all but most one edge (vi, v;) of the true DAG G in each undirected connected component of the MEC G. The possible undirected edge has the smallest absolute covariance of all variables connected by edges in the MEC. satisfying Cov[xj, 23 < |Cov [xft x7]1 for all (k, E E (1,5).
Proof. Throughout the proof, we label the nodes E V such that t;-1 and 0+++1 its neighbors for E (2....,d 1}. We start with the analysis of three arbitrary, consecutive vertices in a chain graph. The three possible subgraphs are depicted in Figure 8. We always find E R such that the variance of the latent root of this directed subgraph is p²o². This relaxed assumption on specifically the root node allows for the root of the subgraph to have potential parents outside the subgraph, or to be the root of the whole chain, when later using this lemma to prove the main theorem.
first write the observed We will follow similar derivations in Section C.4.1. Specifically, variables of the standardized SCM in SCM form, and then invoke Lemma to obtain the covariances of the observed variables. To use Equation (19), we again need compute the marginal variances of the variables before standardization. For the subsystems in Figures 8a and 8b, these are, respectively:
for Figure 8a:
for Figure 8b:
Var[m:] = p²a²
Var[a.] (wf+1,1² + 1)o²
Var[z++1] (wf,+112 + 1)o²
Var[se=1] p²0²
Var[e:42] (wiH1, + 1) + l)o²
Var[ (w + 1)o²
By substituting the expressions for marginal variances into Equation (19), we obtain the weights of the implied models of the standardized SCM. Using Lemma 1, we obtain the covariances between the observed variables xf-1+ 2;, xf+11 1- By construction, the marginal variances of the observed variables equal to 1. We treat each subsystem separately:
Subsystem (Figure 8a) Given the marginal variances and Lemma the covariances are
Cov[2),2/$111 V 107.+110 +1
Ws.++Pp
Cov(2441)
Wi+1,+21
2
1
2 +
w Wi+1.442
11 12 + 1) +
?
23

---

Published as a conference paper at ICLR 2025
Following the same algebraic manipulations in Equation (20), substituting d1 W1,++1P and 31 == W441,42 in the derivation, we obtain
w 1 1P2
|Cov (o], sH+1]l
w 141.42
(23)
w 11i+1 p2
The left-hand side of the right-hand inequality in Equation (23) is upper-bounded by 1, similar to the 3-node case. Therefore, if we assume that [13+1,442| 2 must hold that |Cov] [oolaff #4+1l < [Cov [Cov[2f+1 2142] for any choice of p.
Subsystem (Figure 8b) Given the marginal variances and Lemma 1, the covariances are
Covlue
Wi+1.i 1,iP
/w/+ /wfti,
11+1,1422 1,+4.2
Wu /w2+1,1422² +
The ordering of the covariances in this case depends on the specific choice of the weights.
Subsystem 3 (Figure 8c) Following steps analogous to the symmetric subsystem 1, we conclude that, if [1:+1,a| I 2 1, it must hold that Coo (x] 2f > [Cov[r]+ 22+ for any p.
Given the above, we can now study the relationship between the underlying DAG & and the absolute covariance magnitudes under the assumption that |w,,++1| > 1. We will use the fact that, if the chain does not contain collider, then there can be at most one node contained in edges pointing in opposite directions.
First, we treat the case where there exists vertex i, such that [Cov[z;-1: x;]] that is, where some neighboring covariances are equal. If this occurs in 3-node subsystem, only subsystem 2 can describe the true graph. To be consistent with the assumption that there are no colliders in the graph (see Lemma 5), all other edges must be oriented in direction away from u, which completely identifies the graph G in the MEC.
|Cov [oo][= =ff+1]
In the second case, [Cov[="-1-- in the path. Let æ;, x be the unique pair of consecutive variables the chain that minimizes Coo[==f==f111 We show that this pair is the unique minimizer using proof by contradiction. Suppose there exist two pairs zú - and 13.. 2J+11 such that |Cov [xr], x;+1. = |Cov [z)] ; x is the minimum covariance. Without loss of generality, let j + < i. Then, the triple x:__ 25, 1f41 consistent with only subsystems 2 or based on their relative covariances, which implies that must have U +- U; Using the fact that we have no colliders, we can then orient all edges Uk-1 + Up for 1 < < Thus, we can find a subsystem containing Uj 11 Uj+2, which has been already oriented subsystem 3, meaning [Cov [=f.2 23+1 > |Cov ( #f2ll contradiction.
/f [Cov[a3, x3+ill holds for all nodes vy that have two neighbors
Given zj, xf+1 is the unique pair of consecutive variables that minimizes [Cov[æ (C4x(=),+441)) we now show that we can orient all edges except (U;, 1;41). We will do this in two parts. First, we show that one can orient all edges (1j, U)+1) with j and then we show that we can do the same for all edges (Dy, 1)41) with j > i. if 1, consider the subsystem 0;-1, U; Vi+1. Since (Cov[2;_. x;1] > [Cov[z] x/+1l. only subsystems 2 and 3 are possible for this subgraph. We can therefore orient w--1 us Similarly, if i < d - 1, by symmetric argument on vi, 01+1, U142, we can orient W41 + 0142. Since the graph cannot contain colliders, all other edges must be oriented as by <- Ui+1 for j < and U 2/+1 for j > i. In other words, all edges except (t,, U/+1) point away from the two vertices U, U/41, and one of the two variables must be the root the chain. Therefore, [Cov[zj_1 =31 * Cov[a +1ll holds for all vertices by that have two neighbors, then there exists a unique covariance minimizing pair of, 1f and all edges except (V;, 1;+1) are oriented.
The two cases above are exhaustive, and in the worst at most edge (uj, 0/41) is left unoriented in the chain. This edge always corresponds to the minimizer of |Cov [2). 23lll. This completes the proof.
n
24

---

Published as a conference paper at ICLR 2025
Remark From the proof of Lemma 5, it follows that if we are able to orient all the edges in the chain, then the root of the chain is the node joining the two edges with minimum absolute covariance. When we orient all but one edge (oi, 2)+1), the root node of the chain is either U; 2--
We can extend Lemma 5 to forest graphs. For this, we will make use of the first Meek rule (Meek, 1995). The first Meek rule concerns an MEC g. containing the undirected edges (12, ut); (1y, 2%) but not the edge (1,, UR). It states that, if one can orient V/ + Uj. we must have 1/ + Vk-
Theorem 3 (Partial identifiability of standardized linear SCMs with forest DAGs), Let x" be modeled by a standardized linear SCM (1) with forest DAG G, additive noise of equal variances Var[e:] =o² and | > for all i E pa(l). Then, given p(x²) and the partially directed graph 9 representing the MEC of g. we can idemifyy all but at most one edge of the true DAG G in each undirected connected component of the MEC G.
Proof. The undirected parts of an MEC 9 are disjoint undirected connected components. Orienting the edges in all these undirected connected components without introducing a v-structure produces valid DAG G in & (Andersson et al., 1997). Each undirected connected components represents Markov equivalence class of its own (Andersson et al., 1997). Thus, to prove the theorem, we consider these undirected connected components independently with respect to the rest of the graph and show how to orient the edges in each undirected connected component." ! In the following argument, we therefore consider to be a single undirected connected component, with no directed edges by definition, and show that we can orient all but one edge in G. This argument then extends to all undirected connected components of the original MEC G. implying the statement made in Theorem 3.
If 9 is an undirected connected component with no directed edges, we only have to consider SCMs with a ground-truth DAG G that are members of this MEC 9 to distinguish among possible edge orientations in G. In the case of undirected trees, the ground-truth DAG g must be a tree with no colliders and the same skeleton as G, since any other DAGS would belong to a different MEC.
We give proof by strong induction on the number of vertices | in the MEC g. The base case of the induction argument is an MEC with | = 2 nodes. This case holds trivially, since this MEC can contain at most one undirected edge. For the inductive step, we consider an undirected tree MEC G with | = d and assume that we can orient all but one edge of undirected tree MECS with [ < d.
Our argument will proceed by considering the longest chain of the undirected tree S. We will use Lemma 5 to orient all but at most one edge in this chain and then apply the first Meek rule to possibly orient additional edges in G outside the chain. After orienting these edges, we show that we reduced the original problem of orienting all but one edge in 9 with |V] = d to orienting all but one edge in a single undirected connected component that has strictly fewer than d nodes. This allows us apply the inductive hypothesis and complete the proof (see Figure 9).
Consider a longest undirected chain Gc = (Vc, that is a subgraph of the undirected tree G. Let Gc refer to the directed subgraph of the DAG g induced by considering only the vertices Vc. We label the is vertices in Vo as U,, v..... Uh. with undirected edges (v,, 11+1) E & for all i E {1..... k 1}. The nodes UI, Ug can have no undirected neighbours in 9 outside the chain, because otherwise we could construct a longer chain in g.
The only vertex in Ve that can have parent in the DAG G outside the chain Go, that is, in ViVe. is the unique root of Go. To see this, we first note that all nodes v, have at most one parent in G. because any U, with Ipa(r;))l > 1 in G would be a collider, but contains no colliders. Since non-root nodes in Ge have an in-chain parent, they cannot have parent outside of Vc. Therefore, besides the root node of Gc via its potential outside parent, 9c is a completely disconnected subgraph from the rest of G. This implies that we may treat Go as a separate standardized SCM with undirected chain MEC, in which the potential parent of the root of GG is modeled as part of the exogenous noise of the root. This allows us to apply Lemma 5 to the variables of the subgraph Gc-
'Orienting edges of an undirected connected component that touch directed edge in & never introduces an additional v-structure, If directed edge pointed into the undirected connected component, the undirected edge downstream would have had to already be directed in ς by the first Meek rule. Hence, all directed edges bordering the undirected connected component must be oriented away from it, and none of the possible undirected edge orientations creates new collider at the border node. This implies that all undirected connected components in & upstream of the colliders and directed subgraphs of g.
25

---

Published as a conference paper at ICLR 2025
14-1
1++1
V1-2
Uk
u
Figure 9: Inductive step of the proof of Theorem 3. Ground-truth DAG & underlying undirected connected component G in some given MEC. The nodes Vo = {UI,.... Uk are a longest chain in G. Using Lemma 5, we can orient all edges in 9c except possibly (V,, U++1) (blue). Edges like (V--1, u) are oriented by the first Meek rule. After Lemma 5, we are left with either the single undirected tree of U; (left shaded tree) the single undirected tree consisting of (e,, U/+1) (blue) and both undirected trees of o2 and 1 (both shaded trees). Either & or 4 must be the root of Go. In this specific example, U; the root of Go and is therefore the only node that can have parent outside Gc. Any node in G can have directed, outgoing edges to children in (possibly non-forest) MEC the undirected connected component may be subgraph of.
By applying Lemma 5 Gc, resulting analysis into the two cases of Lemma leaving either 0 or undirected edge. In the first case, we can orient all edges in Go with Lemma 5, In this case, we know that the root of Go the node vú (see Remark of Lemma 5). By the first Meek rule, we can recursively orient all additional edges in & outside of فc away from e, except for the subtrees of connected to v; itself (Figure 9). This leaves at most single connected undirected subtree containing v; and strictly less than d vertices.
orient all but most undirected edge in Gg. We split the
In the second case, we orient all but one edge (v, 0441) in Gc by applying Lemma 5. In this case, we know that the root of 9c either the node i/2 or 0,+1 (see Remark of Lemma 5). Similar to the first case, can recursively use the first Meek rule to orient all additional edges in & pointing away from vi and 0,+1, except for the subtrees & connected to v and 1/41 itself. Since e, and U+41 connected by an undirected edge, are left with single connected subtree containing the undirected edge (wz U1+1) that is strictly smaller than before.
In both cases, orient least undirected edge of G, because the longest undirected chain in & with VI > has at least length 2. We always obtain at most a single undirected connected tree component with strictly less than d vertices, allowing apply the inductive hypothesis and complete the proof.
ISCM
Theorem 4 (Nonidentifiability of linear Gaussian iSCMs with forest DAGs), Let & be modeled by a linear iSCM (1) with forest DAG G and additive Gaussian noise of equal variances Var[es]. Then, for every DAG G' in the MEC of G. there exists linear iSCM with DAG G that has the same observational distribution as &. the same noise variances, and the same weights on the corresponding edges in the MEC.
26

---

Published as a conference paper at ICLR 2025
Un
Uk
(a) First subcase
UP
2
17
Uk
Uj
U
Uk
(b) Second subease (More than parent in G')
(c) Second subease (A single parent in g')
Figure 10: Proof subcases of Theorem (a) Path with collider. In other words, a path blocked by an empty set. In the case of forests, this configuration implies that eú and 2 are d-separated. (b) Unblocked path connecting v; and v2 with one of the path nodes having a parent both in the path and outside the path. The weight Wp..k influences the weight W1,s in the implied model of the iSCM. If this structure is present in a forest, it has to be present other graphs in the same MEC. (c) Unblocked path connecting v; and vy with the only parent of being part of the considered path. The weight Wi,k depends only on wi,ks irrespective of the edge direction.
Proof. Because we consider linear iSCMs with Gaussian noise, the implied model is a linear SCM with additive Gaussian noise (see Appendix A.2). Hence, the observational distribution is multi- variate Gaussian with mean zero. In iSCMs, the marginal variance an observed variable is always 1. Hence, prove the statement if we show that for all Tr By in the iSCM with graph G, and the corresponding H4, 2y in the iSCM with graph g' = (V, E'), Cov [T 3y = Cov (ov[7,,77))
Let 7 and T be the random variables associated with the nodes e and uy from g', respectively, We consider two cases. First, if there is path between v; and ty in the skeleton of g' then there is path between U; and vy in the skeleton of G and hence Cov [T, } = Cov [ 7y] = 0. In the second Ty
case, there a path between i and in the skeleton of g', so there also exists a path in the skeleton of G. as both graphs have the same skeleton. Due to the acyclicity of the skeleton forests, this path is the only one connecting v; and vy in both G and g'.
We further break this second case into two subcases. In the first subcase, this path contains collider in G as shown in Figure 10a. Because the skeleton cannot have undirected cycles under the forest assumption, this collider forms v-structure. G' & implies that the same v-structure must be present in G. Hence, and fy are d-separated in both G and g'. By the global Markov condition, this implies that T and F' are independent, and that 7. and ř are independent. This implies that both Cov [ ø]] = Cov Ty T.] 0.
In the second subcase, there exists an unblocked path between o; and v in both 9 and g'. Here, we denote the weight matrix associated with both iSCMs by W == [wa.j], with W being symmetric, so that W = Wj.i is the linear weight the edge (i,j) regardless of its orientation the graph.
We now derive the analogous weights W. W in the implied SCMs for G. respectively. Ultimately. we will demonstrate that the implied SCMs have the same weights. Specifically, we will show that WR. W Given this, Lemma implies that both iSCMs have the same covariance matrix over the observed variables.
Without loss of generality, since the node labelling is arbitrary, let th have at least many incoming edges as 1 G'. We divide the analysis into two cases: vi having only parent in G', and eg having more than parent. The node Us must have at least one parent, since least one of ug, Vi have an incoming edge g', and we chose U2 to have at least as many incoming edges as et.
More than one parent in g' We know that any collider in g' will appear as part of a u-structure in G due to the forest assumption, and therefore will also be collider in G. Therefore, if eg has more than one parent in g' (see Figure 10b), all pairs of edges incoming to 2g will form u-structures, so i must have exactly the same set of parents in G.
27

---

Published as a conference paper at ICLR 2025
1.00
0.71
0.63
0.60
0.39
0.00
33
U6
U2
0.8
2
4
5
2
0.71
1.00
0.89
0.85
0.66
0.00
89
0.63
0.89
1.00
0.95
0.62
0.00
0.66
U2
11
C3
U6
0.60
0.95
1.00
0.50
0.00
0.4
3
5
0.39
0.55
0.62
0.59
1.00
0.77
0.2
U4
V5
0.00
0,00
0.00
0.00
0.77
1.00
Figure 1I: Illustrating Theorem for trees the same MEC. Covariance matrix of observed iSCM variables for two example forests belonging the same MEC with the same weights assigned to the edges of the skeleton.
Moreover, any two parents of Us are d-separated in and G' by the forest assumption, since the blocked path going through vk is the only path connecting them. By the global Markov condition, the parents are pairwise independent. Hence, we can use Equation (11) to compute ūk Wy Since the parent sets are the same between the two graphs, and W is shared between the iSCMs, the weight associated with the edge (1, k) in both graphs in the implied models is given by
Li = W/
Wi.k
(24)
EmEpa(ts w + o2
A single parent in G' Let (1, k) be the only incoming edge un in G', as depicted in Figure 10c. Then, the edge connecting i and 2k in G is either the only incoming edge to 2kk or the only incoming edge to vy. To this, suppose that it make vg or Uj a collider that would be common to both graphs, implying that i or would have at least parents in G'. We operate under the assumption that vą has least many parents ut so would imply that Uy has more than one parent, contradicting the assumption we made for case we consider in this paragraph. Irrespective the direction, the weight associated with the edge (4, k) in the skeleton of both graphs in the implied model is, similar to Equation (21), given by
not the only incoming edge ey or t in G. This would
Wi.k
Wi.k = Wi
(25)
Wi. + o²
Equations (24) and (25) show that, for the SCM form of each iSCM, the edges connecting the same nodes irrespective of their direction in G' and & have the same weights. By Lemma 1, the covariance between any ã; and ž can be expressed as product of the weights in the implied SCM corresponding to the edges on the path between v;; vy. Hence, Cov [T, Fy] Cov(7, (7,777 Ty).
Figure 11 shows example for Theorem 4 for two trees from the same MEC.
Remark In Figure 12, we empirically demonstrate that Theorem 4 no longer holds if we drop the forest assumption. For data generated from an iSCM and two graphs from the same I with the same weights assigned to the skeleton edges, we observe that the estimated covariances differ. The two systems entail different observational distributions.
D BACKGROUND ON RELATED WORK
D.1 HEURISTICS FOR MITIGATING VARIANCE ACCUMULATION
Here, we review existing heuristics for avoiding the exploding variance in structure learning bench- marking with linear SCMs as defined in Equation (1). We also describe how these heuristics limit the causal dependencies that can be modeled in terms of the correlations among the SCM variables or their cause-explained variance, both which do not occur in linear iSCMs. Finally, in Figure 13 we show that the heuristics fail to induce data that is both not Var-sortable and not sorttaall..
28

---

Published as a conference paper at ICLR 2025
U1
1.0
V1
1.0
1.00
0.93
0.93
0.95
0.9
3
1.00
0.ST
0.33
1.00
0.94
-0.8
-0.8
0.93
0.87
1.00
0.95
0.94
1.00
UB
0.7
1/2
V/
2
2
Figure 12: Non-forest counterexample for Theorem 4. Covariance matrix of observed iSCM
variables for two non-forests belonging to the same MEC with the same weights assigned to the edges of the skeleton.
Scaling weights by the inverse weight norm Mooij et al. (2020, Section 5.2) sample the edge weights in linear SCMs Wij U [0.5, 1.5). To achieve comparable variance of each variable zy in the SCM, they propose re-scaling the sampled weights prior to the data-generating process as Wijj
Wii
-iEppa(j wi,
all parents of x, are i.idd Gaussian with variance 1, this adjustment ensures that the variance of ay is similar for all Zz. However, this approximation does not take into account the covariances of the parents. Moreover, since Varle,l is unchanged, the scaling limits the strength of the causal effect that parents can have z.. For example, when I1 E1 and 22 WT1 E2 with Var[e,l 1 as for Mooij et al. (2020), the adjusted weight is w' =w/vll w < 1. Thus, for any w o 0, we have
Cov(ee, w'e + t2]l
Iw'
[Corraa,
R 0.707
V Var[e1] Varlw'e: + E2] V
V2
This is the maximum correlation between neighbouring variables that any SCM can model under the proposed re-scaling when Var[ey] = 1, since additional parents decrease the parent-child correlations. By contrast, iSCMs can model any level of correlation by sampling arbitrary values of wr while guaranteeing unit-variance observations 2y. Intuitively. iSCMs achieve this by standardizing 2y after the exogenous noise E is added the endogenous contributions of the parents Xpa(j), while weight scaling is done before ēj is added to Z.j-
Scaling weights by the incoming variance Squires et (2022, Section 5.1) sample the weights of linear SCMs waj Unif+ [0.25, 1.0]. Given the initial edge weights, they propose adjusting the weights during the generative process by first estimating the total variance of that the parents of Iy contribute z, from samples drawn under initial level of additive noise with Varle,] and then re-scaling the weights as
Wi.
Wi.j f-
/203
When using additive noise with Varle,] 0.5 to generate the actual samples, this scaling results in Var[r,] with a constant fraction of cause-explained variance CEV = 0.5. In benchmarks, however, we may be interested in evaluating SCMs with arbitrary levels of cause-explained variance. iSCMs allow this by construction. Contrary to Squires et al. (2022), iSCMs scale the variables Ty rather than the weights Wi,j while leaving the exogenous noise Ey unchanged, which enables modeling arbitrarily small or large levels of unexplained variation.
D.2 SORTABILITY METRICS
In this section, we describe the definition of a sortability metric as introduced by Reisach et al. (2024),
which we use in Section 5. For function т, 7-sortability assigns scalar in [0, 1] to the variables and graph 9 (with weight matrix Wg) as
Σd Ep. iner( (xx,((xx)x) s), txx,t) Σ Ep.
1 if a < b where incr(a,
if
0
if >b
29

---

Published as a conference paper at ICLR 2025
Mooij et al. (2020)
Squires et al. (2022)
Var-sortability
1.0
R2 sortability
1.0
Var-sortability
-1.0
R2 ---oottbilityy
0.5
0.5
0.5
0.0
0.0
0.0
0.0
100
200
100
200
100
200
100
200
(a) (d, 2)
(b) SF 2)
Figure 13: Sortabilities for data generated according to heuristics that aim to remove artifacts. Weight ranges were assumed as in the original papers: Wa,j ~ + (nifjo.5.1.3 for Mooij et al. (2020), Wi, + Unif (0.25,1.0) for Squires et al. (2022). For every model, we evaluate 100 systems each using n = 1000 samples. Lines and shaded regions denote mean and standard deviation respectively.
and W is the i-th power of the adjacency matrix Wg and Po-at E W if and only if least one directed path from v, to of length i exists in G. If (xx Var [ we obtain Var-sortability from Reisach al. (2021). If
Varla, p2
(x, t) R [x,] =
Var [4
obtain Re-sortability. Estimating [((x))] requires performing regression of x, onto ()M...,dNM
D.3 STRUCTURE LEARNING ALGORITHMS
To complement the interpretation of the results in Section 5, we provide some background on the structure learning methods we evaluate.
NOTEARS (Zheng et al., 2018) NOTEARS uses continuous optimization minimize the regularized mean-squan equared error (MSE) between the the variables modeled by a linear SCM and the observations, while enforcing differentiable acyclicity constraint. The objective function of NOTEARS is given by F(W)= = X - X /2n + >|[W| where ||- ||r and || || are Frobenius and & norm respectively. When the objective is minimized, weights below fixed threshold are zero
AVICI (Lorch et al., 2022) AVICI is an amortized variational inference method that approximates the posterior distribution over causal structures given dataset through a pretrained inference model. The variational approximation of AVICI uses a fully-factored product of Bernoulli distributions for every possible graph edge. The inference model a neural network that predict the variational parameters of the Bernoulli distributions by minimizing the expected forward KL divergence between the true posterior and the approximation. To train the inference model, AVICI can be optimized on any training distribution of (synthetic) dataset-graph pairs. Lorch al. (2022) publish the pretrained parameters of inference models trained on standardized SCMs with linear and nonlinear mechanisms, which we evaluate in this work.
RortnRegress methods (Reisach et al., 2021; 2024) The SORTNREGRESS methods order the vertices by chosen statistic and sparsely regress every node on all of its predecessors in the obtained order. They use Lasso regression with the Bayesian Information Criterion to learn the regression function for a given variable. Var-SORTNNEGGESS uses estimated marginal variances as the sorting criterion. R² --SORRNREGGESSS uses R2 coefficient of determination estimated after performing regression every variable onto all remaining variables. RAND-SORTNREGRES orders the vertices randomly.
30

---

Published as a conference paper at ICLR 2025
E EXPERIMENTAL SETUP
E.1
DATA
Causal mechanisms
We consider systems with additive noise, where
Ji(x, e;) = h₂(x) + E;:
for chosen function h2s The LINEAR systems used in this experiments have causal mechanisms as defined Equation (1), To model nonlinear systems, we use smooth nonlinear functional mechanisms as used by Lorch et al. (2022). Specifically, the function h; that models the relationship between I; and parents is sampled from a Gaussian Process
h4 ~ GP(O,ki),
where k is squared exponential kernel (x ot exp x (-1xx-x'18/22f) x113/217) with output and length scales 6; and I, respectively. We can approximately express the function sample ht analytically using random Fourier features (Rahimi and Recht, 2007) by sampling
14(x) ctV
Σ
cos (
(0 4 + 8(1))
j=1
where a ~ ~N(0,1), ~ NN I), and 5(0) ~ Unif 2T]. In this work, we use M = 100.
Generating a random model Following prior work (Section 2), we sample random systems in any simulation performed in this work by first drawing graph & from the specified random graph distribution. Given the graph G. we sample function parameters of the structural mechanisms over G. For linear systems, we sample way ~ Unif+ [a, bj. where a, b are fixed, i.i.d. for every graph edge. Similarly, for nonlinear systems, for every graph vertex, we draw the length scales 1, ~ Unif(a1, b1] and output scales c; ~ Unif[ag, by] with predefined a1, b1, 02 b2.
Sampling data from a model Given graph 9, noise distribution Pe, and of functions {f), we sample datapoints from an SCM by traversing & in topological ordering. For every vertex Uy we draw a noise sample & ~ Pa: The sample for x; is then deterministically computed by fi from the exogenous and the parents of x;, To sample from a Standardized SCM, we draw dataset from an SCM and standardize it. To sample from an iSCM, we use Algorithm
E.2 EXPERIMENT CONFIGURATIONS
Sortability For Figures 4a, 16a, and 17 we generate Frdōs-Rényi graphs ER(d, R)) (Erdós and Rényi, 1959), with d denoting the graph size and k the expected node degree. For Figures 4b, 16b, and 17b we generate undirected scale-free graphs SF k) (Barabási and Albert, 1999), where d is the graph size and & the number of outgoing edges generated for each vertex. Then, we orient the edges in the graph according to a random topological ordering. We do not sample directed scale-free graphs initially to avoid high sortability by in-degree, which may confound the results.
For all four figures, we generate LINEAR systems with weights sampled from three possible distribu- tions W.y ~ Unif [0.3, 1.8), wi,j Unif & [0.5, 2.0] Wi, ~ Unif [1.3, 3.0) and noise sampled from ~ (O(0 1). For every model configuration, we sample 100 systems and n =1000 data points each. To create Figures 4 and 16 we sampled graphs sizes {20, 60, 100, 140, 180, 220). To obtain Figure 17, we sampled graphs with & E {4. 8, 12, 16, 203.
Structure Learning (Section 5.2) For Figures 5 and 14, we sample LINEAR systems with weights Wi ~ = [0.5, 2.0]. Following Lorch et al. (2022), NONLINEAR mechanisms have length scales I; Unif[ 7.0, 10.0] and output scales G; Unif[10.0, 20.0). Both mechanisms are defined in Appendix E. For Figures 15a and 15b, we generate LINEAR systems with weights Wi, ~ Unif [0.3. 0.8] and wi Unif+ [1.3, 3.0]. For all four figures, sample random ER(20, 2) and ER(100, 2) graphs with noise E; ~ AO 1). For every model configuration, we sample 20 systems and 1000 data points each.
31

---

Published as a conference paper at ICLR 2025
Noise Transfer For Figure 6 (top), sample SCMs, standardized SCMs, and iSCMs with exactly the same underlying graph and weights sampled from Wi,j ~ Unif+ [0.5, 2.0]. The noise variables drawn from E; _0 1). Then, for every triple of SCM, standardized SCM, and iSCM that shares a graph and weights, we create two more SCMs with the same marginal variances as the SCM, but with the noise variances of the implied models of the standardized SCM and iSCM, respectively. Appendix E.5 provides a motivation and detailed explanation of this procedure. Figure 6 (top) shows the performance of NOTEARS on the original SCMs and the two SCMs with transferred noise.
For Figure (bottom), we sample multiple instances of standardized SCMs, and iSCMs with weights drawn from Wi,j +nif [0.5, 2.0] and noise from E; NV 1). For every model instance, we approximate the density of the inverse of their implied noise variances using kernel density estimation. The figure shows the mean and standard deviation of the p.d.f. values over 100 systems. For both figures, we use (R(000, 2) graphs.
E.3 METHODS
NOTEARS (Zheng et al., 2018) To run NOTEARS, we use the original implementation provided by the authors of Zheng et al. (2018) (Apache-2.0 license). Before benchmarking NOTEARS, we run hyperparameter search to calibrate the weight penalty (A) and threshold on held-out instances of each data generation method. The hyperparameters can be found in Appendix E.4.
AVICI (Lorch al., 2022) To evaluate AVICI, we use the code and model checkpoints provided by the authors of the method (MIT license). Specifically, we use the model trained on linear data to benchmark the method on LINEAR systems and the model trained on nonlinear data benchmark on NONLINEAR systems. We score an edge as predicted if the probability prediction by AVICI is greater than 0.5. Since the parameters are pretrained, the method has otherwise no tuneable hyperparameters.
Sortabilities and SORTN methods (Reisach et al., 2021; 2024) To compute the sortabil- ity metrics and run the BORTNNEGRESS baselines, we use the Causa scoo library (BSD-3-Clause license) created by the authors of the method. The algorithms require no tuneable hyperparameters.
GOLEM (Ng et al., 2020) For GOLEM-EV, tune ê (sparsity penalty coefficient), A2 (acyclicity penalty coefficient) and the threshold for zeroing weights. For GOLEM-NV, we tune the same hyperparameters for GOLEM-EV. We do not initialize the model with the solution returned by GOLEM-EV, as done in the original paper, since we want to evaluate a method that does not assume equal noise variances at any point. Not initializing with the GOLEM-EV weights is consistent with the benchmarking approach of Reisach et al. (2021). We use the implementation of the original work (Ng et al., 2020).
PC Algorithm For linear data, we use Gaussian conditional independence test. For nonlinear data, we use the Hilbert-Schmidt Independence Criterion (HSIC) gamma test. We treat the test significance level hyperparameter that we tune. We use the implementation by the Causal Discovery Toolbox (Kalainathan et al., 2020).
GES GES uses the linear Gaussian BIC score function and does not require hyperparameter tuning. We use the implementation by the Causal Discovery Toolbox (Kalainathan al., 2020).
CAM CAM estimates causal ordering using maximum likelihood and then performs sparse nonlinear regression using splines on the possible parents in this ordering. We use the implementation from the dodi scover library (MIT license) and include the preliminary neighbor search option to make the algorithm scale to large graphs. We tune the cutoff value a for variable selection with hypothesis testing over regression coefficients, and the number and order of splines use for the feature function.
LINGAM LINGAM uses independent component analysis, an algorithm for source separation, to find causal ordering, which is identifiable in linear systems if the additive noise in an SCM is non- -au-sausiinn. We use the implementation from the cdt (Causal Discovery Toolbox) library (MIT license).
32

---

Published as a conference paper at ICLR 2025
E.4 HYPERPARAMETEN SELECTION
For all algorithms that require hyperparameter tuning, we perform the search on separate, held-out systems that follow the same configurations as the ones we present in our final experimental results. We run the algorithms 20 times per configuration and choose the median Fl score the criterion for selecting the best hyperparameters.
To run NOTEARS, we need to specify the regularisation strength and a weight threshold rj for thresholding the final weights for graph structure prediction. To select these hyperparameters, we run parameter search with E {0.0. 0.05, 0.1, 0.15, 0.2, 0.25, 0.3) and three possible values of the weight threshold {0.1, 0.2, 0.3). Table presents all final hyperparameter configurations for NOTEARS, For some hyperparameter configurations, in 20 rums experienced numerical issues caused by the acyclicity constraint. However, this never occurs for the selected. optimal hyperparameters, neither when performing the hyperparameter search nor when running the reported experiments.
To run the PC algorithm, one needs choose a test significance level During the hyperparamter search we consider a E {0.01, 0.001, 0.0001). Table 4 presents all final hyperparameter configura- tions for the PC algorithm.
To run GOLEM-EV and GOLEM-NV we need to tune sparsity penalty coefficient Y₁ acyclicity penalty coefficient A2 and the weight threshold r). We consider À E {0.02, 0.002, 0.0002). A (2.0, 5.0, 8.0} and 1) E (..1, 0.3). Tables 2 and 3 present the best configurations.
To CAM we need tune the cutoff value hypothesis testing over regression coefficients and the number and order of splines to use for the feature function for which consider sets (5, 10) and {2, 3} respectively, Table presents the best configurations.
(0.05, 0.10, 0.15] for variable selection with
E.5 TRANSFERRING NOISE VARIANCES WHILE KEEPING Var-SORTABILITY UNCHANGED
Reisach et al. (2021) show that post-hoc standardization of SCM data strongly impairs the performance of NOTEARS. When comparing the performance of NOTEARS between data sampled from iSCMs and standardized SCMs, there are at least two factors that can affect the performance of NOTEARS, low Var-sortability and the violation of the equal noise variance assumption. Our experiments in Figure 6 of Section 5 aim isolating the effect of the latter. Specifically, we investigate whether NOTEARS performs better on Var-sortable datasets that have the noise scale patterns implied when assuming SCMs generated the data- in fact the data was sampled from iSCMs or standardized SCMs. To achieve this, we ensure that the Var-sortability metrics of the data sampled from the models the same, here close 1.
Given two linear SCMs S and 5hh with the same underlying graph G, our goal is to construct system St with the same marginal variances as S" (condition 1) and the same noise variances as Sh (condition 2). For this task to be well-defined, we assume that the noise variances of the root variables in S and 5h are the same. The first step constructing S to copy the noise variances from 8h so that for every i d}.
2 0
o
This satisfies condition 2. Given this, we define of as
Varla 0z WaT xpa(i) + +
Varlw; xt x
where ef has variance of By construction, the condition of sharing the noise variances with Sh and the marginal variances with Sa is fulfilled for the root variables. For all the remaining variables,
33

---

Published as a conference paper at ICLR 2025
Table 1: NOTEARS hyperparameters for experiments. Final settings for the regularization strength À and the weight threshold 7) after hyperparameter tuning the respective models and data-generating processes together with the FI (median) validation scores achieved by NOTEARS.
(a) (R(00, 2) DAGs, LINEAR mechanisms
Weight Distribution
Model
À
Fl (median)
Unif+ 0.33, 0.8)
SCM
0.05
0.20
0.97
Unif+ 00.3, 0.8
Standardized SCM
0.15
0.10
0.59
Unif= (0.3, 0.8)
iSCM
0.15
0.10
0.57
Unif (0.5, 2.0)
SCM
0.00
0.30
0.98
Unif= (0.5,
Standardized SCM
0.15
0.20
0.30
Unif= 2.05
iSCM
0.15
0.10
0.50
Unif, 11.3,3.0
SCM
0.05
0.30
0.98
Unif4 1.0,
Standardized SCM
0.25
0.10
0.24
Unif+ 3.0,,
iSCM
0.20
0.10
0.40
(b) ER(100, 2) DAGs, LINEAR mechanisms
Weight Distribution
Model
À
F1 (median)
Unif 0.33, 0.8)
SCM
0.10
0.10
0.99
Unif 00.3, 0.8
Standardized SCM
0.10
0.10
0.83
Unif [0.3, 0.8)
iSCM
0.10
0.10
0.84
Unif= [0.5, 2.0]
SCM
0.05
0.30
0.94
Unif+ 2.0
Standardized SCM
0.15
0.10
0.47
Unif_ [0.5, 2.0]
iSCM
0.15
0.10
0.76
Unif= 3.0
SCM Standardized SCM
0.10
0.30
0.82
Unif+ 3.0
0.20
0.10
0.30
Unif [10,3.0]
iSCM
0.15
0.10
0.70
(c) ER(20, 2) DAGs, NONLINEAR mechanisms
Model
A
1
F1 (median)
SCM
0.15
0.30
0.58
Standardized SCM
0.15
0.10
0.33
iSCM
0.15
0.20
0.42
(d) ER(100, 2) DAGs, NONLINEAR mechanisms
Model
7)
Fl (median)
SCM
0.30
0.30
0.50
Standardized SCM
0.15
0.10
0.43
iSCM
0.15
0.10
0.61
(e) Noise transfer experiment: ER(100, DAGs, LinEAR mechanisms way Unif [0.5, 2.0)
Model
À
F1 (median)
Original
0.05
0.30
0.96
Noise from standardized SCM
0.10
0.30
0.72
Noise from iSCM
0.05
0.30
0.82
34

---

Published as a conference paper at ICLR 2025
Table GOLEM-EV hyperparameters for all experiments. Final settings for the sparsity penalty coefficient Å1, acyclicity penalty coefficient A2 and the weight threshold 7 after hyperparameter tuning on the respective models and data data-genetne gentraatnngtingggg processes together with the FI (median) validation scores achieved by GOLEM-EV.
(a) (20, 2) DAGs, LINEAR mechanisms
Weight Distribution
Model
A1
X2
Fl (median)
Unif 00.5, 2.0]
SCM
0.002
5.00
0.30
1.00
Unif 2.0
Standardized SCM
0,020
8.00
0.10
0.15
Unif 2.0
iSCM
0.020
2.00
0.10
0.36
Unif 31.3,3.0]
SCM
0.002
5,00
0.30
1.00
Unif 3.0,
Standardized SCM
0,020
8.00
0.10
0.12
Unif 1.3,3.0
iSCM
0.020
5.00
0.10
0.34
Unif 0.8
SCM
0,020
2.00
0.10
1.00
Unif 0.3, 0.8
Standardized SCM
0.020
5.00
0.10
0.33
Unif [0.3, 0.8]
iSCM
0.020
5.00
0.10
0.36
(b) ER(100, 2) DAGs, LINEAR mechanisms
Weight Distribution
Model
Aa
Fl (median)
Unif [0.5, 2.0)
SCM
0.020
2.00
0.20
1.00
Unif 005, 2.0
Standardized SCM
0,020
8.00
0.10
0.13
Unif [0.5, 2.0
iSCM
0.020
5,00
0.10
0.24
Unif 3.03
SCM
0,020
8.00
0.30
0.90
Unif 3.0,
Standardized SCM
0.020
5.00
0.10
0.08
Unif+
3.0
iSCM
0.020
5.00
0.10
0.19
Unif
00.3, 0.8
SCM
0.020
2.00
0.20
1.00
Unif
00.3, 0.8
Standardized SCM
0.020
5.00
0.10
0.30
Unif
[0.3, 0.8
iSCM
0.020
2.00
0.10
0.40
(c) ER((0, 2) DAGs, NONLINEAR mechanisms
Model
X2
1
Fl (median)
SCM
0.020
8.00
0.30
0.39
Standardized SCM
0.002
8.00
0.10
0.20
iSCM
0.020
2,00
0.10
0.25
(d) ER(100,2) 2) DAGs, NONLINEAR mechanisms
Model
A1
Aa
?
F1 (median)
SCM
0.020
8.00
0.10
0.27
Standardized SCM
0.020
8.00
0.10
0.14
iSCM
0.020
5.00
0.10
0.14
35

---

Published as a conference paper at ICLR 2025
Table GOLEM-NV hyperparameters for all experiments. Final settings for the sparsity penalty coefficient Å1, acyclicity penalty coefficient A2 and the weight threshold 7 after hyperparameter tuning on the respective models and data-genn gentraatnngtingg processes together with the FI (median) validation scores achieved by GOLEM-NV.
(a) (20, 2) DAGs, LINEAR mechanisms
Weight Distribution
Model
A2
1]
F1 (median)
Unif+ 00.5, 2.0
SCM
0.0002
2.00
0.20
0.16
Unif+ 00.5, 2.0
Standardized SCM
0.0200
2.00
0.10
0.38
Unif [0.5, 2.0]
iSCM
0.0200
2.00
0.20
0.45
Unif+ [1.3, 3.0
SCM
0,0002
2.00
0.10
0.20
Unif+ [1.3,3.0
Standardized SCM
0.0002
5.00
0.10
0.37
Unif [1.3,
iSCM
0.0200
2.00
0.20
0.37
Unif+ [0.3,
SCM
0.0020
5.00
0.20
0.13
Unif+ [0.3, 0.8
Standardized SCM
0.0200
2.00
0.20
0.55
Unif [0.3, 0.8]
iSCM
0.0200
2.00
0.10
0.58
(b) ER 2) DAGs, LINEAR mechanisms
Weight Distribution
Model
Aa
Fl (median)
Unif+ [0.5, 2.0)
SCM
0.002
5.00
0.20
0.10
Unif 005, 2.0
Standardized SCM
0,020
2.00
0.10
0.32
Unif [0.5, 2.0
iSCM
0.020
2.00
0.10
0.51
Unif 3.03
SCM
0.002
2.00
0.10
0.21
Unif 3.0,
Standardized SCM
0,002
5,00
0.10
0.18
Unif+
3.0
iSCM
0.020
2.00
0.10
0,46
Unif
00.3, 0.8
SCM
0.020
2.00
0.10
0.18
Unif
00.3, 0.8
Standardized SCM
0.020
2.00
0.20
0.65
Unif
[0.3, 0.8
iSCM
0.020
2.00
0.10
0.67
(c) ER((0, 2) DAGs, NONLINEAR mechanisms
Model
X2
1
Fl (median)
SCM
0.002
8.00
0.20
0.07
Standardized SCM
0.020
2.00
0.20
0.30
iSCM
0.020
2,00
0.10
0.41
(d) ER(100,22 2) DAGs, NONLINEAR mechanisms
Model
A1
F1 (median)
SCM
0,002
5.00
0.20
0.07
Standardized SCM
0.020
2.00
0.10
0.24
iSCM
0.020
2.00
0.10
0.36
36

---

Published as a conference paper at ICLR 2025
Table PC hyperparameters for all experiments. Final settings for the significance level after hyperparameter tuning on the respective models and data-generating processes together with the F1 (median) validation scores achieved by PC.
(a) (2,2 2) DAGs, LINEAR mechanisms
Weight Distribution
Model
a
F1 (median)
Unif 00.3, 0.8)
SCM
0.01
0.71
Unif+ 0.33, 0.8
Standardized SCM
0.01
0.70
Unif (0.3, 0.8]
iSCM
0.01
0.72
Unif 2.05
SCM
0,01
0.47
Unif+ 20.5,
Standardized SCM
0.01
0.46
Unif (0.5,
iSCM
0.01
0.58
Unif+ [1.,
SCM
0.01
0.35
Unif+ 1.0,
Standardized SCM
0,01
0.38
Unif+ 1.0,
iSCM
0.01
0.48
(b) ER(100, 2) DAGs, LINEAR mechanisms
Weight Distribution
Model
a
Fl (median)
U 0)3, 0.8)
SCM
0.01
0.82
Unif+ 0.3, 0.8)
Standardized SCM
0.01
0.85
Unif [0.3, 0.8]
iSCM
0.01
0.86
Unif4 [0.5,
SCM
0.01
0.62
Unif+ [0.5, 2.0)
Standardized SCM
0.01
0.57
Unif+ [0.5
iSCM
0.01
0.79
Unif+ 113,3.0]
SCM
0.01
0.42
Unif+ 1.3,3.0
Standardized SCM
0.01
0.43
Unif+ 1.3,3.0
iSCM
0.01
0.71
(c) ER(00, 2) DAGs, NONLINEAR mechanisms
Model
a
(median)
SCM
0.01
0.53
Standardized SCM
0.01
0.54
iSCM
0.01
0.65
(d) ER(100, 2) DAGs, NONLINEAR mechanisms
Model
F1 (median)
SCM
0.01
0.53
Standardized SCM
0.01
0.63
iSCM
0.01
0.68
37

---

Published as a conference paper at ICLR 2025
Table 5: CAM hyperparameters for all experiments. Final settings for the cutoff value for variable selection with hypothesis testing over regression coefficients, the number and order of splines use for the feature function, together with the Fl (median) validation scores achieved by CAM.
(a) ER(20, 2) DAGs, LINEAR mechanisms
Weight Distribution
Model
Number of Splines
Spline Order
F1 (median)
U 00.3, 0.8
SCM
0.05
5
0.49
Unif+ 003, 0.8
Stand. SCM
0.05
0.46
Unif [0.3, 0.8
iSCM
0.05
5
0.57
Unif+ 2.0
SCM
0.10
10
0.31
Unif+ (0.5,2.0
Stand. SCM
0.10
0,23
U (0.5,
iSCM
0.10
5
0.53
Unif [1.3,3.0]
SCM
0.05
10
0.24
Unif+ 3.0,
Stand. SCM
0.05
10
0,27
Unif+ 3.0
iSCM
0.05
0.42
(b) ER(100, 2) DAGs, LINEAR mechanisms
Weight Distribution
Model
a
Number of Splines
Spline Order
Fl (median)
Unif 00.3, 0.8)
SCM
0.05
10
0.54
Unif+ 00.3, 0.8
Stand. SCM
0.05
10
0.57
Unif+ [0.3, 0.8]
iSCM
0.05
5
0.61
Unif [0.5,
SCM
0.05
10
0.39
Unif [0.5,
Stand. SCM
0.05
0.39
Unif 2.0
iSCM
0.05
5
0.62
Unif 3.0
SCM
0.05
5
0.27
Unif 1.3,3.0
Stand. SCM
0.05
10
0.26
Unif+ 1.3,3.0
iSCM
0.05
5
0.61
(c) ER(20 2) DAGs, NONLINEAR mechanisms
Model
Cx
Number of Splines
Spline Order
Fl (median)
SCM
0.05
10
0.50
Standardized SCM
0.05
5
0.52
iSCM
0.05
5
0.57
(d) ER(100, 2) DAGs, NONLINEAR mechanisms
Model
Number of Splines
Spline Order
F1 (median)
SCM
0.05
10
0.50
Standardized SCM
0.05
10
0.51
iSCM
0.05
10
0.57
38

---

Published as a conference paper at ICLR 2025
it holds that
Varlu 02
Var [x]]] Var
woT Da(s) + E/
Varlw? x xpa(t))
Var[z;] - o3"
Varlw, T
X Xpa(o)] + o7
Varlw? pa(())
Var[a?],
which satisfies condition 1. Since the systems St and S have the same marginal variances, they have the same Var-sortability. In the noise transfer experiment of Figure 6, we transfer the noise variances from the implied models of iSCMs and standardized SCMs. To obtain the noise variances in the implied models, we divide the original noise variances (equal 1) by the estimated marginal variances of the corresponding variable before standardization, which we estimate from n = 1000 datapoints. For iSCM, this corresponds to empirical statistics of Equation (7).
E.6 COMPUTE RESOURCES
Our experiments were CPUs with 3GB of memory per CPU, with an exception of the AVICI runs on graphs with 100 vertices, which used 12GB per CPU. The data generation takes less than a few minutes a single CPU, with the exception of the sortability results (Section 5.1). For the sortability results, it takes around 30 minutes to generate the datasets for single graph specification across all weight supports and graph sizes. This is due to a bigger number of configurations and repetitions than in the other experiments. For single graph specification and across all weight supports and graph sizes, it takes around 6 hours to compute the sortability statistics on single CPU. All benchmarked methods take no longer than few minutes per small graph (d 20) and no longer than half hour per big graph (d = 100). The SORTNREGRESS baselines run in less than 1min per graph.
internal cluster. All experiments in this work were computed using
F ADDITIONAL EXPERIMENTAL RESULTS
F.1 STRUCTURE LEARNING
Figure 14 summarizes the structural Hamming distance (SHD) between the predicted and true graphs for the same datasets and algorithms in Figure 5.
In Figures 15a and 15b, we present the F1 scores and SHD attained by the structure learning algorithms data of LINEAR iSCMs, SCMs, and standardized SCMs, across different weight distribution supports and graph sizes. We find that the difference in performance of NOTEARS on data sampled from iSCM and standardized SCMs is larger for larger weight magnitudes and for bigger graphs. For smaller weights, the difference in the mean Fl score of NOTEARS between the two standardization approaches is smaller, which in line with our proposed explanation about the shifts of the implied noise variance distribution in Section 5.2.
In Figure 15a, we also find that when weight magnitudes are below RR--OONRREESESS performs similarly for both standardized SCMs and iSCMs. We also observe this for AVICI. Meanwhile, for larger weights with support extending above these algorithms achieve significantly higher F1 scores on standardized SCMs. This suggests that our condition of [wa,]| > 1 for all edges (vi, 0y)) in statement of Theorem concerning the identifiability of linear standardized SCMs, may have more fundamental practical significance, rather than being merely artifact of the analysis.
In Figure 18, we report results for when the additive noise in the ground-truth SCMs is Con-Gaussian. In this setting, the causal graphs of SCMs are identifiable from observational data (see Section 2). Here, we also benchmark LINGAM (Shimizu et al., 2006), which designed for this setting. While LINGAM performs very well as expected, it performs significantly worse standardized SCMs, possibly because independent component analysis suffers in practice under the very low noise scales implied by post-hoe standardization. This would be in line with discussion in Section 5.2.
39

---

Published as a conference paper at ICLR 2025
SCM
Standardized SCM
ISCM
SHD
LINEAR
SHDD
NONGINEAR
150
150
ER 2)
100
100
50
ER(100, 2)
1000
500-
NOTBARS
GOLEM
US GOLEM
AVICE
Gbs
CAM
Var-8n
NOTHARS
COLEM
GOL FM
Pc
CES
Var-Sa
RP-88
Hand_88
NV
Rand.
UV
NV
Figure 14: SHD to the true causal graph for LINEAR and NONLINEAR mechanisms. Box plots show median and interquartile range (IQR). Whiskers extend to the largest value inside 1.5xIQR from the boxes. Left (right) column shows results for linear (nonlincar) causal mechanisms with additive noise E ~ (O, 1). LINEAR mechanisms have weights Wi,5 Unif, [0.5, 2.0).
F.2 RE-SORTABILITY
Figure 16 reports the R²-sortability statistics across varying graph sizes and weight distributions but for the denser graphs ER(d, 4) and SF(d, 4). We again observe Re-sortability very close to 0.5 for datasets sampled from iSCM and high degrees of R2 -sottabbbyyy for data drawn from standardized SCMs. Moreover, in Figure 17, we show the R² -ortabbbbyl for varying expected node degrees in the graph. Data sampled from iSCMs remains close not R -oossotbble for denser graphs drawn from the graph families considered here. We omit standard SCMs from the plots as the datasets of SCMs and their standardized versions have the same R RRssortability, since the R² coefficient is scale invariant.
F.3 IMPLIED NOISE SCALES
Figure 19 shows the inverse implied noise scales of standardized SCMs and iSCMs for linear models with smaller weights magnitudes than in Figure of the main text. this setting with smaller weights, the distributions of the implied noise scales of standardized SCMs and iSCMs show significantly greater overlap than in Figure 6. Since the weights are smaller, the effect the exploding marginal variances and thus collapsing implied noise scales is weaker in the SCMs.
In Figure 15 (left), we evaluate the algorithms considered Section on these systems with smaller weights. We that, in this setting, NOTEARS performs very similarly on standardized SCMs and iSCMs, with NOTEARS slightly outperforming on iSCMs for bigger graphs, since we do not remove the growing variance problem completely for weights of small magnitude. This is inline with our reasoning in Section 5.2.
F.4 COVARIANCE MATRICES FOR FIGURE
Figure 20 visualizes the full mean absolute covariance (correlation) matrices of the systems presented in Figure 1. The matrix shows that the pattern of increasing mean absolute covariance in standardized neighboring nodes, but it also occurs for vertex pairs further apart,
SCMs is not only a feature though less strongly. This is not the case for iSCMs, where any two pairs of equally spaced vertices have equal covariances in expectation over the weight sampling distribution.
40

---

Published as a conference paper at ICLR 2025
SCM
Standardized SCM
iSCM
[0.3.0.8
3.0.3,3.0
1.00
ER 2)
0.75
0.25
0.00
ER 2)
0.25-
0.00
NOTLARS
GOLEM
AvrcI
Pe
CAM
Var SB
R R2--
Rand -st
NOTEARS
GoLacI
COLEM
AVcCI
P'c
CAM
Var
R R
Rand SR
EV
EN
(a) Fl scores
SHD
+/0.3, 0.8
SHD
+[13,3300
ER 2)
100
100
50-
3000
1000
2000
ER(100, 2)
1000
NOTRARS
GOLDM
GOLEM
Po
GES
CoM
Vax 30
R2-98
Rand St
NOTEARS
GOLFM
GOLEM
AvicI
Pc
Czs
CAM
Var SR
R2-30
Rand 5R
DV
BV
(b) SHD to the true causal graph
Figure 15: Structure learning results for different LINEAR weight ranges. Results for LINEAR causal mechanisms with additive noise E; A 1) and weights sampled uniformly from support indicated above each column. Box plots show median and interquartile range (IQR). Whiskers extend the largest value inside 1.5xIQR from the boxes. For every model, sample 20 systems and = 1000 data points each.
41

---

Published as a conference paper at ICLR 2025
Staudardized SCM
ISCM
+ 0.8]
[ 2.0]
+ [1.33,3.
[0.33, 0.8)
+ |0.5, 2.0]
+ (1.3,3.0)
R2. i--sortabiitty
1..00
R². s.srtaablityy
$1.00
0.75
0.75
0.50
100
200
200
100
200
100
200
100
200
100
200
d
(a) ER(d, 4)
(b) SF 4)
Figure 16: -trtstrabyyltty for different graph sizes. Linear standardized SCMs and iSCMs with Er ~ (O, 1) and weights drawn from uniform distributions with supports given above each plot. For every model, we sample 100 systems and n =1000 data points each. Lines and shaded regions denote mean and standard deviation of -ossortability across runs. Datasets that satisfy R R²-sortability 0.5 (dashed) are not -R-sortable.
Staudardizel SCM
ISCM
#00.3,
+ 2.0]
+(1.3,3.0 3.0]
+ 0.8)
0.5,,
[ 3.0]
R2 P--sortability
1.00
R² P-rortability
1.00
0.75-
0.73
0.50
0.50
4
12
20
12
20
12
20
4
12
20
4
12 k
20
12
20
(a) ER,01,, A))
(b) SF (100, A))
Figure 17:
R -ortaabbiyyy for different (expected) node degrees. Linear standardized SCMs and
iSCMs with & (o0 1) and weights drawn from uniform distributions with supports given above each plot. For every model, we evaluate 100 systems and =1000 samples each. Lines and shaded regions denote mean and standard deviation. Datasets that satisfy R -sosttrabyly 0.5 (dashed) are not Rº -sottabbe..
SCM
Standardized SCM
iSCM
F1
LINEAR
1.0
0.5
0.0
NOTEARS
GOLEM
GOLEM
AVICI
Pc
Ges
CAM
LINGAM
Var-SR
R.2 R2-SR
Rand -RR
BV
NV
Figure 18: Structure learning results for non-Gaussian noise distributions. Causal mechanisms have additive noise E Unif /3, v3), which induces Var[e,]
and LINEAR mechanisms with weights waj ~ Unif [0.5, 2.0]. Graphs are sampled from ER(20, 2). To obtain the results, we use the same hyperparameters as the ones we used obtain the top-left panel Figure 5. Box plots show median and interquartile range (IQR). Whiskers extend to the largest value inside 1.5x from the boxes.
42

---

Published as a conference paper at ICLR 2025
Standardized SCM
ISCM
10P
10a
10
10
1/82
Figure 19; Distribution over inverse implied noise scales in the implied SCMs for ER(100, 2) graphs with smaller weights Wayj ~ Unifjo.3.0.a estimated with kernel density estimation. Lines and shading denote mean and standard deviation respectively.
Standardized SCM
ISCM
1.0
1.00
0.75
0.64
0.59
0.56 0.53 0.52 0.50 0.49
0.49
1.00
0.75
0.56
0.41
0.31
0.23
0.17
0.13
0.09 0.07
0.77 0.75
1.00
0.86
0.78
0.74
0.70 0.68
0.66
0.65 0.64
0.75
1.00
0.75
0.56
0.41
0.31
0.23
0.17
0.13 0.09
0.64
0.86
1.00
0.91
0.85
0.81
0.79
0.77
0.75 0.74
0.56
0.75
1.00
0.75
0.56
0.41
0.31
0.23
0.17 0.13
0.59
0.78
0.91
1.00
0.9K3
0.89
0.86
0.84
0.82 0.81
0..11
0.56
0.75
1.00
0.75
0.55
0.41
0.31
0.23 0.17
-0.56
0.74
0.85
0.9
1.
0.M5
0.92
0.89
0.87 0.86
sc-0.31
0.41
0.56
0.75
1.00
0.71
0.56
0.41
0.31
0.23
0.6
<2
-0.53
0.70
0.81
0.89
0.96
1.00 0.916
0.933
0.91 0.90
0.31
0.41
0.55
0.74
1.00
0.75
0.56
0.41
0.31
1-
-0.52
0.68
0.79
0.86
0.
92
0LAM6
1.00
0.97
0.95 0.93
0-10.17
0.23
0.31
0.11
0.56
0.75
1.00
0.74
0.55
0.41
00.4
-0.50
0.66
0.77
0.84
0.899 0.933
0.97
1.00
0.98
0.94
-0.1.
0.17
0.23
0.31
0.41
0.56
0.74
1.00
0.75
0.56
< -0.19
0.65
0.75
0.82
0.87
0.91 0.95
0.98
1.00
0.98
C:00.09
0.13
0.17
0.23
0.31
0.41
0.55
0.75
1.00
0.75
-02
-00499
0.61
0.74
0.81
0.
0.86
0.0O) 0.93 0.96
0.98
1.00
--0.07 0.09 0.13 0.17 0.23 0.31
0.41
0.56
0.75
1.00
10
Figure 20: Mean absolute covariance matrices for models in Figure 1. Linear standardized SCMs (left) and iSCMs (right) with 10-variable chain DAGs from xy to Iuu and weights W,, ~ Unif [0.5, 2.0] and additive noise from (00, 1). Mean covariances estimated from 100,000 datapoints and averaged over 100,000 models. Since both models have unit marginal variances, covariance equals correlation.
43


Reviews:
[{"review_id": "b43FmN0VIy", "reviewer": "Reviewer_pYS7", "summary": "Sampling datasets from an SCM results in artifacts that can be exploited to infer causal ordering. This paper proposes a different method to standardize SCMs for generating benchmarkable datasets. Earlier work proposes post-hoc standardization that is vulnerable to deterministic relations downstream in long chains (correlation between adjacent nodes gets closer to 1). This work proposes generating new variables with standardized parents that bypasses this problem, called internal standardization. The paper contains results on linear Gaussian SCMs with forest causal graphs that show that standardizing SCMs can make them partially identifiable in some settings whereas internally standardized SCMs are nonidentifiable. Experimental evaluation includes comparing current causal structure learning algorithms performance on unstandardized, standardized,  and internally standardized data.", "questions": "Evaluation on real-world datasets - Although the whole paper is about standardizing synthetic datasets for benchmarking, it would be interesting to see how the structure learning algorithms perform on standardized /internally-standardized versions of real-world datasets.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Causal structure learning methods require synthetic datasets to benchmark against. This work addresses artifacts that vanilla generating methods have. This makes it an interesting and relevant direction of research. The main contribution is the proposed method that is simple and seems to address these artifacts. The experimental evaluation is thorough and the theorems are relevant to the issues with standardized SCMs. The paper is also written well.", "weaknesses": "Few concerns follow: \n1) Most of the theory is for the linear, Gaussian case where while analysis is tractable, most practical situations are non-Gaussian. Therefore, while the partial identifiability result is still of importance, I am not that convinced with the impact of the nonidentifiability result. \n2) Even in the experimental evaluation, the nonlinear systems are evaluated on samples from a Gaussian process. Is it possible to introduce nonlinearity in a different way? \n3) For a proposal that is relatively incremental, I would urge the authors to also focus on the nonlinear case to strengthen the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TyFiO3vc7K", "reviewer": "Reviewer_HchU", "summary": "In this paper, the authors address an important issue in benchmarking causal discovery algorithms: how do we build robust synthetic datasets. The main challenge is that many popular algorithms make use of the patterns (artifacts) among the available synthetic datasets, such as the correlations increase along the causal ordering to learn the causal graph, even when it should be impossible to do so. In this work, the authors describe a simple standardization process of generating synthetic data that makes them less identifiable from prior knowledge, such as correlations, and can be useful for benchmarking and beyond. The standardization process, resulting in causal models called iSCMS, involves normalizing each variable (to zero mean and unit variance) and then continuing this process iteratively over the topological ordering.", "questions": "1. Didnot define the weights formally before using them in Lemma 1.\n2. In theorem 3, \"all except one edge\". Can you elaborate on this claim a little more?\n3. It is also not entirely clear on how to extend iSCMs to handle interventions.\n4. Are iSCMs a valid way of modeling real world datasets? Can you comment on this a little more?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is very well written, explaining the standardization process clearly. It is interesting to note that the iSCMs are also an SCM.\n\n2. The partial identifiability result of Theorem 3, arguing that for standardized SCMs, given the markov equivalence graph, one can recover the almost entire graph, while iSCMs are robust to such identification (Theorem 4) is an important result. This provides a grounding that iSCMs can be used for benchmarking. \n\n3. The empirical validation of the results that the iSCMs are not easily beatable compared to SCMs.", "weaknesses": "1. The authors approach works for sparse graphs, such as directed acylic forests. It is not entirely clear if the process of normalization (i.e., topological ordering) is the reason why it cannot be extended to graphs beyond forests.\n\n2. It seems that the process of generation each sample is computationally expensive, depends linearly on the depth of the topological ordering. Also it is not clear if it extends for nonlinear causal dependencies.\n\n3. Generalizability Beyond linear models is not clear.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uyH3SRMKI4", "reviewer": "Reviewer_5Dtu", "summary": "The authors propose internally-standardized SCMs (iSCMs), which standardize each variable during data generation to counteract the increase in variance or correlation along the causal order. Unlike post-hoc standardization, iSCMs avoid Var- and $R^2$-sortability and demonstrate robustness in large systems, as shown in the simulation study.", "questions": "I understand that retaining ground truth is challenging, but as noted in the weaknesses, there are cases where ground truth can be obtained. Are there additional instances where $R^2$-sortability values are reported for such cases?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 2, "strengths": "1. The paper proposes a simple yet effective approach to avoid  $R^2$ -sortability, operating under the assumption that the increase of  $R^2$  along the causal order is an artifact.\n2. The paper is well-organized, providing both theoretical justifications and experimental results to support its claims.", "weaknesses": "My main concern is the paper’s treatment of increasing correlation along the causal order as an “artifact.” In reality, this pattern could exist in real-world data. Standardizing variables in the data generation process may, in fact, introduce more “artificiality,” as fewer studies have shown this pattern in real data. As reported in [1], the R²-sortability for a real dataset in Section 4.2 was 0.82. Thus, while iSCMs may serve as a useful supplementary benchmark, they may not be the definitive or sole benchmark for causal modeling.\n\n[1] Reisach, Alexander, et al. \"A scale-invariant sorting criterion to find a causal order in additive noise models.\" Advances in Neural Information Processing Systems 36 (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yXkhcFFk1y", "reviewer": "Reviewer_R8Hm", "summary": "The authors observe that while SCMs are widely assumed to be the underlying causal model in causal structure learning algorithms, variance and pairwise correlations among the observed variables in an SCM may provide extra information about the underlying model. This information may not be removed by simply standardizing the observed variables and may be used by existing structure learning algorithms. Therefore, they propose internally-standardized SCMs, where the standardization is done within the recursive data-generating process. The authors provide partial identification of the proposed model and also test the performance of existing algorithms on their proposed model.", "questions": "1. Line 147: \"existing identifiability results only concern the unstandardized distributions $p(x)$ of SCMs.\" I am not sure if I agree with this claim, as most causal discovery methods either do not rely on any parametric assumptions (e.g., PC), or rely on assumptions that do not depend on variance (e.g., LiNGAM).\n\n2. The chain structure in Figure 1 is a special case where, for each $x_i$, $i=1, \\cdots, 10$, all variables $x_j$ with $j < i$ are ancestors of $x_i$. This structure leads to linear growth in variances, which in turn affects correlations. Would this be resolved if the underlying model had a sparser structure or a smaller tree depth?\n\n3. Line 459: \"in standardized SCMs, the parent-child relationships become more deterministic for larger graphs, which makes independence testing less reliable.\" I am not sure if I agree with this claim either, as the PC algorithm is mainly based on *conditional* independence tests, and it recursively removes edges from the fully connected graph only when the variables are conditionally independent. This means that marginal dependencies (deterministic relations) should not have a huge impact on the the algorithm.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "1. The observation about the variance and pairwise correlation of variables in SCMs is very interesting and has not been widely studied. This idea could be used for further evaluation of the performance of structure learning algorithms, especially when no theoretical identifiability results are provided.\n2. The organization of the paper is clear and easy to follow.", "weaknesses": "1. Lack of empirical evidence for the advantage of the proposed model. The authors mention that SCM may suffer from the problem of increasing correlation, whereas iSCM does not. However, there is no clear evidence that this is actually the case in real-life scenarios. If iSCM offers no advantage over SCM in real-life scenarios (i.e., increasing correlation may actually occur in real data), then this may not be a meaningful benchmark, as iSCM is a submodel of SCM and has more model assumptions than SCM.\n\n2. Limited novelty of the proposed theoretical result. The authors claim to provide the first identifiability result for standardized SCMs. However, since standardized SCMs can be considered a special case of SCM, existing identifiability results for SCMs that do not rely on variances (such as LiNGAM) should also apply here (see Q1 below). Additionally, Theorem 3 states that if the underlying model (1) is a linear SCM, (2) has equal noise variance, (3) has a tree structure, and (4) has all edge weights with absolute values greater than 1, then these conditions are sufficient to identify the causal model from standardized data given the MEC. This is a highly restrictive setting, particularly conditions (3) and (4), and there may be other identification methods under the same setting that do not rely on pairwise correlation properties.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WmKjK1iws9", "reviewer": "Reviewer_QXbX", "summary": "This paper addresses the well-known problem of the presence of shortcuts in structural causal models commonly simulated for the benchmarking of causal discovery algorithms. The authors propose iSCM, a variation of standardized structural causal models where standardization occurs at each step of the generation along the topological order, instead of standardizing the dataset post hoc, when all variables are already sampled. They provide some empirical evidence and theory showing that this removes the dependence of the Cause-Explained-Variance (one source of shortcuts in SCMs, standardized or not) from the depth of the causal graph. Additionally, they carry out experiments to compare several algorithms’ performance over SCMs, Standardized SCMs, and iSCM.", "questions": "1. Theorem 2 does show that there is no convergence of CEV towards 1 when we increase the depth, still, it displays a dependence on the density: this is completely overlooked by the authors, both in the theory and the experiments. Is there a specific reason? As described in the weaknesses section, I would like to see experiments accounting for different density levels. In practice, I expect to see that nodes with a lot of parents have a larger CEV, such that when directing edges, it is more likely that that node with a higher CEV is a child rather than a parent: under this conjecture, R2 is still a signal about the correct topological order which could be exploited (although weaker than the one provided by SCMs). SF graphs with preferential attachment might be a good testing ground to see this effect, as they produce an unbalance in the number of edges between different nodes. \n2. I believe that there is a big point where iSCMs would be beneficial and that is a bit overlooked. One important use of synthetic data is for training amortized inference models such as AVICI. It is reasonable to think that these models exploit shortcuts as R2 or var-sortability. Yet, this is not really studied in the paper: all the experiments of Figure 5 are performed on SCMs that are non-R2 sortable in the first place, so we don’t really understand whether AVICI trained on standardized data benefits of R2 sortability or not.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The paper proposes a practical solution to remove varsortability and R2-sortability in samples from simulated structural causal models. The available empirical evidence suggests that their approach succeeds in removing these shortcuts, and the provided theory (Theorem 2) supports these observations (at least in the linear case). In general, the paper is well presented and has the potential of a valuable contribution to the causal discovery community.", "weaknesses": "1. This paper stresses that iSCMs are preferable to SCM for benchmarking, or generally that SCMs are ill-suited for the purpose (L80-82, L49-52, L187-188). I believe this is not supported by the available empirical evidence, both in the paper and in the literature: methods like e.g. LiNGAM, RESIT, CAM, and SCORE (and all other methods based on score matching that are state of the art) have been shown by extensive benchmarking to be neutral about rescaling (see Montagna et al., 2023). The bottom line is that whether SCMs are ill-suited depends on the algorithms at hand: in general, such claims should be supported by more extensive empirical evidence, as I discuss in the next point.\n2. The empirical section is limited. One reason is the aforementioned lack of well-known / SOTA methodologies that might be unaffected by standardization. Moreover, the authors do not consider the effect of increasing edge density. Given that the authors’ aim is to propose new SCMs for benchmarking, and the common practice of testing at different density and sparsity levels, I believe these experiments should be added to the paper. To control density on ER graphs in a way that reasonably scales with the number of nodes, I suggest fixing the probability of edge rather than the average degree (as the latter does not scale with graph size). Finally, the experiments of Fig. 5 are carried out on datasets that are not R2-sortable in the first place. The only clear evidence that iSCM removes R2-sortability is provided in Figure 4, but this is still limited, at least for the lack of testing on different density levels, which, as I argued before, is important. \n3. The claim that the existing identifiability results only concern the *unstandardized* distributions of SCMs (L147) is not true, or at least I don’t see that: let aside the case where the assumption enabling identifiability is that of equal variance of the noise terms for each variable - which are of minor importance, as they concern the very specific case of LiNGAM models with equal variance of noises: LiNGAM, ANM, and PNL identifiability results are agnostic about the variance. So any identifiability result about SCMs generating according to the aforementioned model classes, would hold both for standardized SCMs and for iSCMs.\n\nI am willing to increase my score given improved experiments and removal/explanation of somewhat controversial claims as those in the first and third points of the weaknesses section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)