Standardizing Structural Causal Models

Authors: Weronika Ormaniec, Scott Sussex, Lars Lorch, Bernhard Schölkopf, Andreas Krause
Primary area: causal reasoning
Keywords: Causality, Causal Discovery, Structural Causal Model, Standardized Structural Causal Model, Variance Artifact, Covariance Artifact, Simulation, Benchmark

Abstract:
Synthetic datasets generated by structural causal models (SCMs) are commonly used for benchmarking causal structure learning algorithms. However, the variances and pairwise correlations in SCM data tend to increase along the causal ordering. Several popular algorithms exploit these artifacts, possibly leading to conclusions that do not generalize to real-world settings. Existing metrics like $\operatorname{Var}$-sortability and $\operatorname{R^2}$-sortability quantify these patterns, but they do not provide tools to remedy them. To address this, we propose internally-standardized structural causal models (iSCMs), a modification of SCMs that introduces a standardization operation at each variable during the generative process. By construction, iSCMs are not $\operatorname{Var}$-sortable. We also find empirical evidence that they are mostly not $\operatorname{R^2}$-sortable for commonly-used graph families. Moreover, contrary to the post-hoc standardization of data generated by standard SCMs, we prove that linear iSCMs are less identifiable from prior knowledge on the weights and do not collapse to deterministic relationships in large systems, which may make iSCMs a useful model in causal inference beyond the benchmarking problem studied here. Our code is publicly available at: https://github.com/werkaaa/iscm.

Decision: accept (poster)
Reviewer scores: 8, 8, 6, 5, 8

Reviews and discussion:
<thread>
creation date:
1729855092756

thread_type:
official_review

<thread_object>
creator: reviewer_qxbx

<content>
summary:
This paper addresses the well-known problem of the presence of shortcuts in structural causal models commonly simulated for the benchmarking of causal discovery algorithms. The authors propose iSCM, a variation of standardized structural causal models where standardization occurs at each step of the generation along the topological order, instead of standardizing the dataset post hoc, when all variables are already sampled. They provide some empirical evidence and theory showing that this removes the dependence of the Cause-Explained-Variance (one source of shortcuts in SCMs, standardized or not) from the depth of the causal graph. Additionally, they carry out experiments to compare several algorithms’ performance over SCMs, Standardized SCMs, and iSCM.

strengths:
The paper proposes a practical solution to remove varsortability and R2-sortability in samples from simulated structural causal models. The available empirical evidence suggests that their approach succeeds in removing these shortcuts, and the provided theory (Theorem 2) supports these observations (at least in the linear case). In general, the paper is well presented and has the potential of a valuable contribution to the causal discovery community.

weaknesses:
1. This paper stresses that iSCMs are preferable to SCM for benchmarking, or generally that SCMs are ill-suited for the purpose (L80-82, L49-52, L187-188). I believe this is not supported by the available empirical evidence, both in the paper and in the literature: methods like e.g. LiNGAM, RESIT, CAM, and SCORE (and all other methods based on score matching that are state of the art) have been shown by extensive benchmarking to be neutral about rescaling (see Montagna et al., 2023). The bottom line is that whether SCMs are ill-suited depends on the algorithms at hand: in general, such claims should be supported by more extensive empirical evidence, as I discuss in the next point.
2. The empirical section is limited. One reason is the aforementioned lack of well-known / SOTA methodologies that might be unaffected by standardization. Moreover, the authors do not consider the effect of increasing edge density. Given that the authors’ aim is to propose new SCMs for benchmarking, and the common practice of testing at different density and sparsity levels, I believe these experiments should be added to the paper. To control density on ER graphs in a way that reasonably scales with the number of nodes, I suggest fixing the probability of edge rather than the average degree (as the latter does not scale with graph size). Finally, the experiments of Fig. 5 are carried out on datasets that are not R2-sortable in the first place. The only clear evidence that iSCM removes R2-sortability is provided in Figure 4, but this is still limited, at least for the lack of testing on different density levels, which, as I argued before, is important. 
3. The claim that the existing identifiability results only concern the *unstandardized* distributions of SCMs (L147) is not true, or at least I don’t see that: let aside the case where the assumption enabling identifiability is that of equal variance of the noise terms for each variable - which are of minor importance, as they concern the very specific case of LiNGAM models with equal variance of noises: LiNGAM, ANM, and PNL identifiability results are agnostic about the variance. So any identifiability result about SCMs generating according to the aforementioned model classes, would hold both for standardized SCMs and for iSCMs.

I am willing to increase my score given improved experiments and removal/explanation of somewhat controversial claims as those in the first and third points of the weaknesses section.

questions:
1. Theorem 2 does show that there is no convergence of CEV towards 1 when we increase the depth, still, it displays a dependence on the density: this is completely overlooked by the authors, both in the theory and the experiments. Is there a specific reason? As described in the weaknesses section, I would like to see experiments accounting for different density levels. In practice, I expect to see that nodes with a lot of parents have a larger CEV, such that when directing edges, it is more likely that that node with a higher CEV is a child rather than a parent: under this conjecture, R2 is still a signal about the correct topological order which could be exploited (although weaker than the one provided by SCMs). SF graphs with preferential attachment might be a good testing ground to see this effect, as they produce an unbalance in the number of edges between different nodes. 
2. I believe that there is a big point where iSCMs would be beneficial and that is a bit overlooked. One important use of synthetic data is for training amortized inference models such as AVICI. It is reasonable to think that these models exploit shortcuts as R2 or var-sortability. Yet, this is not really studied in the paper: all the experiments of Figure 5 are performed on SCMs that are non-R2 sortable in the first place, so we don’t really understand whether AVICI trained on standardized data benefits of R2 sortability or not.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730541186918

thread_type:
official_review

<thread_object>
creator: reviewer_r8hm

<content>
summary:
The authors observe that while SCMs are widely assumed to be the underlying causal model in causal structure learning algorithms, variance and pairwise correlations among the observed variables in an SCM may provide extra information about the underlying model. This information may not be removed by simply standardizing the observed variables and may be used by existing structure learning algorithms. Therefore, they propose internally-standardized SCMs, where the standardization is done within the recursive data-generating process. The authors provide partial identification of the proposed model and also test the performance of existing algorithms on their proposed model.

strengths:
1. The observation about the variance and pairwise correlation of variables in SCMs is very interesting and has not been widely studied. This idea could be used for further evaluation of the performance of structure learning algorithms, especially when no theoretical identifiability results are provided.
2. The organization of the paper is clear and easy to follow.

weaknesses:
1. Lack of empirical evidence for the advantage of the proposed model. The authors mention that SCM may suffer from the problem of increasing correlation, whereas iSCM does not. However, there is no clear evidence that this is actually the case in real-life scenarios. If iSCM offers no advantage over SCM in real-life scenarios (i.e., increasing correlation may actually occur in real data), then this may not be a meaningful benchmark, as iSCM is a submodel of SCM and has more model assumptions than SCM.

2. Limited novelty of the proposed theoretical result. The authors claim to provide the first identifiability result for standardized SCMs. However, since standardized SCMs can be considered a special case of SCM, existing identifiability results for SCMs that do not rely on variances (such as LiNGAM) should also apply here (see Q1 below). Additionally, Theorem 3 states that if the underlying model (1) is a linear SCM, (2) has equal noise variance, (3) has a tree structure, and (4) has all edge weights with absolute values greater than 1, then these conditions are sufficient to identify the causal model from standardized data given the MEC. This is a highly restrictive setting, particularly conditions (3) and (4), and there may be other identification methods under the same setting that do not rely on pairwise correlation properties.

questions:
1. Line 147: "existing identifiability results only concern the unstandardized distributions $p(x)$ of SCMs." I am not sure if I agree with this claim, as most causal discovery methods either do not rely on any parametric assumptions (e.g., PC), or rely on assumptions that do not depend on variance (e.g., LiNGAM).

2. The chain structure in Figure 1 is a special case where, for each $x_i$, $i=1, \cdots, 10$, all variables $x_j$ with $j < i$ are ancestors of $x_i$. This structure leads to linear growth in variances, which in turn affects correlations. Would this be resolved if the underlying model had a sparser structure or a smaller tree depth?

3. Line 459: "in standardized SCMs, the parent-child relationships become more deterministic for larger graphs, which makes independence testing less reliable." I am not sure if I agree with this claim either, as the PC algorithm is mainly based on *conditional* independence tests, and it recursively removes edges from the fully connected graph only when the variables are conditionally independent. This means that marginal dependencies (deterministic relations) should not have a huge impact on the the algorithm.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730721802136

thread_type:
official_review

<thread_object>
creator: reviewer_5dtu

<content>
summary:
The authors propose internally-standardized SCMs (iSCMs), which standardize each variable during data generation to counteract the increase in variance or correlation along the causal order. Unlike post-hoc standardization, iSCMs avoid Var- and $R^2$-sortability and demonstrate robustness in large systems, as shown in the simulation study.

strengths:
1. The paper proposes a simple yet effective approach to avoid  $R^2$ -sortability, operating under the assumption that the increase of  $R^2$  along the causal order is an artifact.
2. The paper is well-organized, providing both theoretical justifications and experimental results to support its claims.

weaknesses:
My main concern is the paper’s treatment of increasing correlation along the causal order as an “artifact.” In reality, this pattern could exist in real-world data. Standardizing variables in the data generation process may, in fact, introduce more “artificiality,” as fewer studies have shown this pattern in real data. As reported in [1], the R²-sortability for a real dataset in Section 4.2 was 0.82. Thus, while iSCMs may serve as a useful supplementary benchmark, they may not be the definitive or sole benchmark for causal modeling.

[1] Reisach, Alexander, et al. "A scale-invariant sorting criterion to find a causal order in additive noise models." Advances in Neural Information Processing Systems 36 (2024).

questions:
I understand that retaining ground truth is challenging, but as noted in the weaknesses, there are cases where ground truth can be obtained. Are there additional instances where $R^2$-sortability values are reported for such cases?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730747686797

thread_type:
official_review

<thread_object>
creator: reviewer_hchu

<content>
summary:
In this paper, the authors address an important issue in benchmarking causal discovery algorithms: how do we build robust synthetic datasets. The main challenge is that many popular algorithms make use of the patterns (artifacts) among the available synthetic datasets, such as the correlations increase along the causal ordering to learn the causal graph, even when it should be impossible to do so. In this work, the authors describe a simple standardization process of generating synthetic data that makes them less identifiable from prior knowledge, such as correlations, and can be useful for benchmarking and beyond. The standardization process, resulting in causal models called iSCMS, involves normalizing each variable (to zero mean and unit variance) and then continuing this process iteratively over the topological ordering.

strengths:
1. The paper is very well written, explaining the standardization process clearly. It is interesting to note that the iSCMs are also an SCM.

2. The partial identifiability result of Theorem 3, arguing that for standardized SCMs, given the markov equivalence graph, one can recover the almost entire graph, while iSCMs are robust to such identification (Theorem 4) is an important result. This provides a grounding that iSCMs can be used for benchmarking. 

3. The empirical validation of the results that the iSCMs are not easily beatable compared to SCMs.

weaknesses:
1. The authors approach works for sparse graphs, such as directed acylic forests. It is not entirely clear if the process of normalization (i.e., topological ordering) is the reason why it cannot be extended to graphs beyond forests.

2. It seems that the process of generation each sample is computationally expensive, depends linearly on the depth of the topological ordering. Also it is not clear if it extends for nonlinear causal dependencies.

3. Generalizability Beyond linear models is not clear.

questions:
1. Didnot define the weights formally before using them in Lemma 1.
2. In theorem 3, "all except one edge". Can you elaborate on this claim a little more?
3. It is also not entirely clear on how to extend iSCMs to handle interventions.
4. Are iSCMs a valid way of modeling real world datasets? Can you comment on this a little more?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730996775615

thread_type:
official_review

<thread_object>
creator: reviewer_pys7

<content>
summary:
Sampling datasets from an SCM results in artifacts that can be exploited to infer causal ordering. This paper proposes a different method to standardize SCMs for generating benchmarkable datasets. Earlier work proposes post-hoc standardization that is vulnerable to deterministic relations downstream in long chains (correlation between adjacent nodes gets closer to 1). This work proposes generating new variables with standardized parents that bypasses this problem, called internal standardization. The paper contains results on linear Gaussian SCMs with forest causal graphs that show that standardizing SCMs can make them partially identifiable in some settings whereas internally standardized SCMs are nonidentifiable. Experimental evaluation includes comparing current causal structure learning algorithms performance on unstandardized, standardized,  and internally standardized data.

strengths:
Causal structure learning methods require synthetic datasets to benchmark against. This work addresses artifacts that vanilla generating methods have. This makes it an interesting and relevant direction of research. The main contribution is the proposed method that is simple and seems to address these artifacts. The experimental evaluation is thorough and the theorems are relevant to the issues with standardized SCMs. The paper is also written well.

weaknesses:
Few concerns follow: 
1) Most of the theory is for the linear, Gaussian case where while analysis is tractable, most practical situations are non-Gaussian. Therefore, while the partial identifiability result is still of importance, I am not that convinced with the impact of the nonidentifiability result. 
2) Even in the experimental evaluation, the nonlinear systems are evaluated on samples from a Gaussian process. Is it possible to introduce nonlinearity in a different way? 
3) For a proposal that is relatively incremental, I would urge the authors to also focus on the nonlinear case to strengthen the paper.

questions:
Evaluation on real-world datasets - Although the whole paper is about standardizing synthetic datasets for benchmarking, it would be interesting to see how the structure learning algorithms perform on standardized /internally-standardized versions of real-world datasets.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732024592255

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General response to all Reviewers

comment:
We would like to thank all of the five reviewers for their careful reading and overall positive feedback on the paper. We are pleased that the reviewers find our research “interesting” and “relevant” [R8Hm,pYS7,HchU] and have the “potential of a valuable contribution” [QXbX] while being “simple yet effective” [5Dtu]. The reviewers specifically emphasized the importance of our theoretical results [HchU, pYS7] and that they justify our approach [5Dtu], despite missing some references to prior results, which we addressed in our uploaded revision. Our experimental evaluation is rated as thorough [pYS7,5Dtu] and supportive of our proposed approach [QXbX,5Dtu]. Finally, it is fantastic to read that all 5 reviewers individually find our manuscript to be well-presented [5Dtu,QXbX], well-written [HchU, pYS7], or easy to follow [R8Hm].

We reply to each of the reviewers’ questions and concerns individually below. To track the significant changes to the manuscript, *we colored any additional text content added to the PDF in red* if they go beyond minor edits.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes internally standardized structural causal models (iSCMs) to address artifacts such as variance and covariance patterns often exploited in causal structure learning benchmarks. By introducing standardization at each generative step, iSCMs mitigate these artifacts, providing a robust framework for evaluating causal discovery algorithms. The paper presents rigorous theoretical contributions, demonstrating that iSCMs avoid R2-sortability and are robust against deterministic relationships in large graphs. The empirical evaluation is thorough, showing the utility of iSCMs for causal discovery benchmarking. While reviewers raised concerns about scalability to real-world datasets, the focus on synthetic benchmarking and the comprehensive discussion of limitations appropriately position iSCMs as a complementary tool rather than a replacement for existing models. The authors’ responses to feedback were constructive, addressing specific questions about identifiability, computational complexity, and broader applicability. Given the theoretical and practical significance of the contributions, as well as the authors’ successful engagement during the rebuttal, this paper merits acceptance.