Demystifying amortized causal discovery with transformers
Authors: Francesco Montagna
Max Cairney-Leeming
Dhanya Sridhar
Francesco Locatello
Venue: Accepted by TMLR ()

Abstract: Supervised learning for causal discovery from observational data often achieves competitive performance despite seemingly avoiding the explicit assumptions that traditional methods require for identifiability. In this work, we analyze CSIvA (Ke et al., 2023) on bivariate causal models, a transformer architecture for amortized inference promising to train on synthetic data and transfer to real ones. First, we bridge the gap with identifiability theory, showing that the training distribution implicitly defines a prior on the causal model of the test observations: consistent with classical approaches, good performance is achieved when we have a good prior on the test data, and the underlying model is identifiable. Second, we find that CSIvA can not generalize to classes of causal models unseen during training: to overcome this limitation, we theoretically and empirically analyze \textit{when} training CSIvA on datasets generated by multiple identifiable causal models with different structural assumptions improves its generalization at test time. Overall, we find that amortized causal discovery still adheres to identifiability theory, violating the previous hypothesis from Lopez-Paz et al. (2015) that supervised learning methods could overcome its restrictions.

Published in Transactions on Machine Learning Research (12/2025)
Demystifying amortized causal discovery with transformers
Francesco Montagna
Institute of Science and Technology Austria Chan Zuckerberg Initiative francesco montagna@lst.ac.at
Max Cairney-Leeming Institute of Science and Technology Austria
Dhanya Sridhar
Mila Quebec AI Institute and Université de Montréal
Francesco Locatello
Institute of Science and Technology Austria
Reviewed on OpenReview: https://pparrrrieeaae////////sss
Abstract
Supervised learning for causal discovery from observational data often achieves competitive performance despite seemingly avoiding the explicit assumptions that traditional methods require for identifiability. In this work, analyze CSIvA (Ke et al.] 2023bb on bivariate causal models, transformer architecture for amortized inference promising to train synthetic data and transfer to real ones. First, we bridge the gap with identifiabilit, theory, showing that the training distribution implicitly defines a prior on the causal model of the test obscrvations: consistent with classical approaches, good performance is achieved when we have a good prior on the test data, and the underlying model identifiable. Second, we find that CSTvA can not generalize to classes of causal models unseen during training: to overcome this limitation, we theoretically and empirically analyze when training CSIvA on datasets generated by multiple identifiable causal models with different structural assumptions improves its generalization at test time. Overall, we find that amortized causal discovery with transformers still adheres to identifiability theory, violating the previous hypothesis from Lopez-Paz et al. 2015 (hat supervised learning methods could overcome its restrictions.
1 Introduction
Causal discovery aims to uncover the underlying causal relationships between variables of a system from pure observations, which is crucial for answering interventional and counterfactual queries when experimentation is impractical or unfeasible (Peters et al. 2017 Pearl 2009: Spirtes 2010). Unfortunately, causal discovery
is inherently ill-posed (Glymour al. 2019): unique identification of causal dircctions requires restrictive assumptions on the class of structural causal models (SCMs) that generated the data (Shimizu et. al. 2006)
Hoyer et al. 2008 Zhang & Hyvärinen 2009 These theoretical limitations often render existing methods inapplicable, as the underlying assumptions are usually untestable or difficult to verify in practice (Montagna et al. 2023a
Recently, supervised learning algorithms trained on synthetic data have been proposed to overcome the need for spocific hypotheses, which restrains the application of classical causal discovery methods to real-world problems
(e al. 2023b Lopez-Paz al. 2015 Li et al. 2020 Lippe et al. 2022 Lorch et al.) (002). Seminal work from
Lopez-Paxe al. (215)) argues that this Learning-based approach to causal discovery would allow dealing with

---

Published in Transactions on Machine Learning Research (12/2025)
complex data-generating processes and would greatly reduce the need for explicitly crafting identifiability con- ditions a-priori: despite this ambitious goal, the output of these methods is generally considered unreliable, as no theoretical guarantee is provided. A pair of non-identifiable structural causal models can be associated with different causal directed acyclic graphs (DAGs) g 9, while entailing the same joint distribution p on the sys- tem's variables. would be able to overcome these thcoretical limits and correctly identify a unique cansal structure. However, the available empirical evidence seems not to reflect impossibility results, as these methods yield surprising general- ization abilities on several synthetic benchmarks. Our work aims to bridge this gap by studying the performance of a transformer architecture for causal discovery through the lens of the theory of identifiability from obser- vational data. Specifically, we analyze the CSIvA (Causal Structure Induction via Attention) model for causal discovery (Ke el. al. 2023b), locusing on bivariate graphs, as they offer a controlled yet non-trivial setting for the investigation. As our starting point, we provide closed-form examples that identify the limitations of CSIvA in recovering causal structures of lincar non-Gaussian and nonlinear additive noise models, which are notably iden- tifiable, and demonstrate the expected failures through empirical evidence. These findings suggest that the class of structural causal models that can be identified by CSIvA is inherently dependent the specific class of SCMs observed during training. Thus, the need for restrictive hypotheses on the data-generating process is intrinsic to causal discovery, both in the traditional and modern learning- based approaches: assumptions on the test distri- bution either are posited when selecting the algorithm (traditional methods) or in the choice of the training data (learning- based methods). To address this limitation, we theoretically and empirically analyze when training CSIvA on datasets generated by multiple identifiable SCMs with different structural assumptions improves its generalization at test time. Our experimental findings are based on the analysis of 1M test runs. In summary:
thus unclear how a learning algorithu presented with observational data generated from p
We empirically show that the class of structural causal models that CSIvA can identify is defined by the class of SCMs observed through samples during the training. We reinforce the notion that identifiability in causal discovery inherently requires assumptions, which must be encoded in the training data in the case of learning algorithms for amortized inference: this violates a previous hypothesis in Lopez- Paz et al. (2015), which suggests that these methods could exceed the boundaries of identifiability.
We empirically show that CSIvA is expected to fail to generalize on datasets generated by structural causal models characterized by mechanism types noise terms distributions unseen during training. While this appears as a significant limit of amortized causal discovery, systematic analysis has been disregarded by previous work in the literature.
To mitigate this limitation, we study the benefits of CSIvA training on mixtures of causal models. We analyze when algorithms learned on multiple models are expected to identify broad classes of SCMs (unlike many classical methods). Empirically, we show that training on samples generated by multiple identifiable causal models with different assumptions on mechanisms and noise distribution results in significantly improved generalization abilities.
2 Related works
In this paper, we study amortized inference of bivariate causal graphs, i.e. supervised optimization of an inference model to directly predict a causal structure from newly provided data. In particular, this is the first work that draws a connection between identifiabliity theory and amortized inference of causal DAGs, Dai ct al. (222)) studies supervised learning of the graph skeleton, limiting its analysis to the role of identifiability of unshielded triplets. Several algorithms have instead been proposed.
Algorithms for amortized inference closely related to CSIvA. In the context of purely observational data, Lopez-Paz et al. (2015) defines a classification problem mapping the kernel mean embedding of the data distribution to a causal graph, while Li et al. 2020) relies on equivariant neural network architectures. More recently, Lippe et al. (2022) and Lorch et al. (2022) proposed learning on interventional data, in addition to observations (in the same spirit as CSIvA). Despite different algorithmic implementations, the target object of estimation of most of these methods is the distribution over the space of all possible graphs, conditional on the
2

---

Published in Transactions on Machine Learning Research (12/2025)
input dataset (similarly, the ENCO algorithm in Lippe et al. (2022) models the conditional distribution of indi- vidual edges). This justifies our choice of restricting our study to the CSTvA architecture (despite this being a clear limitation), as in the infinite observational sample limit, these methods approximate the same distribution.
Other learning-based algorithms for causal discovery. Out of the scope of this work, there are
methods that necessarily require interventional data (Brouillard et al. 2020 Ke et al. 2023a Scherrer et al.
2022
and learning- based algorithms unsuitable for amortized inference (Lachapelle et al. 2020 Ng al.
2020
Zheng et al. 2018 Zhang et al.
2022 Bello et al. 2022
Differences with Lopez-Paz et al. (2015) . Before moving forward, remark on the main differences
between our paper and Lopez-Paz et al. (0015), as both works concentrate on supervised learning for the inference of bivariate causal graphs. Lopez-Paz et al. (2015) frames causal discovery as a classification problem, where the goal is estimating and mapping the kernel mcan embeddings of the distribution of the observed data to the correct causal order (assuming a causal relation is in place). Building on the theory of reproducing kernel Hilbert spaces, they provide finite sample learning rates. In particular, their study assumes observations generated by identifiable causal models. In contrast, we aim to (i) empirically investigate what conditions enable identifiabilty in amortized causal discovery (fi) theoretically and empirically investigate how to exploit the known identifiability results to train algorithms with improved test generalization.
3 Background and motivation
We start introducing structural causal models (SCMs), intuitive framework that formalizes causal relations. Let X be set of random variables in R defined according to the set of structural equations:
X, = fA(X =i=
(1)
N, R are noise random variables. The function fa is the causal mechanism mapping the set of direct causes X of X; and the noise term N₁, to X₂'s value. The causal graph G is directed acyclic graph (DAG) with nodes X {X1,.... X and edges {X, X, X, E xpaç), with PA indices of the parent nodes of X, in G. The causal model induces a density px over the vector X.
3.1 Causal discovery from observational data
Causal discovery from observational data is the inference of the causal graph & from a dataset of i.i.d. obscrvations of the random vector X. In general, without restrictive assumptions on the mechanisms and the noise distributions, the direction of edges in the graph not identifiable, i.e. it can not be found from the population density px. In particular, it is possible to identify only Markov equivalence class, which is the set of graphs encoding the same conditional independencies as the density px. To clarify with an example, consider the causal graph X1 + X2 associated with structural causal model inducing density Px,,,, If the model is not identifiable, there exists an SCM with causal graph X₂ + X, that entails the same joint density PX..x.. The set {X1 + X2, + {1} is the Markov equivalence class of the graph X1 + X2, ie. the set of all graphs with X,, X2 mutually dependent. Clearly, in this setting, even the exact knowledge of PX1.X₃ cannot inform us about the correct causal direction.
Definition 1 (Identifiable causal model). Consider structural causal model with underlying graph G and px joint density of the causal variables. We say that the model is identifiable from observational data if the density px can not be entailed by a structural causal model with graph I + G.
We define the post- aad--additive noise model (post-ANM) the causal model with the set of equations:
X, = fex(f, + N.), =i= 1....,d
(2)
with f21 invertible map and mutually independent noise terms. When f2, is nonlinear function, the post-ANM amounts the identifiable post-nonlinear model (PNL) Zhang & Hyvärinen 2009). When fe.: is the identity function and fı, nonlinear, simplifies to the nonlinear additive noise model (Hoyer
3

---

Published in Transactions on Machine Learning Research (12/2025)
et al. 2008 equat lion::ons:
Peters et al. 2014 which is known to be identifiable, and is described by the set of structural
x= [x,(( + N,-
(3)
Tf, additionally, we restrict the mechanisms fi, to be linear and the noise terms N₂ to non- Con-Ganssian distribution, we recover the identifiable linear non- Gaussion additive model or LiNGAM (Shimizu et al.) 2006
x; E azX, No, aj E R.
(4)
JEPA
The special case of linear Gaussian models, with noise lerms with equal variance, is also known to be identifiable [Ptters & Bühlmann 2013
3.2 Motivation and problem definition
Causal discovery from observational data relies specific assumptions, which can be challenging to verify in practice (Montagna et al. 2023a). To address this, recent methods leverage supervised learning for the amortized inference of causal graphs (or simply amortized cousal discovery), Le, optimization of an inference model to directly predict a causal structure from newly provided data [Lopez Pazz--aaeee et al. 2015 Li et al. 2020;
Lippe et al. 2022; Lorch et al. 2022 Ke et al., 2023a, Löwe et al. 20200- While these approaches also aim to reduce reliance on explicit identifiaility assumptions, they often lack a clear connection to the existing causal discovery theory, making their outputs generally unreliable. We illustrate this limitation through example.
We consider the CSIvA transformer architecture proposed by Ke et al. (2023b), which can learn a map from observational data to a causal graph. The authors of the paper show that, in the infinite sample regime, the CSIvA architecture exactly approximates the conditional distribution p(-ID) over the space of possible graphs, given datasel D. Identifiability theory in causal discovery tells that if the class of structural causal models that generated the observations is sufficiently constrained, then there is only one graph that can fit the data within that class. For example, consider the case of a dataset that is known to be generated by nonlinear additive noise model, and let p(-ID, ANM) be the conditional distribution that incorporates this prior knowledge on the SCM: then pD-1D, ANM) concentrates all the mass on a single point g", the true graph underlying the observations. Instead, in the absence of restrictions on the structural causal model, all the graphs in a Markov equivalence class are equally likely to be the correct solution given the data. Hence, p(-ID), the distribution learned by CSIvA, assigns equal probability to each graph in the Markov equivalence class of G*.
Example 1.
The claims made in our Example are valid for all lcarning methods that approximate the conditional distribution over the space of graphs given the input data (Ke et al. 2023b Lopez-Paz el al. 2015 et al. 2020 Lippe et al. 2022 Lorch et al., 2022), and suggest that these algorithms are at most informative about the equivalence class of the causal graph underlying the observations. However, the available empirical evidence does not seem to highlight these limitations, in practice these methods can infer the true causal DAG on several synthetic benchmarks. Thus, further investigation is necessary if we want to rely their output in any meaningful sense. In this work, we analyze these "black-box' approaches through the lens of established theory of causal discovery from observational data (causal inference often lacks experimental data, which we do not consider). We study in detail the CSIvA architecture (Ke et al.) 2023b) (described in Appendix A). , variation of the transformer neural network (Vaswani et al., (017)) for the supervised learning of algorithms for amortized causal discovery. This model is optimized via maximum likelihood estimation, i.. finding θ that minimizes -Eg, II 6(g DD Q)), where $(9 DD; +) is the conditional distribution of graph G given dataset D parametrized by O. We limit the analysis to CSIvA as it is simple yet competitive end-to-end approach to lcarning causal models. While this is clearly a limitation of the paper, our thcorctical and empirical conclusions exemplify both the role of thea identifiability in modern approaches and the new opportunities they provide. Additionally, it fits well within a line of works arguing that specifically transformers can learn causal concepts Jin et al. (2024): Zhang et al. (0024); SSetboon et al. (2024) and can be explicitly trained to identify different assumptions, such as lincarity of the mechanisms or normality of
the noise terms, when doing in-context learning (Gupta al. 2023 Brown al. 2020
4

---

Published in Transactions on Machine Learning Research (12/2025)
4 Experimental results through the lens of theory
In this scction, we present comprehensive analysis of bivariate causal discovery with transformers and its relation to the theoretical boundaries of causal discovery from observational data We show that suitable assumptions must be encoded in the training distribution to ensure the identifiability of the test data, and we additionally study the effectiveness of training on mixtures of causal models to overcome these limitations, improving generalization abilitics. In Appendix C we discuss how our findings can be naturally extended to the case of multivariate causal models: the intuition that, in this case, identifiability can be guaranteed by iteratively verifying that the causal order of all bivariate subgraphs is individually identifiable (Theorem 28 in Peters et al. (0014)) As result, it is common to limit the analyscs to bivariate graphs (see e.g. Hoyer et al. p0008; Zhang & Hyvärinen 2009 Immer et al. 20023] Xi et al. 2025 ) which justifies our choice.
4.1
Experimental design
We concentrate our research on causal models of two variables, causally related according to one of the two graphs X + Y, Y + >X. Bivariate models are the simplest non-trivial setting with a well-known theory of causality inference (Hoyer et al. 2008: Zhang & Hyvärinen 2009; Peters et al., 2014). but also amenable to manipulation. This allows for comprehensive training and analysis of diverse SCMs and facilitates a clear interpretation of the results.
Datascts. Unless otherwise specified, in our experiments we train CSIvA on a sample of 15000 synthetically generated datasets, consisting of 1500 i.i.d. observations. Classes of SCMs are defined by the mechanism type and the noise terms distribution (e.g., linear non-Gaussian); each dataset is generated from a single SCM instance sampled from that class. The coefficients of the linear mechanisms are sampled in the range [-3, (o5 3]. removing small coefficients to avoid close-to- dlose-touuffatthe un-aattffhu effects (Uhler et al. 2012). Nonlinear mechanisms are parametrized according to a neural network with random weights, a. strategy commonly adopted in the literature of causal discovery (see Appendix B.2 alternatively, we provide experiments on data generated simulating nonlinear mechanisms by sampling from a Gaussian process, as described in Appendix D.7. The post-nonlinearity of the PNT. model consists of a simple map 2 + 23. Noise terms are sampled from common distributions and randomly generated density that call mlp, previously adopted in Montagna jet al. (2023a), defined by standard Gaussian transformed by multilayer perceptron (MLP) (Appendix B.2 We name these datasets mechanism-noise to refer to their underlying causal model. For example, data sampled umnlinarrgguussiar More details on the synthetic data generation schema are found in Appendix B.2 All data are standardized by their empirical variance to
from a nonlinear ANM with Gaussian noise are named
remove opportunities to learn shortcuts (Geirhos et al. 2020 Reisach el al. 2021 Montagna el al. 2023b)
Metric and random baseline. As our metric we use the structural Hamming distance (SHD), which is the number of edge removals, insertions or flips required to transform the predicted graph to the ground-truth. In the context of bivariate causal graphs with a single edge, this simply an error count, so correct inference corresponds to SHD 0, and an incorrect prediction gives SHD 1. Additionally, we define reference random baseline, which assigns causal direction according to a fair coin, achieving SHD = 0.5 in expectation. Each architecture we analyze in the experiments is trained 3 times, with different parameter initialization and training samples: the SHD presented in the plots is the average of each of the 3 models on 1500 distinct. test datasets of 1500 points each, and the error bars are 95% confidence intervals.
We detail the training hyperparameters in Appendix B.T. Next, we analyze our experimental results, starting by investigating how well CSIvA generalizes on distributions unseen during training.
4.2 Warm up: is CSIvA capable of in and out-of-distribution generalization?
In-distribation generalization. First, we investigate the generalization of CSTvA on datasets sampled from the structural casual model that generates the train distribution, with mechanisms and noise
The code for CSIvA implementation can be found here The code for reproducing the experiments of the paper can be found here
5

---

Published in Transactions on Machine Learning Research (12/2025)
CSIVA
benchmark
Naise distributions
gamme
gumbel
mpp
0.0
0.2
0.4
0.0
0.2
0.4
0.0
0.2
0.4
SHD
SHD
(a) Linear
(b) Nonlinear
(c) PNT
Figure 1: In-distribution generalization of CSIvA trained and tested on data generated according to the same structural causal models, fixing mechanisms, and noise distributions between training and testing. baselines for comparison, use DirectLiNGAM linear SCMs and NoGAM nonlinear ANM (we use their (causal-lear) and dodiscover implementations). CSIvA performance is clearly non-trivial and generalizing well.
distributions fixed between training and lesting. We call this in-distrübution generalization. The main goal of these experiments is to validate that the performance of our CSIvA implementation is non- son-trivial. As a benchmark, we present the accuracy of two statc-of-the-art approaches from the literature on cansal discovery: we consider the DirectLiNGAM and NoGAM algorithms (Shimizu et al. 2011 Montagna et al., 2023e3, respectively designed for the inference on LiNGAM and nonlinear ANM generated data?] The results of Figure [ show that CSIvA can properly generalize to unseen samples from the training distribution: the majority of the trained models present SHD close to zero and comparable to the relative benchmark algorithm.
Out-of-distribution generalization. In practice, we generally do not know the SCM defining the test distribution, so we are interested in CSIvA's ability to generalize to data sampled from a class of cansal models that is unobserved during training. We call this out-of-distrihutee generalization to different noise terms, analyzing the network performance on datasets generated from causal models where the mechanisms are fixed with respect the training, while the noise distribution varies (c.g., given linear-mlp training samples, testing occurs linear-uniorm data). Orthogonally empirically validate CSIvA's OOD generalization over different mechanism types (lincar, nonlinear, post- nonlinear), while leaving the noise distribution (mlp) fixed across test and training. In Figure 2a we observe that CSIvA cannot generalize across the different mechanisms, as the SHD of a network tested on unseen causal mechanisms approximates that of the random baseline. Further, Figure 2b shows that out-of-distribution generalization across noise terms does not work reliably, and it is hard to predict when it might occur. We note that these findings are novel the literature: OOD experiments the sense we define can not be found in Ke et al. p ; Li et al. (2020) empirical results are limited to the OOD generalization from linear Gaussian models to linear models with Exponential, Gumbel and Poisson noisc. Lorch ct al. (2022) analyscs generaliza- tion from SCMs with Gaussian noise to Laplace and Cauchy distributions, and lixed mechanisms class. The remaining literature amortized inference that we discuss in Section 2 ggneraally disregards these experiments.
generalization (OOD). We study OOD
these experiments,
Implications. CSIvA generalizes well to test data generated by the same class of SCMs used for training, in line with the findings in Ke et al. (003b)), which validates our implementation and training procedure. However, it struggles when the test data are out-of-distribution, generated by causal models with different.
synthetic nost-noulinear data, and observed that this due the sensitivity of the algorithm to the variance scale, So report the plot of Figure Ic without benchmark comparison. We remark that the point of this experiment is not make any claims on CSIvA being state-of-thhe but validate that the performance we obtain in our re-Implementation non-trivial. This clear for PNT., without comparison.
The causal-learn implementation of the PNL algorithm could not perform better than random h

---

Published in Transactions on Machine Learning Research (12/2025)
beta gamma
gumbel
mip uniform
llnear
nonlinear
exp
0.8
0.4
0.6
SHD
SHD
0.4
0.2
0.2
0.0
0.0
linear nonlinear pni Training dataset
linear
nonlinear
pnl
Training dataset
(a) Mechanisms
(b) Noise distributions
Figure 2: Out-of-distribution generalisation. We train three CSIvA models on data sampled from SCMs with linear, nonlinear additive, and post-nonlinear mechanisms; and fixed mlp noise distribution. Tn Figure 2a we test across
different mechanism types, with mlp-distributed noise terms both in test and training, In Figure 2b we test across
different noise distributions, with test mechanism types fixed from training. CSlvA struggles generalize to unseen causal mechanisms and often displays degraded performance over new noise distributions.
mechanism types and noise distributions than those it was trained on. From a practical perspective, this is a relevant finding, given that existing works on amortized causal discovery lack systematic experiments on the OOD setting. While training on a wider class of SCMs might overcome this limitation, requires caution. The identifiability of causal graphs indeed results from the interplay between the data-generating mechanisms and noise distribution. However, as we argue in our Example [ the class of causal models that supervised learning algorithm can identify is generally not clear. In what follows, we investigate this point and its implications for CSIvA, showing that the identifiability of the test samples can be ensured by imposing suitable assumptions on the class of SCMs generating the training distribution.
4.3
How does CSIvA relate to identifiability theory for causal graphs?
The CSIvA algorithm does not make structural assumptions about the causal model underlying the input data. This implies that the output of this method unclear: as CSTvA should target the conditional distribution p(-[D) over the space of graphs, in the absence of restrictions on the functional mechanisms and the distribution of the noise terms, the causal graph X + Y is indistinguishable from Y X, they are both equally likely to underlie the joint density px,y generating the data. As we discuss in Example [ the graphical output of the trained architecture could at most identify the equivalence class of the true causal graph. Yet, our experiments of Section 4.2 show that CSIvA is capable of good in-distribution generalization, often inferring the correct DAG at test time. We explain this seeming contradiction with the following hypothesis, which motivates the experimental analysis the remainder of this section.
Hypothesis (informal). At test time, the datasets on which CSIvA is expected to generalize well are determined by the family of SCMs on which training occurred, and can be analyzed using the tools from identifiability theory.
Notably, if this hypothesis is verified, we can analyse when CSIvA is expected or not to work well; the remainder of this work empirically studies this claim. Before moving forward, we present the following example adapted from Hoyer et al. (2008), which supports and clarifies our statement.
Example 2. Consider the causal model Y = f(X) + N, where f(X) = -X and PN are Gumbel densities px(x) = exp(-r - exp(-x)) and (N(n)) = exp(-n exp(-n)). This model satisfies the assumptions of the LINGAM, so it is identifiable, in the sense that a backward linear model (i.e., Y + with lincar mechanism f) with the same distribution does not exist. However, in this special case, we can build backward nonlinear additive noise model X g(Y) + Ñ with independent noise terms: taking py (v) exp(-y- log(1 +exp(-y)) to be the density of logistic distribution, p ( = exp(- sxp(-2ñ-expp) (inn-exx(-)))
7

---

Published in Transactions on Machine Learning Research (12/2025)
linear invertible
nanlinear invertible
validation
test (linear gauss)
0.5
0.75
SHD
SHD
0.25
0.25
0.00
0:100
33:67
50:50
67:33
100:0
50:50
Ratio of linear:nonlinear training datasets
Ratio of linear:nonlinear training datasets
(a) Iraerfäble data
(b) Linear Gaussian test data
Figure Experiments on identifiability theory. Figure 3a shows the SHD of models trained on different ratios of linear and nonlinear invertible data of Example 2 In Figure 3b we test the performance linear- Gaass-Gn data. Models are trained with different ratios of samples from linear and nonlinear SCMs with Gaussian noise terms. The validation results showcase that the networks were trained successfully. In both cases, CSTvA A behaves according to identifiabiliy theory, failing to predict on invertible data (50:50 ratio) and linear Gaussian models.
and g(v) = log(1 + exp(-u)); we see that Px,Y can factorize according to two opposite causal directions, as (2; v) = pw (y- - (xx)))xxx(() pN (y Given a dataset D of observations from the forward linear model X + Y, causal discovery methods like DirectLiNGAM (Shimizu et al. 2011 can provably identify the correct causal direction (X-> Y), assuming that sufficient samples are provided. Instead, the behavior of CSTvA seems hard to predict: given that the network approximates the conditional distribution p(-[D) over the possible graphs, for D with arbitrary many samples we have p(X Y|D) p(Y - X 0.5. On the other hand, given the prior knowledge that the data-generating SCM is a linear non-gaussian additive noise model, we have p(X + LiNGAM) = 1, because the LiNGAM is identifiable. In this sense, the class of structural causal models that CSIvA correctly infers appears to be determined by the structural causal models underlying the generation of the training data. Under this hypothesis, training CSIvA exclusively on BUIGAM-ggneratee data equivalent learning the distribution p(- [D, LiNGAM), such that the network should be able to identify the forward linear model, whereas it could only infer the equivalence class of the causal graph if its training datasets include obscrvations from nonlincar additive noise model.
The empirical results of Figure 3a show that CSTvA behaves according to our hypothesis: when training ex- clusively occurs on datasets {D.,+ generated by the forward linear-gumbel model of Example 2 the network can identify the causal dircction of test data generated according to the same SCM. Similarly, the transformer trained on datasets (D.,-h: from the backward nonlinear model of the example can generalize to test data com- ing from the same distribution. According to our claim, instead, the network that is trained on the union of the training samples {D U (D. Ji from the forward and backward models (50:50 ratio in Figure 3ap displays the same test SHD (around 0.5) as a random classifier assigning the causal direction with equal probability.
Further, we investigate CSTvA's relation with known identifiability theory by training and testing the architecture on data from a linear Gaussian model, which is well-known to be unidentifiable. Not surprisingly, the results of Figure 3b show that none of the algorithms that we learn can infer the causal order of linear Gaussian models with test SHD any better than random baseline.
Implications. Our experiments show that CSIvA learns algorithms that closely follow identifiability theory for causal discovery. In particular, while the method itself does not require explicit assumptions on the data-generating process, the chosen training data ultimately determines the class of causal models identifiable during inference. Notably, previous work has argued that supervised learning approaches in causal discovery would help with 'dcaling with complex data-generating processos and greatly reduce the need of explicitly crasting identifiability conditions a-priori', Lopez-Paz et al. (2015). In the case of CSIvA, A, this expect intecaaiion does not appear to be fulfilled, as the assumptions still need to be encoded explicitly in the training data. However, this observation opens two new important questions: (1) Can we train single network to correctly classify multiple (or even all) identifiable causal structures? (2) How much ambiguity might exist between these identifiable models? We start by answering this second question.
8

---

Published in Transactions on Machine Learning Research (12/2025)
4.4 An identifiability argument in favor of learning from multiple causal models
Example 2 of the previous section shows that elements of distinct classes of identifiable structural causal models, such as LiNGAM and nonlinear ANM, may become non-identifiable when we consider their union. In this section, we discuss the identifiabiliyy of the post-additive noise models. Previously, Hoyer ct al. (2008) showed that the set of distributions generated according to the additive noise model equation @ and that is non-identifiable negligible. Tater, /Zang Hyvärinen (2009) characterized non-identifiable post-nonlinear models in terms of the properties of their functional mechanisms, and the distribution of the noise terms. In this section, discuss how these results put together show that the set of distributions generated according to ANM that is non-identifiable is negligible.
Let X, Y be a pair of random variables generated according to the causal direction X - Y and the post-additive noise model structural equation:
= f
(5)
where Ny and X are independent random variables, and f2 is invertible. If the SCM is non-identifiable the data-generating process can be described by a backward model with the structural equation:
X= = (2) Nx),
(6)
Nx independent from Y, and 92 invertible. We introduce the random variables X, Y, such that the forward and backward equations can be rewritten
Y = fa()), == f(x) + Ny,
X 9x))
X = gi (Y) Nx.
We note that equivalently the following invertible additive noise models on X, hold:
y = by hX²++ Ny, by = fi 92. x= = (P) Nx, hx = 91 f2.
(7)
(8)
Equations 7 and (8] reduce the problem of studying the identifiability of post-ANM to that of studying
the identifiability of an additive noise model, done in Theorem 1 of Hoyer et al. (2008), which we repropose in Appendix E intuitively, the statement of the theorem says that the space of all continuous distributions generated according to a bivariate additive noise model and that is non-identifiable is contained in a 2-dimensional space. As the space of continuous distributions of random variables is infinite-dimensional, we couclude that the ANM is generally identifiable. Given that, according to Equation equation 7 and Equation equation X the post-ANM can be refactored in an additive noise model, the guarantees of identifiability still hold (for the formal statement and proof see Appendix B
Implications. As we discussed, the post-ANM is generally identifiable, which suggests that the setting of Example 2 is rather artificial. This result provides the theoretical ground for training causal discovery algorithms on datasets generated from multiple identifiable SCMs. This is particularly appealing in the case of CSIvA, given the poor OOD generalization ability observed in our experiments of Section 4.2
4.5
Can train CSIvA on multiple causal models for better generalization?
In this section, investigate the benefits of training over multiple causal models, Le. on samples generated by combination of classes of identifiable SCMs characterized by different mechanisms and noise terms distribution. Our molivation is as follows: given that our empirical evidence shows that CSIvA is capable of in-distribution generalization, whereas dramatically degrades the performance when testing occurs out-of- distribution, is thus desirable increase the class of causal models represented in the training datascts. We separately study the effects of training over multiple mechanisms and multiple noise distributions and compare the testing performance against architectures trained on samples of single SCM.

---

Published in Transactions on Machine Learning Research (12/2025)
mixed
single
in-distribution
Training datasets
lin,
Iin, nl
lin, pnl
lin, pnl
nl, pnl
nl. pnl
all
all
all
nl
pnl
nl
pnl
lin
lin
0.25
0.5
0.25
0.5
0.25
0.5
SHD
SHD
SHD
(a) Linear test data
(b) Nonlinear test data
(c) PNL test data
Figure 4: Mixture of causal mechanisms. We train four models on samples from structural casual models with different mechanism lypes. We compare their test SHD (the lower, the better) against nelworks trained on dalasets generated according to single type of mechanism. The dashed line indicates the test SHD of model trained samples with the same mechanisms as test SCM. Training on multiple causal models with different mechanisms (mized bars) always improves performance compared training single SCMs.
Mixture of causal mechanisms. We consider four networks optimized by training of CSIvA on datasets generated from pairs (or triples) of distinct SCMs, with fixed mlp noise and which differ in terms of their mechanisms lype: linear and nonlinear; nonlinear and post-nonlinear; linear and post-nonlinear; línear, nonlinear and post-nonlinear. The number of training datasets for each architecture is fixed (15000) and equally split between the causal models with different mechanism types. The results of Figure show that the networks trained on mixtures of mechanisms all present significantly better test. SHD compared to CSIvA models trained on a single mechanism type. We find that learning on multiple SCMs improves the SHD from ~ 0.5 to 0.2 both on linear and nonlinear test data (Figures 4a and 4b and even better accuracy is achieved on post-nonlincar samples, as shown in Figure
Mixture of noise distributions. Next, analyze the test performance of three CSIvA networks optimized samples from structural causal models that have different distributions for their noise terms, while keeping the mechanism types fixed. Figure 5 shows that training over different noises (beta, gamma, gumbel, exponential, mip, uniform) always results in a network that is agnostic with respect to the noise distributions of the SCM generating the test samples, always achioving SHD < 0.1, with the exception of datasets with mlp error terms (0.2 average SHD on linear and nonlinear data).
Implications. We have shown that learning on mixtures of SCMs with different noise term distributions and mechanism types leads to models generalizing to much broader class of structural causal models during testing. Hence, combining datasets generated from multiple models looks like promising framework to overcome the limited out-of-distribution generalization abilities of CSIvA observed in Section 4.2 However, it is easier to incorporate prior assumptions on the class of causal mechanisms (linear, non-linear, post-non-linear) compared to the noise distributions (which are potentially infinite). This introduces a trade-off between amortized inference and classical methods for causal discovery: for example, RESIT, NoGAM, AdaScore, and LNMIX (Peters al.] 2014 Montagna al., 2023c 2025 Liu al.) 2024) algorithms require assumptions the noise type, but only work for a limited class of mechanisms (linear and nonlinear with additive noise).
5 Conclusion
In this work, we investigate the interplay between identifiabilty theory and supervised learning for amortized inference of causal graphs, using CSIvA as the ground of our study. Consistent with classical algorithms, we demonstrate that good performance can be achieved if (i) we have a good prior on the structural causal model generating the test data (ii) the setting is identifiable. In particular, prior knowledge of the test distribution is encoded in the training data the form of constraints on the structural causal model underlying their generation. With these results, we highlight the need for identifiability theory in modern learning-based approaches to causality, while past works have mostly disregarded this connection. Further, our findings
10

---

Published in Transactions on Machine Learning Research (12/2025)
distributions
only mlp
beta
gamma
Test noise
gumbel
exp
mip
uniform
0.00
0.25
0.50
0.75
0.00
0.25
0.50
0.75
0.00
0.25
0.50
0.75
SHD
SHD
SHD
(a) Linear test data
(b) Nonlinear test data
(c) PNL test data
Figure 5: Mixture of noise distributions. We train three networks on samples from SCMs with different noise terms distributions and fixed mechanism types: linear, nonlinear, and post-noulinear. We present their test SHD (the lower, the better) on data from SCMs with the mechanisms fixed with respect to training, and noise terms changing between each dataset. Training on multiple causal models with different noises (all distributions bars) always improves performance compared to training on single SCMs with fixed mlp noise (only mlp bars).
provide the theoretical ground for training on observations sampled from multiple classes of identifiable SCMs, strategy that improves test generalization broad class of causal models. Finally, we highlight an interesting new trade-off regarding identifiability: traditional methods like LiNGAM, RESIT, and PNL require strong restrictions the structural mechanisms underlying the data generation (linear, nonlinear additive, post-nonlinear) while generally being agnostic relative to the noise terms distribution. Training on mixtures of causal models instead offers an alternative that is less reliant on assumptions on the mechanisms, while incorporating knowledge about all possible noise distributions in the training data is practically impossible to achieve. We leave to future work to reproduce our analysis on wider class of architect as well as extend our study interventional data with more than two nodes.
References
Kevin Bache and Moshe Lichman. Uci machine learning repository. 2013.
Kevin Bello, Bryon Aragam, and Pradeep Kumar Ravikumar. DAGMA: Learning DAGs via m-matrices and a log-determinant acyclicity characterization. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL https: //openreriew.ne//forreeeoe wwwtteeeeeeeeieeee Sett/fforum?dd-B82277FPU
Alexis Bellot and Mihacla van der Schaar. Conditional independence testing using generative ad- versarial networks. In H. Wallach, H. Larochelle, A. Beygelkimer, F. d' d'Alché-Bue, Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume Curran Associates, Inc., 2019. URL http:://ppooeetingg..eerrpppp. httppppppr..lll//pppp////////1/ dec4a357785166222770444662222000
pdf
Philippe Brouillard, Sébastien Tachapelle, Alexandre Lacoste, Simon Lacoste-Julien, and Alexandre Drouin. Differentiable causal discovery from interventional data. In H. Larochelle, M. Ranzato, Hadsell, M.F. Balcan, and H. Lin (eds.), Advonces in Neural Information Processing Systems, volume 33, pp. 21865 21877. Curran Associates, Inc., 2020. URL https: https://proceedings. htwp://ectaagg...rrrrp..c/pppppp fttppprrrfflle/ppppeeeeee
22000//10/00rraa0044400444444000 Paper. pdf
Philippe Brouillard, Perouz Taslakian, Alexandre Lacoste, Sebastien Lachapelle, and Alexandre Drouin. Typing assumptions improve identification in causal discovery. ar preprint arXim:2107.10711 2021.
11

---

Published in Transactions on Machine Learning Research (12/2025)
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Hemighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Πya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 1877 1901. Curran Associates, Inc., 2020. URL http:: htttps://proceeddinggg neurips.cc/papent faarr_ffellpppppp http://////o/ooooooooo....ooo/a.
.
Peter Bühlmann, Jonas Peters, and Jan Ernest. CAM: Causal additive models, high-dimensional order search and penalized regression. The Annals of Statistics, 42(6), dec 2014. URL https://doi. 1214%
2F14-aos1260
Tianyu Chen, Kevin Bello, Bryon Aragam, and Pradeep Ravikumar. iscan: Identifying causal mechanism shifts among nonlinear additive noise models, 2023.
Haoyue Dai, Rui Ding, Yuanyuan Jiang, Shi Han, and Dongmei Zhang. ML4C: Seeing Causality Through Latent Vicinity, pp. 226 234. 2023. doi: 10. 9118/1..781111997663..2226 URL htttps://ppuss... siam. siam.org/
doi/abs/10 1137/1 1te.1118111199798 ch26
Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix Wichmann. Shorteut learning in deep neural networks. Nature Machine Intelligence, 2:665-60 673, 11 2020. doi: 00.108//422266-00--11 0200066002200077-22
Clark Glymour, Kun Zhang, and Peter Spirtes. Review of causal discovery methods based on graphical models. Frontiers in Genetics, 10, 2019. ISSN 1664-8021. doi: //www. http:///wiin.oo/rrrraaee////...s
2019.00524. URL https
20100000
Shantanu Gupta, Cheng Zhang, and Agrin Hilmkil. Learned causal method prodiction, 2023.
Patrik Hoyer, Dominik Janzing, Joris M Mooij, Jonas Peters, and Bernhard Schölkopf. Nonlinear causal discovery with additive noise models. In D. Koller, D. Schuurmans, Y. Bengio, and L. Bottou (eds.), Advances in Neural Information Processing Systems, volume 21. Curran Associates, Inc., 2008. URL. https: pdf
//proceedings. crciingg.neeeeee wttp:///wwwppp....ooooooooooooaa
Alexander Immer, Christoph Schultheiss, Julia Vogt, Bernhard Schölkopf, Peter Bühlmann, and Alexander Marx. On the identifiability and estimation of causal location-scall noise models. In Proceedings of the 40th International Conference on Machine Learning, TCMT.23. JMR.R.rrg 2023.
Zhijing Jin, Yuen Chen, Felix Leeb, Luigi Gresele, Ojasv Kamal, Zhiheng Lyu, Kevin Blin, Fernando Gonzalez Adauto, Max Klciman- Kleiman-Weiner, Mrinmaya Sachan, et al. Cladder: A benchmark to assess causal reasoning capabilities of language models. Advances in Neural Information Processing Systems, 36, 2024.
Nan Rosemary Ke, Olexa Bilaniuk, Anirudh Goyal, Stefan Bauer, Hugo Larochelle, Bernhard Schölkopf, Michael Curtis Mozer, Christopher Pal, and Yoshua Bengio. Neural causal structure discovery from interventions. Transactions on Machine Learning Research, 2023a. ISSN 2835-8856. URL https:// Expert Certification.
opwnreview.net/ htt.:a/w.wettttoo.iiddddddveee.e
Nan Rosemary Ke, Silvia Chiappa, Jane X. Wang, Jorg Bornschein, Anirudh Goyal, Melanie Rey, Theophane Weber, Matthew Botvinick, Michael Curtis Mozer, and Danilo Jimenez Rezende. Learning to Induce Causal Structure. In International Conference on Learning Representations, September 2023b. URL https:://penneeiew..nettt
RwhKDJJ5
Jannik Kossen, Neil Band, Clare Lyle, Aidan Gomez, Tom Rainforth, and Yarin Gal. Self-attention between datapoints: Going beyond individual input-output pairs in deep learning. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021. URL
12

---

Published in Transactions on Machine Learning Research (12/2025)
Sébastien Lachapelle, Philippe Brouillard, Tristan Deleu, and Simon Lacoste-Julien. Gradient-based neural dag learning. In International Conference on Learning Represenlations, 2020. URL https://openreviier net/forun'idd=<klkkAaa
Hebi Li, Qi Xiao, and Jin Tian. Supervisod Whole DAG Causal Discovery, June 2020.
Juan Lin. Factorizing multivariate function classes. In M. Jordan, M. Kearns, and S. Solla (eds.), Advances in Newral Information Processing Systems, volume 10. MIT Press, 1997. URL htttp:://proceeddnng
Paper popperrp.ffff.
Phillip Lippe, Taco Cohen, and Eistratios Gavves. Efficient neural causal discovery without acyclicity
constraints. In International Conference on Learning Representations, 2022. URL https://openreviee
het/fforum'iddeecciPPl....
Wenqin Liu, Biwei Huang, Erdun Gao, Qiuhong Ke, Howard Bondell, and Mingming Gong, Causal discovery with mixed linear and nonlinear additive noise models: A scalable approach. In Francesco Locatello and Vanessa Didelez (eds.), Proceedings of the Third Conference on Causal Learning and Reasoning, volume 236 of Proceedings of Machine Learning Research, pp. 1237-1263. PMLR, 01-03 Apr 2024. URL
https:/ betps://proceedingg mlr. www.mmrr.peeeeeee arress/v236/1ine
html
David Lopez-Paz, Krikamol Muandet, Bernhard Schölkopf, and Ilya Tolstikhin. Towards learning theory of cause-effect inference. In Proceedings of the 32nd International Conference on International Conference on Machine Learning- - Volume 37, ICMI.'15, 15, pp. 1461. JMLR.org, 2015.
Lars Lorch, Scott Sussex, Jonas Rothfuss, Andreas Krause, and Bernhard Schölkopf. Amortized inference for causal structure learning. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL https://openreviee: http://pmreviiee...eefffrrrooooo
net/forum?id-
eVeJI-MMeX
Sindy Löwe, David Madras, Richard S. Zemel, and Max Welling, Amortized causal discovery: Learning to infer causal graphs from time-scrics data. In CLEaR, 2020. URL http:ss//app..snnattccoolll/////
Corpus1D:219955883
Riccardo Massidda, Francesco Landolfi, Martina Cinquini, and Davide Bacciu. Constraint-free structure learning with smooth acyclic orientations, 2023. URL httpp::/aaxxv..rrrg.e/// wwtp/ag//g/.g.o..o///gg 08406
Francesco Montagna, Atalanti Mastakouri, Flias Fulig, Nicoletta Noceti, Lorenzo Rosasco, Dominik Janz- ing, Bryon Aragam, and Francesco Locatello. Assumption violations in causal discovery and the robustness of score matching. In A. Oh, T. Neumann, A. Globerson, K. Sacnko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 47339-47377 Cur- ran Associates, Inc., 2023a. URL httpptttpp/oocceeeeiiiiii www.ree.e.ipp.eee. cccppppppppppppppppppppp.e... wttpc/ppr.ieeeeeppppeeee////e//e files//aperr/222//112 9hae7//888..aaabbccccc222c2222e2 Poreeeaeeeeeaapa--oooorrreeeeeer
Francesco Montagna, Nicoletta Noceti, Lorenzo Rosasco, and Francesco Locatello. Shortcuts for causal discovery of nonlinear models by score matching, 2023b.
Francesco Montagna, Nicoletta Noceti, Lorenzo Rosasco, Kun Zhang, and Francesco Locatello. Causal discovery with score matching on additive models with arbitrary noise. In 2nd Conference on Causal Learning and Reasoning, 2023c. URL /ttpps://openreviet net//forumiid=z200B.e.
Francesco Montagna, Philipp Michael Faller, Patrick Blöbaum, Elke Kirschbaum, and Francesco Locatello. Score matching through the roof: linear, nonlinear, and latent variables causal discovery. In Biwei Huang and Mathias Drton (eds.), Proceedings of the Fourth Conference on Causal Learning and Reasoning, volume 275 of Proceedings of Machine Learning Research, pp. 552 605. PMLR, 07 09 May 2025. URL htttss//prroceedngg wwwiiinnn.........mmmmmmm.. //5ms////2 html
RP Monti, Zhang, and A Hyvärinen. Causal discovery with general non-linen relationships using non-linear ica. 10 2019.
13

---

Published in Transactions on Machine Learning Research (12/2025)
Joris Mooij, Jonas Peters, Dominik Janzing, Jakob Zscheischler, and Bernhard Schölkopf. Distinguishing cause from effect using observational data: Methods and benchmarks. Journal of Machine Learning Research, 17: 1 102, 04 2016.
Joris M Mooij, Dominik Janzing, Tom Heskes, and Bernhard Schölkopf. On causal discovery with cyclic additive noise models. In J. Shawe-Taylor, R. Zemel, P. Bartlett, F. Pereira, and K.Q. Weinberger (eds.), Advances in Neural Information Processing Systems, volume 24. Curran Associates, Inc., 2011. URL https:/pprocedings.eeet pdf
Ignavier Ng, AmirEmad Ghassami, and Kun Zhang. On the role of sparsity and dag constraints for learning linear dags. In Proceedings of the 94th International Conference on Nearal Information Processing Systems, NIPS '20 Red Hook, NY, USA, 2020. Curran Associates Inc. ISBN 9781713829546.
Judea Pearl. Causality, Cambridge University Press, Cambridge, 2nd edition, 2009.
J. Peters and P. Bühhlann. Identifiability of gaussian structural equation models with equal error variances. Biometrika, 101(1):219 228, 11 2013. ISSN 0006-3444, doi: 10.1093/bionne aaoome/ URL https://doi org/10. 1tgg/00.1920 10/////ttt///tmm//tttt
Jonas Peters, Joris M. Mooij, Dominik Janzing, and Bernhard Schölkopf. Causal discovery with continuous additive noise models. J. Mach. Learn. Res., 15(1):2009 2053, jan 2014. ISSN 1532-4435.
Jonas Peters, Dominík Janzing, and Bernhard Schölkopf. Elements of Causal Inference: Foundations and Learning Algorithmss. Adaptive Computation and Machine Learning. The MIT Press, Cambridge, Mass, 2017. ISBN 978-0-266 278-0-262-03731-0.
Alexander G. Reisach, Christof Seiler, and Sebastian Weichwald. Beware of the simulated dag! causal discovery benchmarks may be easy to game. In Newral Information Processing Systems, 2021. URL
https://apl. peemmanncsccollaorggoooooo : 239998404
Patrik Reizinger, Yash Sharma, Matthias Bethge, Beruhard Schölkopf, Ferenc Huszár, and Wieland Brendel. Jacobian- based causal discovery with nonlinear ICA. Transactions on Machine Learning Research, 2023.
ISSN 2835-8856. URL https:://ppenreviex wtt.://eiw....eeeooooooo//e/oooo
Paul Rolland, Volkan Cevher, Matthäus Kleindessner, Chris Russell, Dominik Janzing, Bernhard Schölkopf, and Francesco Locatello. Score matching enables causal discovery of nonlinear additive noise models. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 18741-18753. PMLR, 17-23 Jul 2022. URL https::/prooceedngs.. mlr prees/vv62//ollaad22.. html
Karen Sachs, Omar Perez, Dana. Pe'er, Douglas A Lauffenburger, and Garry P Nolan. Causal protein-signaling networks derived from multiparameter single-cell data. Science, 308(2))): 529, 2005.
Meyer Scetbon, Joel Jennings, Agrin Hilmkil, Cheng Zhang, and Chao Ma. Fip: fixed-point approach for causal generative modeling, 2024.
Nino Scherrer, Olexa Bilaniuk, Yashas Annadani, Anirudh Goyal, Patrick Schwab, Bernhard Schölkopf, Michael C. Mozer, Yoshua Bengio, Stefan Bauer, and Nan Rosemary Ke. Learning neural causal models with active interventions, 2022.
Rajat Sen, Ananda Theertha Suresh, Karthikeyan Shanmugam, Alexandros G Dimakis, and Sanjay Shakkottai. Mlodel-powered conditional independence test. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (ods.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL
14

---

Published in Transactions on Machine Learning Research (12/2025)
Shohei Shimizu, Patrik O. Hoyer, Aapo Hyvärinen, and Antti Kerminen. A linear non-gaussian acyclic model for causal discovery. Journal of Machine Learning Research, 7:2003- dec 2006. ISSN 1532-4435.
Shohei Shimizu, Takanori Inazumi, Yasuhiro Sogawa, Aapo Hyvarinen, Yoshinobu Kawahara, Takashi Washio, Patrik Hoyer, and Kenneth Bollen. DireetTiNGAM: A direct method for learning linear non-gaussian structural equation model. Journal of Machine Learning Research, 12, 01 2011.
Peter Spirtes, Introduction to causal inference. Journal of Machine Learning Research, 11(54):1643 1662, 2010. URL http://mmr..orr//pppper/////////
html
Quang-Dus Duy Tran, Phuoc Nguyen, Bao Duong, and Thin Nguyen. Constraining acyclicity of differentiable bayesian structure learning with topological ordering. Knowledge and Information Systems, 66, 05 2024. doi: 00.1007//10115--22-0224444 02200-010444-
Ruibo Tu, Kun Zhang, Hedvig Kjellstrom, and Cheng Zhang. Optimal transport for causal discovery. In International Conference on Learning Representations, 2022. URL http:://ppenrrrier..rt www.rerriieeeeeetotoounmiiii qwBK94cPly
Caroline Uhler, G. Raskutti, Peter Bühlmann, and B. Yu. Goometry of the faithfulness assumption in causal inference. The Annals of Stalislics, 41, 07 2012. doi:
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL http:: htttps://proceeeinggg www.eediinnnnnnnnnn...mmm cc/paper files/paper/ Pappaaaapppppppp..
Xiaoqiang Wang. Yali Du, Shengyu Zhu, Liangjun Ke, Zhitang Chen, Jianye Hao, and Jun Wang. Ordering- based causal discovery with reinforcement learning. In International Joinl Conference on Artificial Intelligence, 2021. URL http::://ppp..eeaatt...ooooooooo atte//orpuulI::2346110000
Johnny Xi, Hugh Dance, Peter Orbanz, and Benjamin Bloem-Reddy. Distinguishing cause from effect with causal velocity models, 2025. URL
Jiaqi Zhang, Joel Jennings, Agrin Hilmkil, Nick Pawlowski, Cheng Zhang, and Chao Ma. Towards causal foundation model: on duality between causal inference and attention, 2024.
Kun Zhang and Aapo Hyvärinen. On the identifiability of the post-nonlinear causal model. In Proceedings of the Twenly- Fifth Conference on Uncerlainly in Arlificial Intelligence, UAI '09, pp. 647-655, 655, Arlington, Virginia, USA, 2009. AUAI Press. ISBN 9780974903958.
Zhen Zhang, Tgnavier Ng. Dong Gong, Yuhang Liu, Ehsan M Abbasnejad, Mingming Gong, Kun Zhang, and Javen Qinfeng Shi. Truncated matrix power iteration for differentiable DAG learning. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL https://openrevier. wwt.:/ernee/eiiideeooooooo |tte://ormmmmidiIaSSsasaaaa
Xun Zheng, Bryon Aragam, Pradeep Ravikumar, and Eric P. Xing. Dags with tears: Continuous optimization for structure learning. Tn Neural Information Processing Systems, 2018. URL https
//api. semanticscholar sew.articccco.ae...oooooooooaaaa orgg/oorpussD:55217777
15

---

Published in Transactions on Machine Learning Research (12/2025)
A Learning to induce: causal discovery with transformers
A.1 A supervised learning approach to causal discovery
First, we describe the training procedure for the CSIvA architecture, which aims to learn the distribution of causal graphs conditioned on observational and/or interventional datasets. We omit interventional datasets from the discussion as they are not of interest to our work. Training data are generated from the joint distribution PG,D between a graph 9 and a dataset D. First, we sample set of directed acyclic graphs {G Ji with nodes XI,..., Xd, from a distribution pç. Then, for each graph we sample dataset of observations of the graph nodes D {xt....,," ", = n.. Hence, build training dataset {G', Di y==1
The CSIvA model defines a distribution p= () of graphs conditioned on the observational data and parametrized by @. Given an ible map G + A from a graph to its binary adjacency matrix represent ation of d x entries (where As = 1 X, + X, in g), we consider an equivalent estimated distribution paip(: (), which has the following autoregressive form:
(AD(AA II [A - fe(A ,.. (.AAA...,,AAI,,)))))
where p(:ip) is a Bernoulli distribution parametrized by p. itself is a function of fe defined by the encoder-decoder transformer architecture; taking as input previous elements of the matrix A (here represented as a vector of d² entries) and the dataset D. è is optimized via maximum likelihood estimation, i.c. O* arg min - Egg, [In (G)DD; θ)}, which corresponds to the usual cross-entropy loss for the Bernoulli distribution. Training is achieved using stochastic gradient descent, in which each gradient update is performed using a pair (D',A") i 1..., d. In the infinite sample limit, we have p-) Q*) pt) while in the finile-cppccity case, it is only an approximation of the larget distribution.
A.2 CSlvA architecture
In this section, summarize the architecture of CSIvA, transformer neural network that can learn map from data to causally interpreted graphs, under supervised training.
Transformer neural network. Transformers (Vswwnni et al. 2017 are a popular neural network architec- ture for modeling structured, sequential data data. They consist of an encoder, a stack layers that learns a representation of each element in the input sequence based on its relation with all the other sequence's elements, through the mechanism of self-attention, and a decoder, which maps the learned representation to the target of interest. Note that data for causal discovery are not sequential in their nature, which motivates the adaptations introduced by Ke et al. 2023b: in their CSIvA architecture.
CSIvA embeddings. Each element x!. of an input dataset is embedded into a vector of dimensionality E. Half of this vector is allocated to embed the value x! itself, while the other half is allocated to embed the unique identity for the node X,. We use a node-specific embedding because the values of each node may have very different interpretations and meanings. The node identity embedding is obtained using standard ID transformer positional embodding over node indices. The value embedding is obtained by passing x/, through a multi-layer perceptron (MLP).
CSIvA alternating attention. Similarly the transformer's encoder, CSIvA stacks a number of identical layers, performing self-attention followed by a nonlinear mapping, most commonly MLP layer. The main difference relative to the standard encoder is in the implementation of the self-attention layer: transformers are in their nature suitable for the representation of sequences, given an input sample of D elements, self-attention is usually run across all elements of the sequence. However, data for causal discovery are tabular, rather than sequential: one option would be to unravel the n x matrix of the data, where n is the number of obscrvations and d the number of variables, into a vector of n-d d clements, and let this be the input sequence of the encoder. CSIvA adopts a different strategy: the self-attention in each encoder layer consists of alternate passes over the attribute and the sample dimensions, known as allernating allention Kossen et al.
16

---

Published in Transactions on Machine Learning Research (12/2025)
Hypeparameter
Value
Hidden state dimension
64
Encoder transformer layers
8
Decoder transformer layers
8
Num. attention heads
8
Optimizer
Adam
Learning rate
10-4
Samples per dataset (n)
1500
Num. training datasets
15000
Num. iterations
150000
Batch size
5
Table 1: Hyperparameters the training of the CSTvA models of the experiments Section
20211. As a clarifying example, consider dataset ((.i,))) of n i.id. samples from the joint distribution of the pair of random variables X1,X2 For each layer of the encoder, in the first step (known attention between allributes), attention operates across all nodes of a single sample (11.=)) to encode the relationships between the two nodes. In the second step (attention between samples), attention operates across all samples (xt......=))) x,), 21,2} of a given node, to encode information about the distribution of single node values.
CSIvA encoder summary. The encoder produccs a summary vector with clements for cach node X,, which captures essential information about the node's behavior and its interactions with other nodes. The summary representation formed independently for each node and involves combining information across the n samples. This achieved with method often used with transformers that involves a weighted average based on how informative each sample is. The weighting is obtained using the embeddings of a summary 'sample' + 1 to form queries, and embeddings of node's 's samples {ar to provide keys and values, and then using standard key-value attention.
CSIvA decoder. The decoder uses the summary information from the encoder to generate a prediction of the adjacency matrix of the underlying G. It operates sequentially, at cach step producing a binary output indicating the prediction A., As,j, proceeding row by row. The decoder is an autoregressive transformer, meaning that each prediction Aig is obtained based on all elements of A previously predicted, as well as the summary produced by the encoder. The mothod does not enforce acyclicity, although Ke et al. (023b shows that in cyclic outputs genereally don't occur, in practice.
B Training details
B.1 Hyperparameters
In Table [ we detail the hyperparametors the training of the network of the experiments. We define an iteration as a gradient update over a batch of datasets. Models are trained until convergence, using a patience of (training until five consecutive epochs without improvement) on the validation loss this always occurs before the 25-th epoch (corresponding to 150000 iterations). The batch size is limited to 5 due memory constraints.
B.2 Synthetic data
In this section, we provide additional details on the syuthetic data generation, which was performed with the causally Python library (Montagna al.) 2023a). Our data-gene framework follows that of Montagna al. (2023a) extensive benchmark of causal discovery methods different classes of SCMs.
lnttps:/causslly www.rreeedcs.io/eeeee..ee
17

---

Published in Transactions on Machine Learning Research (12/2025)
Distribution of the noise terms. We generated datasets from structural causal models with the following distribution of the noise terms: Beta, Gamma, Gaussian (for nonlinear data), Gumbel, Exponential, and Uniform. Additionally, we define the mip distribution by nonlinear transformations of gaussian samples from a guassian distribution centered at zero and with standard deviation a uniformly sampled between 0.5 and 1. The nonlinear transformation is parametrized by a neural network with one hidden layer with 100 units, and sigmoid activation function. The weights of the network are uniformly sampled the range [-1.5, 1.5). We additionally standardized the output of each mlp sample by the empirical variance computed over all samples.
Causal mechanisms. The nonlinear mechanisms of the PNL model and the nonlinear ANM model are generated by a neural network with one hidden layer with 10 hidden units, with a parametric ReLU activation function. The network weights are randomly sampled according to a standard Gaussian distribution (we refer to data with nonlinear mechanisms sampled according to this approach as NN-dala). The linear mechanisms are generated by sampling the regression coefficients in the range [-3, 00.5 [0.
NN-data generation: literature review. We present extensive list of works adopting neural networks
for the sampling of nonlinear mechanisms, similarly to our work: [Brouilard et al. (2020 2021 Lippe et al.
(2022 Bello et al.] (2022): Montagna et al. (2023a bb: Ke et al. (2023a b): Reizinger et al. (0023 ; Massidda
et al. 2023 Tran et al. (2024). This suggests that our data generation strategy established the literature
of causality. Additional experiments with sampling of nonlinear mechanisms from Gaussian processes are presented in Appendix D.7
Data are standardized with their empirical variance, which removes the presence of shorteuts which could be Iearned by the network, notably warsortability (Reisach et al., 2021 and score-sortability (Montagna et al., 2023b
B.3 Computer resources
Our experiments were replication purposes, GTX 1080 Ti's are entirely suitable, as the batch size was set to match their memory capacity, when working with bivariate graphs. All jobs ran with 10GB of RAM and 4 CPU cores. The results presented this paper were produced after 145 days of GPU time, of which 68 were on GTX 1080 Ti's, 13 on RTX 2080 Ti's, 11 on A10s, 19 on A40s, and 35 on RTX 3090s. Together with previous experiments, while developing our code and experimental design, we used 376 days of GPU time (for reference, at total cost of 492.16 14 Euros), similarly split across whichever GPUs were available at the time: 219 on GTX 1080 Ti's, 38 RTX 2080 Ti's, 18 on A10s, 63 RTX 3090s, 31 on A40s, and 6 on A100s.
local computing cluster, using any and all available GPUs (all NVIDIA). For
C CSlvA identifiability properties on multivariate SCMs
In the main manuscript, we limit our empirical and theoretical analysis of the identifiability guarantees provided by CSIvA to the case of bivariate causal models. In this section, we show how our findings are expected to extend to the multivariate setting. Our starting point is Theorem 28 from Peters et al. (2014): the intuition is that identifiability of multivariate additive noise models can be guaranteed by itcratively verifying that the causal order of all bivariate subgraphs is individually identifiable We formalize this reporting the following set of definitions and results from Peters et al. 2014)).
Condition 1 (Condition 19 of Peters et al. 2014 Consider additive noise model with structural equations X₂ = s(x)) + N,, X1, N independent rundom variables. The triple (f,xx, pPN) does not solve the following differential equation for all pairs æ1, x2 with ((z2))"(z)) - f(x1)) # 0:
5 1 (F-"""))
J" (('")2 f" + 1/ ft
5" =
(9)
1/1
f
Here, == logpx,; V === logPN, ,N. the logarithms of the strictly positive densities. The arguments x2 f((())) æi; and xı of U, and respectively, have been removed to improve readability.
18

---

Published in Transactions on Machine Learning Research (12/2025)
The intuition is that a bivariate additive noise model (which can be seen reparametrization of post- nonlinear model, as shown in our Section 4.4p is identifiable if it has density that satisfies the above Condition I This can be generalized to the case of multivariate ANMs where, for identifiability to hold, Condition must be verified for each pair of causally related variables in the SCM: when this is verified, we refer to a restrieled additive noise model.
Definition 2 (Definition 27 of Peters et al. 2014)) ). Consider additive noise model with structural independent noise terms and causal graph g. We call this
equations X; == fi(X + N; 1,,.... ,,....,, 1...=
SCM a restricted additive noise model if for all X3 X, X; E XPAς, and all sets Xs S X, S C N, with
Pa \ (Xx) C Xs XND XX), (where ND; is the set of non descendants of the node X3 in the graph 9); there value æg with plas) 0, such that the triplet
(FA May ), DxPxx PN,
satisfies Condition I Here, f) (TpA ((p+)) denotes the mechanism function x; + ()(g)))) Additionally, we require the noise variables to have positive densities and the functions fj to be continuous and three times continuously differentiable.
In the above definition we adopted the following notation: for a random vector = (Xi...., and set S C f1, we define Xs as the vector with elements {X:: E)..
Finally, the next theorem formalizes the intuition we've advocated far: the restricted additive noise model of Definition 2 i.e. an SCM whose pairwise causal relations are individually identifiable, is itself identifiable Thcorem (Thoorem 28 of Peters et al. (2014)). Let X be generated by a restricted additive noise model with graph 9, and assume that the causal mechanisms ʃ3 are nol conslant in any of the inpul arguments, i.e. for X, Xpag, there ezist x; & z such that f5(EPgg v.) # fy(EPAg 2.). Then, G is identifiable.
We note that this relation between bivariate and multivariate identifiability was recently exploited for causal discovery with optimal transport. by Tu et al. 2022
Discussion and multivariate identifiability guarantees of transformers. The above theorem states that a restricted ANM is identifiable. According to our Definition 2 an additive noise model X, = fn (x(g)A)) + NN, h = I,....,k is restricted if each pair connected nodes X, + X,, for each XpAg \ {X.} S Xs C CXND, \ XX), (think of Xs as the set of all possible causes of X,, except Xx),, can define a bivariate SCM of the form (F)( ()))=)), x
that satisfies Condition 1 i.c. that is identifiable. How does this rolate to our findings? Our experiments and analysis of Scction transformers align with the theory of identifiability in the case of training and inference on bivariate graphs. Given Theorem D we know that multivariate identifiability is a property of SCMs where each pair of causes and effects can define a bivariate structural causal model that itself identifiable; this implies that the empirical guarantees of identifiaillity we verify for transformers (via CSIvA) on bivariate models must extend to multivariate models. This is apparent by contradiction: say we train a CSTvA architecture that can infer the causal direction of a multivariate lincar Gaussian model (which is notoriously non-identiftable.. This means that our algorithm can infer the causal direction for each bivariate subgraph consisting of two variables connected according to a linear Gaussian structural equation: this would contradict our experimental results presented Fig. 3h and analyzed in Section 4.3
validate the hypothosis that
D Further experiments
In this section, provide additional experiments real-world data, the scaling properties of CSIvA in the number of training samples, and benchmark CSIvA performance in comparison to several well-cstablished or stale-of-the-art methods for causal discovery, with identifaaiiiity guarantees under different assumptions: DirectLiNGAM (Shimizu et al.) 2011) for inference on linear non-Gaussian models, CAM (ühhlmann et al., 2014) for inference of additive noise models with additive mechanisms, NoGAM (Montagna et al. 2023c) and GraNDAG (Lachapelle et al. (220)) for inference on ANMs (the latter taken from the continuous optimization literature of causal discovery, already mentioned in the related works Section
2
19

---

Published in Transactions on Machine Learning Research (12/2025)
SHD on real datasets
Training dataset
0.8
anm-mip
linear-mlp
mixed-mixed
0.6
pni-mip
SHD
0.4
0.2
0.0
autompg
sachs
sprinkler
tuebingen
Test dataset
Figure 6: Average SHD (the lower, the better) real-world datasets of CSIvA models that are trained synthetic datasets generated with linear, nonlinear additive, and post-nonlinear mechanisms and fixed mip noise distribution (linear-mlp, anm-mip, pul-mip bars) and méxed mechanisms and mized noise distributions (mixed-mixed bar). Performance is tested on bivariate models. We observe that the model optimized with mixed training is on par or outperforms the other algorithms.
D.1 Experiments real-world datasets
We consider the accuracy of CSTvA trained on different dataset configurations and tested on real-world datasets. In particular, we perform evaluation on the Tübingen pairs dataset (Mooij al. (0166), the Sachs biological dataset (Sachs et al. 2005), ; the AutoMPG dataset cars fuel consumption (Bache & Lichman 2013) and the Sprinkler dataset, simple dalaset on the causal relations between the binary categorical variables rain, sprinkler on/off, wet grass. Given that our algorithms are trained on bivariate models, from each multivariate dataset we extract all possible two variables subgraphs where this operation does not introduce new confounding effects. This results in 9 datasets from Sachs, datasets from AutoMPG, and 2 datasets from Sprinkler. We consider 102 pairs from the Tübingen dataset.
Real-world generalization of mixed-trained models. In Fig. 6 we illustrate the average accuracy per dataset type (Sachs, AutoMPG, Sprinkler, Tübbingen) of each CSTvA model. In particular, we want to probe the goodness of mixed training in real-world scenarios. To this end, we train four architectures on the following dataset configurations: linear-mlp, anm-mlp, pnl-mlp, mrized-mized, where the latter denotes the model trained SCMs with linear, additive nonlinear and post-nonlinear mechanisms, and Beta, Gamma, Gumbel, Exponential, MLP, and Uniform noise distributions. We find the following interesting outcome: the mixed-mixed architecture ure is on par with the others the Sprinkler and the Sachs datasets and outperforms the other methods on the AutoMPG and the Tübingen pairs datasets. Despite these results must be taken cautiously, they provide evidence of strong result, that mixed training appears to be beneficial even in real-world scenarios, those of actual interest in applications.
Benchmark with classic causal discovery. We probe CSIvA test generalization in comparison with DirecLiNGAM,, CAM, NoGAM, and GraNDAG methods. According to Fig. 7 interestingly we find that the mixed-mixed CSIvA model (trained on SCMs with linear, additive nonlinear and post-non mechanisms, and Beta, Gamma, Gumbel, Exponential, MIP, Uniform noise distributions) matches with or outperforms the other methods on all the test tasks. This provides additional empirical evidence on the benefits of the mixed training procedure propose to achieve better test generalization.
20

---

Published in Transactions on Machine Learning Research (12/2025)
SHD on real datasets
Algorithm
cam
csiva (mixed)
grandag
0.8
lingam
nogam
0.6
SHD
0.4
0.2
0.0
autompg
sachs
sprinkler
tuebingen
Test dataset
Figure 7: Average SHD (the lower, the better) real-world datasets. 'hee CSIvA model is trained synthetic datasets generated with mixed mechanisms and mized noise distributions (csiva (mixed) bar). As benchmark methods, consider DGMMM,,,NGAM, CAM, NoGAM, and GraNDAG. Performance tested bivariate models. We observe that the model optimized with mixed training is on par or outperforms the other algorithms.
D.2 Benchmarking CSlvA generalization with classical causal discovery algorithms
In this section, we analyze the results of Fig- where we compare the CSTvA trained on mixed mechanisms (linear, nonlinear, post-nonlinear) and mixed noises (all noise except for Gaussian) with the benchmark methods DirectLiNGAM, CAM, NoGAM, GraNDAG. Given that want to probe CSIvA test generalization, run inference over the following dataset configurations: linear-mixed (i.e. SCMs considering all possible noise distributions, except for Gaussian), anm-mixed, pul-mixed, and mixed-mlp (i.c. SCMs with lincar, nonlinear, post-nonlinear mechanisms). Fig. g shows results in line with our expectations: DiG,,,,,NGAM, CAM, NoGAM, and GraNDAG achieve their best accuracy on data generated by SCMs respecting their assumptions, while degrading their performance on the other models; the CSTvA architecture trained on a mixture of SCMs with different mochanisms and noise distributions matches with or tops all other methods, in all the considered settings (while being outperformed the and linear data, CSIvA still retains good average SHD accuracy).
D.3 Experiments with different sizes of the training dataset
In this section, we explore how CSIvA test generalization scales when training occurs on different numbers of training samples. In the experiments the main manuscript, each algorithm is optimized 15000 datasets, where each dataset and the underlying causal graph corresponds to training data point. We now compare the test SHD when training occurs on 5000 and 10000 datascts. One clear point emerges from the results of Fig. g that is our results on the benefits of the mixed training procedure are consistent for each size of the training dataset. we considered. Moreover, we note that the performance of CSTvA does not appear to degrade due to the decrease the number training points.
D.4 Can we learn to infer causal order from linear Gaussian data?
We ask whether CSIvA trained on non-identifiable models can implicitly learn to predict the causal direction of identifiable SCMs. For this purpose, we consider CSIvA optimized on linear Gaussian data and test. its performance on several datasets sampled from structural causal models with different configurations of mechanisms and noise distributions: lincar-mixed (with noise terms sampled according to all distributions except for Gaussian), anm anm-mixed, pnl-mixed, mixed-mlp (with mechanisms generated according to linear, nonlinear, post-nonlinear equations), anm-gauss, and pnl-gauss. The results of Fig. 10 present strong evidence
21

---

Published in Transactions on Machine Learning Research (12/2025)
SHD on simulated datasets
Algorithm
cam
csiva (mixed)
grandag
lingam
nogam
0.6
SHD
0.4
0.0
anm-mixed
linear-mixed
mixed-mip
pni-mixed
Test dataset
Figure 8: Average SHD (the lower, the better) simulated datasets. The CSTvA model is trained synthetic datasets generated with mized mechanisms and mired noise distributions (csiva (mized) bar). As benchmark methods, consider DirectLiNGAM, CAM, NoGAM, and GraNDAG. Performance tested bivariate models. We observe that, in general, the model optimized with mixed training is on or outperforms the other algorithms.
SHD for different size of the training daraset linear training
1.0 SHD for different size of the training dataset anm training
Tainng datasat Incarmme
Training dataset
5k
Inesembs-15k
3.6
0.8
0.2
0.2
innerrmmxdd dataset
mixedmmp
pnl-mixed
anm mixed
lincarmiisd dataset
mixoó-rii mip
pni-mixed
(a) LiNGAM training data (mlp noise)
(b) ANM training data (mlp noise)
SHD for different size of the training dotaset - pnl training
SHD for different size of the training datasat mixed training
1.0
1.0
Traningg dataset
Taining Sabast
pol-mip-sk
mined-mined-33
pol-mb--10x
mixed mixad 19k
prl-nlp-15k
mined-mized.
3.6
0.5
0.2
0.2
0.5
Tast miveedemi
(c) PNL training data (uupp noise)
(d) Mixed mechanisms and noise training data
Figure 9: Average SHD (the lower, the better) of CSIvA models trained with 5000, 10000, 15000 data points. The algorithums are tested on simulated datasets generated with linear, noulinear and post-nonlinear mechanisms (linear-mixed, anm-mixed, pnl-mixed entries on the x axis) and mixed-mip datasets, generated with mized mechanism types and fixed mlp noise distribution. We observe that (i) the mixed training improves the test generalization, irrespective of the training dataset size; (ii) CSIvA maintains its performance stable across different training dataset sizes.
22

---

Published in Transactions on Machine Learning Research (12/2025)
Test SHD for CSlvA trained on linear-gaussian data
0.8
0.6
SHD
linear-mixed anm-mixed
pni-mixed
mixed-mip
anm--euus
pni-gauss
Test dataset
Figure 10: Average SHD (the lower, the better) of CSIvA trained on datasets generated by linear Gaussian models, which are non-identifibble Performance is tested on simulated datasets generated according to several SCM configurations. We observe that training nou-identifiable data yields algorithm that performs with average accuracy of 0.5, equivalent to a coin flip random baseline, across all the test tasks.
that models trained on non-identifiable SCMs can not infer the causal order: in fact, we see that consistently across all datasets CSIvA average SHD approximates 0.5, the performance of classification with coin flip. This is in line with our expectations. In agreement with our motivating hypothesis (Section 4 in Section 4.3 have empirically shown that CSIvA can model the class of the SCM generating the obscrved data and exploit this information to infer the correct causal DAG (instead of a less specific Markov equivalence class) when this is identifiable. Moreover, in our Section 4.2 our experiments show that CSIvA can not generalize to SCM classes unseen during training. In light of these findings, it is intuitive that an architecture trained on non-identifaable linear Gaussian data can only try to fit a linear Gaussian model, irrespective of the input data. Then, when inferring the causal direction, given that CSTvA assumes the data to be generated according to linear Gaussian SCM, both the forward and backward directions are equally plausible, which explains the observed SHD close 0.5.
D.5 Experiments with bivariate independent graphs
In the main manuscript, we consider training and testing of CSTvA on bivariate graphs with an edge: -> Y, Y + X. This can be phrased as a classification problem with two labels. We motivate our choice by noticing that, in the bivariate set setting, identifiability defines a Markov equivalence class with one element, i.e. a singleton. This is known be identifiable without explicit assumptions on the functional form of the mechanisms the noise term distributions in the causal model. The goal of this section is to show that, if we include datasets generated according to empty graph in the training procedure, CSIvA can learn to disambiguate between the three classes (the empty graph, X Y + X). To motivate our claim, we notice that classifying empty and connected graphs can be done by testing independence between the input variables: previous works phrase independence testing as a classification lask and show that this can be learned via deep neural nerksrks (Bellol van der Schaar 2019 Sen et al. 2017). The experiments of Fig. [I sustain our claim. We consider three CSIvA architectures, each trained on independent pairs and one between lincar, nonlinear, or post-nonlincar data. Our results show that the neural network can learn to disambiguate between the three classes in all scenarios.
property of connected graphs: the emply graph with no edge
D.6 Mixed training with unlimited budget
We present our experimental results on one further question, to help clarify the results in the main text the paper. We aim understand when make tradcoffs between computational resources, and having models that have been trained on a wider variety of SCMs. We compare training on multiple SCMs single-SCM training, when all models the amount of training data. from each SCM type non-mixed model
23

---

Published in Transactions on Machine Learning Research (12/2025)
SHD on datasets with empty and connected graphs
0.08
Traringg catasets Infer-emptt anm cmppy
0.06
pri-emply
0.00
Inneremply
anm-emply Test daraset
pai-emptty
Figure 11: Average SHD (the lower, the better) for CSIvA trained on independent pairs and one between linear, nonlinear, or post-nonlinear data. Each algorithm is tested on the same class of structural causal models was trained We note that in all three scenarios, CSIvA learns to distinguish all classes with almost optimal accuracy (i.e., SHD close to O.
(i.e. a mixed network trains on 15, 000 linear datasets and 15, 000 PNL datasets, instead of 15, 000 divided between the two SCM types).
In the main text of this paper, we compare neural networks trained on mix of structural causal models (e.g. noise distributions, or mechanism types), to models trained on a single mechanism-noise combination, where all models have the same amount of training data, 15, 000 datasets. In mixed training, we split these evenly, so a "lin, ul' model trained on 7,500 datasets from linear SCMs, and 7, 500 from nonlinear SCMs. Our results in this framework are promising, and show that for many combinations of SCM types, we can train model instead of two, and achieve good progress, while making 50% savings training costs. However, if our training budget is high/ unlimited, we should also ask whether can achieve the same performance as model trained models trained with the same number of datasets per SCM type as an unmixed model had similar (or even better, for PNI. data) performance as the un-mixed model trained on the same SCM type as the test data. These mixed models are also siguificantly more useful than having or 3 separate models per SCM type, as they have good across-the board performance. However, we used the same computational resources to train separate networks (one for each mechanism type) and wanted to use them for causal discovery on dataset with unknown assumptions, we would be left with the rather difficult task of deciding which model to trust.
single SCM type. Fig. 12 shows good results in this direction the
D.7 Experiments with Gaussian process nonlinear mechanisms
In this section we present results obtained training and testing CSIvA on synthetic data with nonlinear mechanisms sampled from a Gaussian process with a unit bandwidth RBF kernel (we call data generated according to this approach as GP-data). In particular, for each variable X, node of the graph G generated according model cquation we define the nonlinear mochanism fx(X)A)) = AO ((XpAe. Xpaę)), a multivariate normal distribution centered at zero and with covariance matrix as the Gaussian kernel K ((Xpa,, Xpag), where XpAø are the observations the parents of the node X,. Together with our strategy adopted in the experiments in the main text of parametrizing nonlinearities with random neural networks (NN-data), this is one of the most common approaches in the literature.
GP-data generation: literature review. We present an exlensive list of works adopting Gaussian processes for the sampling of nonlinear mechanisms: Rolland et al. (2022); Montagna et al. (2023a b c):
Bühlmann al. (2014); Mooij al.] (2016); Lachapelle al. (2020); Wang al. (2021); Chen et al. (2023) ;
Mooij et al. (2011); Monti et al. (2019). This suggests that our data generation strategy established in the
causality literature.
24

---

Published in Transactions on Machine Learning Research (12/2025)
mixed (15k each)
mixed
single
in-distribution
Training datasets
lin, nl
lin,
lin, pnl
lin, pnl
nl, pnl
nl. pnl
nl
lin pnl
lin
pnl
0.25
0.5
0
0.25
0.5
0
0.25
0.5
SHD
SHD
SHD
(a) Linear test data
(b) Nonlinear test data
(c) PNT. test data
Figure 12: Mixtures of causal mechanisms, with varying amounts of training data. We train eight models on samples from structural casual models with different mechanisms. Four (in purple), were trained 15, 000 samples for each SCM type (so the "lin,nl" model saw 30, 000 samples in total, and the 'all' model saw 45, 000), and the other four (blue) are the same were trained on. We compare their test SHD (the lower, the better) against networks trained on datasets generated according single type mechanism. The dashed line indicates the test SHD model trained on samples with the same mechanisms the lest SCM. Traiuing on multiple causal models with different mechanisms (mixed bars) al always improves performance compared to training on single SCMs.
Fig. and were trained on 15, 000 samples in total, evenly split between the SCM types they
CSIvA
benchmark
beta
exp
Noise distributions
gumbel
mip
unform
0.0
0.2
0.4
0.0
0.4
0.2
SHD
SHD
SHD
(a) Linear
(b) Nonlinear
(c) PNI
Figure 13: In-distribution generalization (GP-data) of CSIvA trained and lested on data generated according the same structural causal models, fixing mechanisms, and noise distributions between training and testing, Nonlinear mechanisms for nonlinear and pnl data are sampled from a Gaussian process. As baselines comparison, we use DirM on linear SCMs and NoGAM on nonlinear ANM (we use their causal-learn) and dodiscover implementations). CSIvA performance is clearly non-trivial and generalizing well.
Summary of the GP-data experiments. Figures from 13 to 16 replicate the main text experiments involving nonlinear mechanisms either in the training or testing data. The results on GP-data agree with our findings on NN-data: CSIvA still shows poor OOD generalization under different training and test mechanisms, and generally for different training and testing noise distribution (except for PNL data). Similar to the case with NN-data, test gencralization improves under mixod training.
E Theoretical results and proofs
In this section, we state and prove the identifiability of the post-ANM discussed in Section as a corollary of Theorem of Hoyer al. 2008). The forward and backward models of equations equation and equation
25

---

Published in Transactions on Machine Learning Research (12/2025)
Inear
nonlinaar
pni
beta
gumbel
mip
gemma
exp
uniform
0.8
0.4
SHD
SHD
0.2
0.0
0.0
linear
nonlinear pni
Iiner
nonlinear
pni
Training dataset.
Training dataset
(a) Mechanisms
(b) Noise distributions
Out-of-distribution generalisation (GP-dala). We train three CSIvA models data sampled from SCMs with linear, nonlinear additive, and post-nonlinear mechanisms; and fixed mlp noise distribution. Nonlinear mechanisms for nonlineer and pnl data are sampled from Gaussian process. In Figure 14a we test across different mechanism types, with mlp-distributed noise terms both in test training. Figure 14b we test across different noise distributions, with test mechanism types fixed from training. CSTvA struggles to generalize to unseen causal mechanisms and often displays degraded performance over new noise distributions.
Figure 14:
mixed
single
in-distribution
Training datasels
lin. nl
lin,
lin, pn!
lin, pnl
ni, pnl
ni, pnl
all
pnl
pnl
0
0.25
0.25
0.25
SHD
SHD
SHD
(a) Linear test data
(b) Nonlinear test data
(c) PNL test data
Figure 15: Mixture of causal mechanisms (GP-data). train four models on samples from structural casual models with different mechanism types. Nonlinear mechanisms for nontinear and pnl data are sampled from a Gaussian process. We compare their test SHD (the lower, better) against networks trained on datasets generated according to single type of mechanism. The dashed line indicates the test SHD of model trained on samples with the same mechanisms as test SCM. Training on multiple causal models with different mechanisms (mized bars) always improves performance compared to training single SCMs.
26

---

Published in Transactions on Machine Learning Research (12/2025)
distributonss
only mip
beta
gamma
Test noise
gumbal
mlp
uniform
0.00
0.50
0.00
0.50
9.00
0.25
0.75
SHD
SHD
SHD
(a) Linear test data
(b) Nonlinear test data
(c) PNT test data
Figure 16: Mixture of noise distributions (GP-data). We train three networks on samples from SCMs with different noise terms distributions and fixed mechanism types: linear, nonlinear, and post-nonlinear. Nonlinear mechanisms for nonlineor and pal data are sampled from Gaussian process. We present their test SHD (the lower, the better) on data from SCMs with the mechanisms fixed with respect to training, and noise terms changing between each datasel. Training multiple causal models with different noises (all distributions bars) always improves performance compared to training on single SCMs with fixed mlp noise (only mlp bars).
for the pair of random variables X, Y is given by:
Y (x(f(( Ny) f2(9), fix) +N, ( Nx) = 92({)) x= (Y) NN.
with f2.92 invertible functions, Ny. independent random variables, and Nx, Y independent random
variables. Equivaaently, we can frame forward and backward causal models for X,Y, as in equations equation and equation
y = hx) Ny, by = f1 92. X hx Nx, hx g1 f2.
We ready to provide our identifiability statement for post-ANMs.
Proposition (Corollary of Theorem of Hoyer et al. (2008)). Let pNy hxx by be ftaed, and define My = Pog & = log Suppose that PNy and Px are strictly positive densities, and that vy, fis f2,91. 91, and 92 are three times differentiable. Further, assume that for fined pair hy, by crists ÿ E IR s.t. v (ī hy (z)) f 0 is satisfied for all but a countable set of points ā E R. Then, the set of all densities Px of X such that both equations equation and equation C are satisfied is contained in a 2-dimensional space.
Before stating the proof of Proposition we show under which condition the pair of random variables X, Y satisfies the forward and backward models of equations equation 5 equation 6 this is relevant for our discussion, as the proof of Proposition consists of showing that this condition almost never satisfied.
Notation. We adopt the following notation: vx loggNxx by = logPNy: E = loggx 17 py, and logPx.8
Theorem 2 (Theorem of Zhang & Hyvärinen (209)). Assume that X, Y satisfies both causal relations of equations equation 3 and equation o Further, suppose that PNy and px positive densities on the support of Ny and X respectively, and that vy,E, f1, 92, 91, and g2 are third order differentiable. Then, for each pair (z, ū) salisfying (i - hy (T)) (π) & 0, the sollowing differential equalion holds:
E.E,
2ithh ) V
24 (/(2) 22 + v,
5" g"
27

---

Published in Transactions on Machine Learning Research (12/2025)
and hx is constrained in the following way:
1 5" 14 (1ly)2 14-
(10)
x
Villy
where the arguments of the functions have been left out for clarity.
Proof of Theorem We demonstrate separately the two statements of the theorem.
Part 1. Given that equations equation and equation 6 hold, this implies that the forward and backward models on X,Y of equations equation 7 and equation g are also valid, namely that:
Y by + Ny.
X = hx + Nx.
These are the structural equations of two causal models, associated with the forward X + y and backward Y + X graphs, respectively. Applying the Markov factorization of the distribution according the forward direction, we get:
(x. = Prix (01 pxy (ū by (x))
which implies
(), vy (5)) + E
(11)
for any 8, ÿ. Similarly, the Markov factorization on the backward model implies:
x(2,0) xx hx (ū)) + 7(0).
(12)
From equation 12 we have that:
02
(w. 1) = v (2 - hx
0g2
0
( () = -1' hx hxx))))xxx
0roy
which implies
0
OF (i i 3)
8
0.
(13)
x (T. 7)
Jady
Computing the same set of partial derivatives from equation 11 we find:
0²
07.2 n (
82
n = - - by (1)
from which follows:
π [ 9)
S"
2214
by +
05
(2,
9)
(14)²
G20g
0.
where we drop the input arguments for conciseness. The equality with is given by the equality with equation 13 Manipulating the above expression, the first claim follows.
28

---

Published in Transactions on Machine Learning Research (12/2025)
Part 2. Next, prove the constraint derived on hx. To do this, exploit the fact that Y is independent of Nx, which implies the following condition
Lin
1997
0²
log p(u n2) = 0,
(14)
Ojōng
for any (ū, nz)- According to equations equation
equation
have that:
y by (X) + Ny,
Nx = X hx (Y),
such that we can define an invertible map & (7. nz)) + (ī, ny). It is easy to show that the Jacobian of the transformation has determinant |Jo] = 1, such that
P(ù ny) = p(ž. ny);
where (i, - (8, Thus, being X, Ny independent random variables, have that: logp(ū, nx) = logr(z) + logp(ny) = 5(Φ) + by (nx)).
Given that X = hxx(P) + Nx, we have that
0² Ōjōnx
log t'
while Ny - -X) implics
2² dnjnx
logp(ny) -i + - v'
such that
(T) = 5n1 + - + u (1x)) v'y
which must equal to zero, being equal to the LHS of equation
14
Thus, we conclude that
1 5" + / (hly) uy
N'
proving the claim.
E.1 Proof of Proposition
Proof. Under the hypothesis that equations equation 5 equation 6 hold, iee. when the data generating process satisfy both a forward and a backward model, by Theorem we have that:
Ø) H(
(15)
where
144 V., h4,
G(ī
1 h',,h, v4
(h4)2
H (ö, ()
20/h"/ + 1,,1"
Define 2 = -" such that the above equation can be written as 2'(8) = z(z)G(F, Ø) H(E Ø). given that such function exists, it is given by:
2(0) = z(ão)e G(t.))dt
L c. G(t.)dtt H df.
+
(16)
n0
29

---

Published in Transactions on Machine Learning Research (12/2025)
Let such that (Ø by (z)) # holds for all but countable values of i. Then, is determined by z(āo), can extend equation equation 16 to all the remaining points. The set of all functions & satisfying the differential equation equation 15 is a 3-dimensional affine space, as fixing E(Fo), ("o)) <" (ão) for some point žo completely determines the solution ç. Morcover, given Vy, hx, by fixed, 5" is specified by equation of theorem which implies:
Hhy h'x
<"
-
which confines & solutions of equation
15 a 2-dimensional affine space.
30


Reviews:
[{"review_id": "zwQkivkzIX", "reviewer": "Reviewer_kcRi", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors highlight a tension between the limitations of identification of graphs from observed data (i.e. that bivariate graphs are not a priori identifiable from observed data alone)  and the empirical evidence, which highlights that in practice many causal discovery methods perform well on synthetic benchmarks.  The authors investigate this phenomenon to understand why and under what conditions it is possible to identify a graph in spite of the lack of identification guarantees associated with the class of algorithms they consider, in particular the CSIvA architecture. \n\nPerhaps unsurprisingly, they show that suitable assumptions much be encoded in the training distribution for the successful identification of graphs in the test data. They stick to bivariate graphs but lean on the intuition that if one can identify a bivariate graph, one can identify a multivariate graph by the same logic (by extending the guarantees to all bivariate subgraphs).", "claims_and_evidence_yn": "Yes", "claims_explanation": "For evaluation the authors use a sample of 15k synthetic datasets each consisting 1500 observations. They standardise variance of the two variables (a detail which is often overlooked) to avoid any possible shortcuts for inference.\n\nAlso as expected, they find that CSIvA can generalise from train to test within distribution, but not out of distribution. Interesting this failure to generalise applies to both the mechanism mapping X to Y (or vice versa) as well as the choice of noise distribution.\n\nMotivated by the confirmation that CSIvA follows what one might expect from identifiability theory, they explore whether a model could be training on multiple (or in principle all) identifiable causal structures. One interesting example which suggests that this might be difficult relates to particular combinations of examples which result in a failure to identify the correct causal graph.", "audience_yn": "Yes", "audience_explanation": "Overall the paper concerns an important topic which is the limits of these amortized approaches to causal discovery.", "requested_changes": "Major:\n\nIt’s good to have this kind of empirical validation in ‘one place’ but I found  (a) the exploration rather simple and (b) the results rather obvious. Neither of these are reasons for rejecting a paper per se, but nonetheless, as I was reading the paper I kept waiting for the ‘Aha!’ moment, which never really arrived. If I had to paraphrase the conclusion, its that these amortised approaches to causal discovery provide no guarantees for identification (which we already knew), but empirically they show some reasonable performance if the examples they are trained on look like the examples they are tested on in terms of mechanism and distribution (which we also kinda knew). If the authors would argue that we really didn’t understand this already then maybe they can make a strong case by ‘identifying’ (excuse the pun) the failure of prior literature. This might make the work more compelling.\n\nI don’t want to be unfair to the authors, because the paper was well written, well structured, and easy to follow. I’d be interested to hear what the other reviewers think in terms of possible suggestions for making the contribution a bit more substantial. \n\nMinor:\n\nFigure 1 caption says that CSIvA is ‘generalising well’, suggest this is just clarified in the text to be in-distirbution generalisation so that its super clear when scanning through (I had to double check this was within-distribution generalisation and not OOD). Figure 2’s caption is clearer, for example.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ZuVpM8vIx1", "reviewer": "Reviewer_kUsp", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper offers a careful theoretical and empirical analysis of amortized causal discovery methods, with a particular focus on CSIvA. The authors’ main contributions are threefold:\n\n1. Empirical characterization of out-of-distribution generalization limits: They show convincingly that CSIvA does not generalize to classes of SCMs not included in training, particularly when mechanisms differ.\n2. Clarification of the dependence on training classes: They demonstrate that the SCM classes a model can correctly identify are precisely those it was trained on, in regimes where those families are identifiable. This ties amortized causal discovery performance to identifiability theory.\n3. Theoretical justification for training on mixtures: By repurposing identifiability results for post-additive noise models, they argue that mixtures of SCM classes dramatically improve generalization, and they support this claim with strong empirical evidence.\n\nTogether these contributions provide important clarity on what amortized causal discovery can and cannot achieve, unifying and demystifying previous work in the area.", "claims_and_evidence_yn": "Yes", "claims_explanation": "I answered “yes” here because the evidence is both accurate and convincing. The authors back each claim with clear experimental design:\n1) Large synthetic datasets (15k datasets with 1.5k samples each) generated under precisely defined SCM classes.\n2) Rigorous evaluation with structural Hamming distance (SHD), baselined against random guessing.\n3) Replication with three random initializations and 95% confidence intervals reported.\n\nThe empirical results directly support each claim: Figures 2a/b establish the OOD generalization failure; Example 2 illustrates how identifiability drives behavior; Figures 4 and 5 clearly demonstrate the benefit of training on mixtures of SCMs. In addition, the theoretical repurposing of Hoyer et al.’s identifiability result for post-ANMs provides a sound explanation for why the empirical findings make sense. Overall, the claims are neither overstated nor speculative; they are carefully aligned with the provided evidence.", "audience_yn": "Yes", "audience_explanation": "I also answered “yes” here. The findings are of clear relevance to multiple communities in the TMLR readership:\n1) Causal discovery: The results clarify when amortized approaches can be expected to succeed, and when identifiability barriers persist.\n2) Causal ML and causal representation learning: The analysis connects supervised learning methods like CSIvA to classical identifiability theory bridging two fields.\n3) Applied researchers: Anyone considering amortized causal discovery in practice gains an immediate understanding of what kinds of training/test settings are viable.\n\nBecause amortized causal discovery has attracted growing interest in recent years, this paper’s clarifications and demystifications are timely and important.", "requested_changes": "I have a few suggestions for clarity and polish:\n\nSection 3.2 wording: Replace “Our arguments of Example 1” with “The claims made regarding Example 1.” Clarify the final sentence (“Additionally, it…”): specify that CSIvA can be explicitly trained to recognize assumptions, and give a concrete example of what assumptions/contexts are meant.\n\nSection 4.1 wording: Instead of “Each dataset is generated according to a single class of SCMs,” write: “When studying a given SCM class (e.g. nonlinear-Gaussian), we generate each dataset from a single SCM instance sampled from that class.”\n\nSection 4.2 wording: Remove “in the first place” in “can not be found in Ke et al. (2023b) in the first place.”\nCorrect “that those” → “than those” in “different mechanism types and noise distributions than those it was trained on.”\n\nSection 4.3 wording: In Example 2, define “forward” and “backward” models before use. For example: “a backward linear model (i.e., Y → X with linear f()).” Also consider changing “encompass” in the posed question to “correctly classify.”\n\nGeneral: In the hypothesis box, consider a more precise phrasing, e.g.: “CSIvA generalizes to the SCM families it was trained on, in the regimes where those families are identifiable.” This would sharpen the main insight while remaining informal.", "broader_impact_concerns": "N/A", "additional_comments": "Nice work! Looking forward to your reply.", "extra_scores": {}, "extra_text": {}}, {"review_id": "tOvf71Tylb", "reviewer": "Reviewer_gJrK", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "1. The paper’s main contribution lies in connecting the emerging paradigm of transformer-based causal discovery with the foundational principles of classical identifiability theory. It helps explain why these seemingly black-box models work by arguing that the training data distribution effectively serves as an implicit set of assumptions (a prior).\n2. The paper also offers a clear and actionable insight: training on a mixture of datasets generated from different, individually identifiable causal models (e.g., ANM, PNL) significantly enhances the model’s out-of-distribution generalization.\n3. From my perspective, the core message is that when using amortized causal discovery methods, one should incorporate as much diversity as possible into the training datasets by drawing from more complex and identifiable data-generating processes. While this idea is, in essence, quite intuitive and straightforward, the authors provide value by formalizing it and presenting it in a clear and structured way.", "claims_and_evidence_yn": "Yes", "claims_explanation": "Yes. The claims made in the submission are supported by clear and convincing empirical evidence. The experiments are well-designed, and the results consistently align with the authors’ stated hypotheses. While the central claim is framed as an informal hypothesis rather than a formal proof, the empirical validation is sufficient and appropriate for the scope of the paper.", "audience_yn": "Yes", "audience_explanation": "Because it addresses a timely, interesting and also important problem in the new paradigm namely amortized causal discovery. By connecting transformer-based amortized causal discovery with the classical principles of identifiability, the paper provides both conceptual clarification and practical insights that can guide future research.", "requested_changes": "- Include the linear Gaussian model with equal variances in Section 3.1, as it is also identifiable under the ANM assumptions.\n\n- Try to evaluate a second model architecture. To strengthen the generality of the findings, validate key results on another transformer-based model in addition to csiva. This would demonstrate that the conclusions are not limited to a single architecture.\n\n- Discuss high-dimensional and complex settings. When extending the method, it would be valuable to include a discussion comparing your approach with methods specifically designed to handle complex mixture mechanisms, such as Causal Discovery with Mixed Linear and Nonlinear Additive Noise Models: A Scalable Approach. Such a comparison would help to better contextualize the limitations of the current work and to clarify promising directions for future research.", "broader_impact_concerns": "NA", "additional_comments": "Here are some limitations of this paper: \n\n1.The paper’s central claim, that transformers can learn to encode identifiability assumptions from training data, is presented as an informal hypothesis that is then empirically supported, rather than formally proven. While this is acceptable given the paper’s empirical emphasis, it should be clearly stated that the contribution is not a theoretical guarantee.\n\n2.The investigation focuses exclusively on a single architecture, CSIvA. However, the conclusions are phrased in broader terms as if they apply to amortized causal discovery with transformers in general. To avoid overgeneralization, the authors are encouraged to adjust the framing, particularly in the title and abstract, so that it better reflects the narrower scope of their empirical findings.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)