Do-PFN: In-Context Learning for Causal Effect Estimation
Authors: Jake Robertson
Arik Reuter
Siyuan Guo
Noah Hollmann
Frank Hutter
Bernhard Schölkopf
Venue: NeurIPS 2025 spotlight ()
Keywords: Prior-Data Fitted Networks
Causality
Causal Effect Estimation
CATE Estimation
Amortized Inference
In-Context Learning
Primary area: deep_learning
TL;DR: We demonstrate that the PFN-framework allows for the accurate estimation of causal effects under weakened assumptions.

Abstract: Causal effect estimation is critical to a range of scientific disciplines. Existing methods for this task either require interventional data, knowledge about the ground-truth causal graph, or rely on assumptions such as unconfoundedness, restricting their applicability in real-world settings. In the domain of tabular machine learning, Prior-data fitted networks (PFNs) have achieved state-of-the-art predictive performance, having been pre-trained on synthetic causal data to solve tabular prediction problems via in-context learning. To assess whether this can be transferred to the problem of causal effect estimation, we pre-train PFNs on synthetic data drawn from a wide variety of causal structures, including interventions, to predict interventional outcomes given observational data. Through extensive experiments in synthetic and semi-synthetic settings, we show that our approach allows for the accurate estimation of causal effects without knowledge of the underlying causal graph.

Do-PFN: In-Context Learning for Causal Effect Estimation
Jake Robertson 1,2mm,12, ,233+
Arik Reuter Siyuan Gu0
Noah Hollmann'
Frank Hutter 22,311 Bernhard Schölkopf 2.44
Prior Labs, Freiburg. Germany
ELLIS Institute Tübingen, Tübingen, Germany
University of Freiburg, Freiburg, Germany
Max Planck Institute for Intelligent Systems, Tübingen, Germany 5 University of Cambridge, Cambridge, United Kingdom
Equal contribution
Equal supervision
Abstract
Causal effect estimation is critical to a range of scientific disciplines. Existing methods for this task either require interventional data, knowledge about the ground-truth causal graph, or rely on assumptions such as unconfoundedness, restricting their applicability in real-world settings. In the domain of tabular machine learning, Prior-data fitted networks (PFNs) have achieved state-of-the- art predictive performance, having been pre-trained on synthetic causal data solve tabular prediction problems via in-context learning. To assess whether this can be transferred to the problem of causal effect estimation, we pre-train PFNs synthetic data drawn from wide variety of causal structures, including interventions, to predict interventional outcomes given observational data. Through extensive experiments in synthetic and semi-synthetic settings, show that our approach allows for the accurate estimation of causal effects without knowledge the underlying causal graph.
Introduction
The estimation of causal effects fundamental scientific disciplines such as medicine, economics, and the social sciences (Pearl 2009 Varian 2016 Imbens] 2024 Wu et al.] 2024). Questions such as "Does a new drug reduce the risk of cancer?" and "What is the impact of minimum wage on employment?" can only be answered by taking the causal nature of the problem into account.
The widely accepted gold standard for assessing causal effects (RCTs). While RCTs allow for the direct estimation of causal effects, they can sometimes be unethical or expensive, and, in many cases, simply impossible. In contrast experimental data from RCTs, observational data is more accessible, collected without interfering in the independent and identically distributed (i.i.d) data data-generating process. Estimating causal effects from observational data alone can be challenging or even impossible without strict assumptions (Spirtes et al.) 1993).
randomized controlled trials
Various methods have been proposed address the problem of causal effect estimation, typically relying on the assumption of unconfoundedness (Rosenbaum and Rubin 1983). This assumption states that, conditional on set of observed covariates, treatment assignment is independent of the potential outcomes. While this condition enables identification of causal effects from observational data, it can be difficult verify or justify in practice, it requires that relevant confounders are observed and properly accounted for (Hernán and Robins 2010 Imbens and Rubin (015)). Under unconfoundedness, variety of estimation techniques have been developed, including causal forests
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Pre-training
Simulated Inference
gab x
Do-PFN
e XPC
ye
dacetty
Figure Do-PFN overview: Do-PFN performs in-context learning (ICL.) for causal effect estimation, predicting conditional interventional distributions (CIDs) based on observational data alone. In pre- training, a large number of structural causal models (SCMs) is sampled. For each SCM, we sample an entire dataset of Mo observational data points Dob {(#)0 x U ) M We also sample M interventional data points D { x 5k (({=),"" Ar ))km
To simulate inference, we input (t"h x²1) along with the entire observational dataset Dob. which can have various sizes and dimensionalities. Subsequently, the transformer makes predictions ÿ. and we calculate the pre-training loss L(ū, y²") between the predictions and the ground truth interventional outcomes y"" Pre-training repeats this procedure across millions of sampled SCMs to meta- meta-learn how to perform causal inference in context. In upplications, Do-PEN leverages the many simulated interventions it has during pre-trainn pre-training to predict CIDs, relying only on observational data and requiring no information about the causal graph.
(Wager and Athey 2018). or "doubly robust" methods that combine propensity score modeling and outcome regression (Chernozhukov al. 2018).
Many applications of causality involve tabular data. Prior-data fitted networks (PFNs) Müller et al. have recently transformed the landscape of tabular machine learning. TabPFN (Hollmann et al. 2025), an application PFNs to tabular classification tasks, was initially met with skepticism, arguably because of its radically different working principle compared to other saate-of-the-att tabular machine learning methods; in a nutshell, it is model that takes as input an entire dataset of labeled training data points ((x,, y: )r)) and unlabeled test data points Fnew and "completes" with an in-confex! prediction of the corresponding ynew, akin to a large language model answering a question by producing a likely text continuation. The approach can handle datasets of various sizes and dimensionality (in which case matrices get zero- saroo-adddd). TabPFN is trained to execute this completion task across millions of synthetic datasets, for each dataset computing the likelihood of the true Vnew under the PFN's predictive distribution and performing one step of stochastic gradient descent on this likelihood. The context refers to the chunk of data that is fed into the model at time. In language model, these are thousands of words; in the case of TabPFN it is whole training set plus a query data point: and in our case also an added intervention. In-context learning refers to the ability output desired quantity based on what is provided in the context; the term learning indicates that this solves a task that would classically often require learning (e.g., estimating regression function and using it to predict a target). For us, the desired quantity is the effect of an intervention. Training such a model then consists of meta-learning (across millions of meta meta-training training datasets) the ability provide the desired answer given context. The term meta indicates that is an outer loop around a procedure that itself is already a form of learning or estimation.
2022
2023
In spite of being pre- pre-trained only on synthetic data, TabPFN has produced impressive results on real- world machine learning benchmarks (McElfresh et al.) 2023: Xu al., [025|] Hollmann et al., 2025). Given these remarkable findings, it is timely to assess whether similar meta leeaanle approach could help us tackle harder problems that are causal rather than merely predictive. Due to the delicate nature of causal inference, the sensitivity of the tasks associated with it, and the scarcity of causal effect estimation data with ground truth, we will do so by systematically exploring synthetic data with known ground truth.

---

Recent developments have shown that certain limitations in inferring causal structures and causal effects can be addressed by using domai- data in the form of mixtures of i.i.d. observational data (Guo et al. 2023 2024). Interestingly, PFNs also leverage pre-training on mixture of i.i.d. data to meta-learn how to solve predictive tasks at test time. We thus hypothesize that some causal tasks at test time could also be addressed through meta-learning on multi-domain data, As first step, our goal is to extend PFNs to the problem of estimating conditional interventional distributions (CIDs).
In contrast to TabPFN, we not only simulate observational tabular data in order to predict target feature. Rather, we additionally simulate causal interventions, teaching our model, which we call Do-PEN, to meta-learn how to perform causal inference.
Our contributions
1. Do-PEN: We propose Do-PFN, a foundation model pre-trained on data from structural causal models (SCMs) that can predict interventional outcomes and causal effects from observational data.
2. Semi-synthetic evaluation: We evaluate the performance of Do-PFN on six case studies across more than 1,000 synthetic datasets, the popular RealCause benchmark (Neal et al. 2020p, as well as two observational datasets with widely agreed upon causal graphs. We provide ablation studies within our prior, calibration, and evaluate Do-PFN against a competitive set of meta-learners, doubly robust, and deep-learning-basee
outto-ddistriuutin analysis, assess uncertainty
methods in the task of CATE estimation.
3. Theoretical results: In providing a mathematical underpinning for Do-PEN, we prove that it can achieve optimal approximation of the conditional intervention distribution (CID) concerning the chosen prior over data-generating functions. We also provide a characterization of the sources of uncertainty our model, and present consistency argument to show which types of uncertainty vanish with infinite data.
2 Background and related work
Structural causal models Structural causal models (SCMs; Pearl 2009 Peters al. 2017
represent the structure of a data-generating process. The first component of an SCM & is directed acyclic graph (DAG) Gw. which we assume have K nodes, each representing variable zu- Furthermore, the SCM specifies the mechanisms to generate the variables from their (causal) parents via structural equations 2 = f(((P)()): (((A()))()))) ck), where fi function, 2Px(k) denotes the parents of variable l in G and 6 is random noise variable. We use (e1, (2,-..,<K) denote the vector comprising the noise terms. In our simulations these will be taken as jointly independent, but our methodolody does not require this.
Interventions and causal effects In the context of SCMs, performing an intervention do(t) for a binary variable T { ... 2x} } that is part of the SCM & corresponds to removing all incoming edges into the node representing and fixing the value of the variable T to the value t. We assume the "treatment" T to be binary such that & (0, 1). The causal effect of this intervention on outcome y is captured by p(y) (do(t), v))
A central object of interest for this paper is the conditional interventional distribution (CID; Shpitser and Pearl 2006) that additionally conditions on a vector x comprising several variables the SCM,
plyldo(t),
(1)
A CID answers question like "What is the distribution of outcomes given that (i) a patient has features and (ii) an intervention do(t) is performed?" CIDs enable the estimation of conditional average treatment effects (CATEs): 7 = E (y) x] - E [y (0), x].
Estimating causal effects Various methods allow for the direct estimation of causal effects from experimental data (Shalit et al. 2017 Kennedy, 2023 Nie and Wager 202T). However, RCT data is often difficult to access. It might be easier, even the only option, to access observational dataset
D.a (v 19hh .xgg)) Map of passively collected samples (ug²) 19th ,ngh) ~ p(y, x))
When approaching causal effect estimation from the framework of do-caleulus (Pearl 2009), prac- titioners first need to construct SCM va that they believe (or have inferred) to represent the
3

---

ground-truth data gettrratnggnng process. The rules of do-calculus subsequently allow to determine whether and how the desired causal effect can be estimated from the data. Back-door and front-door adjustment are popular methods to allow for estimation of the desired causal effects.
The Neyman-Rubin framework (Imbens and Rubin 2013) defines causal effects as contrasts between potential outcomes /1 ~ p(y]do(I)) and yo ~- p(y y do(())) and relies on set of key assumptions, critically ignorability (or unconfoundedness), which requires that treatment assignment is independent of potential outcomes given a set of observed covariates. Machine-learning based methods that are conceptualized in this framework include causal trees (Athey and Imbens, 2016), causal forests (Wager and Athey 2018), as well as T-, S- and X-learners (Künzel et 2019).
Prior-data fitted networks and amortized Bayesian inference In our context, we define amortized (Bayesian) inference as learning the mapping D + ply x, D) from dataset to a posterior; that is, the amortization occurs at the dataset level. The model that parameterizes this mapping can be obtained by simulating large number of samples of the form (Di, Vr. Xz), followed by training the model to predict y, while conditioning on x; and D., Neural processes (Garnelo et al.) 2018a bb Nguyen and Grover 2022) and various techniques from the field of simulation-bart
inference (Wildberger et al. 2023; Gloeckler et al., 2024 Vasist et al. 2023) perform amortized inference the aforementioned manner. Recently, PFNs have been proposed as an amortized inference framework, emphasizing the role of sarge-scale pre- pre-training and realistic simulators of synthetic data, referred to as the prior (Müller et al. 2022). The PFN framework has been successfully applied to diverse problems, such as
time-series prediction (Dooley et al. 2023] Hoo et al. 2024). Bayesian optimization (Müller et al. Rakotoarison al. (022)) and counterfactual fairness (Robertson et al. 2024).
2023
Amortized causal inference Amortized inference beyond observational distributions was first
explored for causal discovery (Ke et al. 2022 Lorch et al. 2022 Dhir et al.. 2025 Sauter et al. (2025)
consider the problem of meta-learning causal inference, proposing to learn the shift in distributions of all nodes in the SCM when performing an intervention. However, this approach fails to outperform
conditioning- based baseline even in a two-variable setting. Nilforoshan et al. (2023) consider the problem of zero-shot inference for CATES using meta-learning approach to faci litate generalization to unseen treatments. Concurrent to our work. [ynum et al. (2025) propose to use amortized inference
to learn various causal effects; however, they only focus low-dimensional SCMs with up to three nodes and do not target the CID, but only point estimates, thus ignoring uncertainty.
3 Methodology: causal inference with PFNs
Modeling assumptions We now formalize how to do causal inference with PFNs, more precisely how to estimate conditional interventional distributions (CIDs) defined as p(y))o((), x) from ob- servational data Dub A central component of our approach to causal effect estimation is to posit a prior p(v)) over SCMs. We further require that every sampled SCM &b ~ p(v) allows to simulate observational data from ply goh, xab (v) by sampling noise ~ p(e) that is propagated through the SCMs v). We furthermore assume a prior p((")) over possible values for the treatment variable when performing an intervention do((*"). This prior is only required to sample values for the intervention and does not affect how we model the CID (Equation]] I provided it has sufficient support. Samples from the distribution p(y" xx [ do(t")) over outcomes and covariates given this intervention then result from gewward-roopggating through the intervened upon SCM. Please refer to Algorithm and Appendix C for more details on the geteraannng process.
The assumptions above imply the following form of the CID:
ply""Id((t")) (do(t")) x²") = f ply""Ido(("") (do(t")) xi". v) p(u)x )dbp
ply" in
Assuming a prior p()) over SCMs, and thus also over causal graphs G3. can be seen an extension of the classical do-calculus approach where typically a fixed causal graph Go. or even a fixed SCM vb is used as the basis for further inference. Compared to the assumptions typically made in the potential outcomes framework, our method also includes scenarios without the unconfoundedness assumption.
Note that in our framework p( p() A p((+) since knowing the feature vector in provides information on the SCM that generated it.

---

Algorithm 1: Prior-fitting with SGD. Do-PFN is pre-trained on pairs of synthetic observational and interventional datasets; the model is trained to predict interventional outcomes y given covariate- covariate-vector x"", the value of an intervention fd and an observational dataset Dob.
for i ..,....,NV. do Draw uh ~ p(());; // Draw an SCM
2
Initialize Dath +- 00:
Draw Mas ~ Uniform( ([Mmin Mmin + 1,..., M Mmaz)): // Number of
for observational data points
Mas do Sample noise Ej p(e);
Draw U5 tab. x3 ~ ~p("h, tab (x); Draw observational data
D - -Do00 (15 15 x
end
Initialize D:" + 0;
10
Set Min Mmax= Mobi for A = 1,2..... Min do
12
Sample noise Cj p(c);
Draw x "(xx" ex); Pre-treatment values of covariates
15
Draw 't p())((())))) Draw value for intervention
Draw yk ply Ido(( (x); Sample interventional outcomes
D. + DD:n {vii" th xxxx)) x
17
end
Compute (0) En - log qe (vi. [do(tt)), D; // Loss computation 0 0 VV. Gradient descent
19
20
end
Approximating the conditional interventional distribution Ultimately, we are interested in obtaining a model (o do(tt)),x" Dobb that as close as possible the CID ply"" [do["d" v)) for all relevant treatment values f, SCMs sb and devariate-vectone xt while only taking observational data Dob into account. The core idea of PFNs is to achieve this by prior fitting, i.c., minimizing the negative log- likelihood - log (y (do(t"), xt D on data from the synthetic data-generatie generating process (Müller et al. (002)) via stochastic gradient descent (lines 19 and 20 in Algorithm D] The following proposition shows that prior-fitting according to Algorithm achieves the goal of yielding an optimal approximation of the CID from observational data:
Proposition 1. Performing stochastic gradient descent according Algorithm 7 corresponds to minimizing the expected forward Kullback-Leii divergence between the conditional interventional distribution ply" x do(tt"), 3) and the distribution qo(y" in (do(ti)) t5n) x Dob) parameterized by the model,
Egin Sain,einpp Do p [DL [x do((fn), (lggg( (do(t"),, x [Dob]] J.
Here, the expectation is taken with respect to the Auta-generating distribution defined in Algorithm 7
The proof, given in Appendix follows from applying the conditional independences between variables implied by the data- matta-genraaing process Algorithm
Let us try to provide some insight about Proposition ! (i) It does not state that we can estimate all causal effects the traditional sense. To see this, note that the expectation is taken with respect the synthetic data-generating process. We could even drop the assumption of independent noise terms our SCMs, to train model that covers the aon-Markovinn case, and the proposition would still hold. (ii) Moreover, since our prior over SCMs does not necessarily imply identifiability of causal effects, an ideal property of our model would be that qo ( [do"" (""), ), x"n DDob accurately captures the uncertainty in the outcome arising from the unidentifiability of the causal effect of do(t")) on
"". Section 4.4 discusses empirical results indicating that Do-PEN is indeed able to do so.
We the do-notation in qo (y" (do(t")) ), x Dob indicate that model approximates the distribution of the outcome ' given an intervention on t'n This is formally not the result of applying the do-caleulus to an observational distribution qo (y (' x Dob).
5

---

Proposition 1 implies that the learned posterior-predictive distribution go go(y" [do((t)))) ). xin Draby forward-KL-optimal approximation of p>y""(>"" ) D
Sources of uncertainty
ply'n dd((t)) xtn 15) p( )dd, in expectation over p(x tin, D The details can be found in Appendix A.I Furthermore, when defining the observational equivalence relation Vi ~ we
- ply" fi (b)) ply"", xtn, 1aT| (11)))), where a selector y maps any SCM & onto the
representative r(3)) of its equivalence class [ab], one can obtain the following:
ply" |x"", do(t")), D = / ply"" Ix"" do(t"") ()p(()r)p(((DD")drr
(4)
This factorization allows to distinguish three types of uncertainty that the posterior predictive of Do-PFN captures: First, p(y" [x"" do(tt)), ) 1)) represents represents aleatoric uncertainty about the outcome yd caused by the noise terms the SCMs. Second, p(æ)r is the distribution over observationally equivalent SCMs given equivalence class [vb], and thus represents uncertainty due to unidentifiability. Finally, the distribution p(r) p(r(Dob) quantifies the epistemic uncertainty about the Markov equivalence class of ground truth SCM caused by finite observational dataset Dob
Consistency For infinite number of observational datapoints, i.e. |Dob + oc, the uncertainty in p(r(D=t) vanishes completely, but the other sources of uncertainty remain, More specifically, when the observational data Dob comes from an (assumed) ground-truth SCM tho the posterior conditional interventional distribution concentrates up to the Markov-equivalence class [Vo. of tho:
p(y" tm in do(t"
))")) + p(y""Ix" |x do(tt"), [sbo]) for [D + cc.
(5)
Please refer to Appendix B for rigorous formulation and proof of the statement above.
Architecture and training details Do-PFN transformer with a similar architecture to TabPFN (Hollmann et al. (002).. In order to specialize this architecture for predicting CIDs, we simply add special indicator to the internal representation of each input dataset specify that the first column is the treatment and the rest are covariates, Do-PFN has 7.3 million parameters and trained with Algorithm [ with details in Appendix C We primarily evalauate two versions of Do-PFN, vi and v1. 1, which are pretrained for 48 hours and 96 hours respectively on single RTX 2080.
4 Experiments
We thoroughly evaluate Do-PFN's performance and predictive behavior in CID prediction and CATE estimation against competitive set of causal and tabular machine learning baselines on combination of synthetic and hybrid synthetic-real peaeticc-rellleeee world datasets. First, we introduce our synthetic case studies (Section I. and evaluate Do-PFN against a competitive set of baselines in CID prediction (Section 4.3) and ATE estimation. Next, we perform several ablation studies (Section 4.4) across datasets within our prior, conduct an out-of-distribution analysis, and assess calibration of Do-PFN's uncertainty. Finally, evaluate Do-PP Do-PFN our semi-synthetic datasets (.55), where Do-PEN shows competitive performance with meta-learner, doubly robust, and deep-learning-basee CATE estimators RealCause, and outperforms them in our known graph case-studies. We provide our tra-ted model, pre- pre-training data generating code, and case study datasets /tttts:://gtthiii wtt.aaoo/ooo/rr//////////
4.2) as well as CATE (Section
4.1 Synthetic Data
Case studies We introduce several causal case studies that pose unique challenges for causal effect estimation, requiring adjustment based on the satisfaction of front-door and back-door criteria (Figure We therefore evaluate Do-PEN's ability to automatically perform this adjustment given only observational data. Please refer to Appendix D.I for more details.
2).
Data generation For each case study visualized in Figure 2 we independently sample 100 datasets with the corresponding graph structure, varying the SCM parameters as described in Appendix C We also vary the number of samples, standard deviation of noise terms, as well as edge weights and non non-lincasisies. The structural equations for case studies, as well as details regarding how

---

Observed
Observed
Confounder Mediator
Unobserved Confounder
Front-Door
Back-Don
Confounder
Mediator
Criterion
Criterion
Visualization the graph structures of our six causal case studies, requiring Do-PEN automatically perform adjustment based the front-door and back-door criteria. Treat- ment variables are visualized in orange, covariates x in red, and outcomes y in blue. Gray variables represent unobservables, not shown any of the methods yet influencing the generated data.
Figure 2: Case studies:
SCM parameters are sampled, are provided Appendix D.I and Appendix Table We additionally generate three case studies not visualized in Figure 2 which ablate over smaller dataset sizes Mmaz Uni 100]), complex graph structures with number of nodes ~ Uniform([4, 10]). and finally "Common Effect" case study which we show to be easily solved even by standard regression models (Appendix Figure T6).
4.2 Predicting conditional interventional distributions
First, we evaluate Do-PFN against competitive set of baselines for the task of predicting the CID p(y)do(t), x). In Figure 3 (first row). we visualize box-plots of normalized mean squared error (MSE) across our studies. For description of MSE, please see Appendix D.2
Effectiveness of pre-training objective In Figure significantly better the following tabular regression models: Random Forest, TabPEN (v2), as well our own regression model pre-trained on our prior to predict observational outcomes (dubbed "Dont-PFN"). This result offers two interesting findings.
first observe that Do-PFN performs
First, the substantial difference in performance between Do-PFN and Dont-PFN provides empirical evidence that our pre-training (Algorithm [ approximates something fundamentally different from just a standard posterior predictive distribution of observational outcomes, which in turn allows Do-PEN to precisely estimate causal effects.
Second, Do-PFN effectively handles the fact that samples in the context Dob and query Din sets come from systemically different data gentraggneaaingg processes, namely the original SCM and the intervened- upon SCM (in which the incoming edges to the treatment variable are removed). This causes mismatch the joint distributions p(106 yya)) and p(1 y ") precisely when there a directed edge between treatments / and covariates When the treatment and covariates are sampled independently, this mismatch disappears. This empirically validated by the similar performance of Do-PFN and tabular regression models on the Hammon-Effect case study (Appendix Figure [6). Across other case studies, we observe that TabPFN (v2) and Do-PFN exhibit similar performance when base rate treatment effects are small (Appendix Figure 9h.
Adjustment without graph knowledge Next, show that Do-PFN correctly perform the appropriate form of adjustment without explicit knowledge of the causal graph. When comparing Do-PEN "gold standard" baselines DoWhy (Int.) and DoWhy (Cntf.) (Appendix Figure 12). we observe that Do-PEN performs competitively with DoWhy (Int.) in CID prediction. Further, when comparing different variants of Do-PEN in Appendix Figure 15 observe that Do-PFN-Short, our shortest pre-trained model, performs competitively with Do-PFN-Graph a version of Do-PFN which is pre-trained solely the ground truth graph structure for each case study. These results together exhibit Do-PEN's ability to correctly perform adjustment as if applying the front-door and back-door criteria, while having no knowledge of the graph structure.
2 OOur par-plots visualize median values the 95% confidence interval. 'Please refer to the critical difference plots in Appendix E.6 Significance is assessed using post-hoc Nemenyi test implemented the Autorank package (Herbold 2020

---

Obeerved Genbondder
Diuerodd Modiator
Confounder + Medinter
Unobearved Confonddr
Front-Door Oritrrio
Back-Door Criterion
TAIPEN (A
UO0--FWN
ILANSS 0L000
0.0
213
0.2
(0:0)
I1:1 MIBB (CLL)
1000
0,06 MSE (CIL))
DUD (LIKI
0,004
(1,000
0,0055 (LIN3
MISE (CII)
XIS (CID)
MSE (CID)
Miski (CID)
PEN 2013
Concal Toreet DML
$1 Lecner (TAPPN)
T.
Lenner (TAPPE))
BMI TARNet
DrngonNet
10 SISE (GNTS)
10P MSE (CATE)
10-
10"
10'
10
MSE (CATI))
MSE (CATE)
MSE (CATTR)
SESE (CATI))
PEN 1e0)
Concal Torott DML
$ Loaener
(TSEPNN)
T- (TAEPEN)
Loumer
DMU TARRNe
Dongoniee
0.5 10
10
10
Rel. Eir (AT
Rel. Fer (ATE)
Rel. Ex (ATS)
Bel. Eer (ATE)
Rel. En (ATE)
Rel. Bix (ATE)
Figure 3: Results on synthetic data: Performance of Do-PEN in estimating conditional interventional distributions (CIDs, first row), conditional average treatment effects (CATEs, second row), and average treatment effects (ATEs, third row). Do-PEN provides strong performance across tasks.
4.3 Estimating conditional average treatment effects
We now evaluate Do-PEN's ability in CATE estimation, by calculating
+ +(x²)) Byinege (yen (do(1) 1e p.ot [yr"lly"" [y""] E Byin (y"n)d((0)) (di( (Daa) [y [g]
(6)
Robustness to causal assumptions In estimating CATE values in settings where the ground truth
is known, observe that Do-PFN significantly outperforms meta- meta--eerrners (Künzel et al.] 22199),
causal forest double machine learning (DML) approach (Wager and Athey [2018] Chernozhukoy methods DragonNet and TARNet (Curth al. 2021]. even on our relatively simple case studies (Figure 3 second row). We observe a similar trend in terms of ATE estimation (Figure 3 third row). In Appendix Figure IT, show that the performance of
al., 2018). as well as deep-learning-bast
our CATE baselines depends on the satisfaction of unconfoundedness, which causes our baselines to deviate in performance when this assumption not met. Do-PFN, on the other hand, remains more consistent in either case, further supporting our previous observation that Do-PFN is able to perform the appropriate adjustment in identifiable scenarios.
Robustness to unobserved variables We also observe that in the ground-trutt CATE estimation setting, Do-N performs closer the gold standard DoWhy (Cntf.) than previously in the more challenging CID prediction (Appendix Figure (3). Note that Do-PFN is especially competitive on the "Front-Door" and "Back-Door" case studies, where none of the models are given access to the unobserved variable; hence DoWhy loses the fundamental advantage that it had in other settings. This improved performance in the CATE estimation task further explained in Appendix Figure [4] in which we show via bias-decomposition that Do-PEN slightly over-predicts the CID in such way that cancels out when predicting the CATE.
4.4 Ablation studies
Furthermore, we perform several ablation studies to elucidate Do-PFN's predictive behavior across datasets with various different causal and statistical characteristics.
Dataset size and complexity In an evaluation of MSE in CATE estimation across datasets with a varying number of samples drawn such that Mmar ~ Uniform([5, 2000)), we observe that Do-PFN performs competitively with DoWhy (Cntf.) and its performance continues to improve and becomes more consistent as dataset size grows (Appendix Figure 9 We also find that Do-PFN performs competitively to DoWhy (Cntf.) across graph complexities, with slightly larger improvements for
8

---

OOD Noise Distribation
OOD Noa Limearity
0OD. Graph Size
Do-PEN (v])
0.20
DoPPN
DoPPN (v1)
0.06
D&PFN (yL.))
0.05
MSE. (C11))
0.04
0.05
0.10
0.04
0.02
0.00
0.00
0,00
10D
Laglare
T
Gombel Hlgh
Minvi
10D
ELU
Post
Moen Negntinn
10D
11-20
21-30
31-40
41-50
Nobee
Graydi (Nodee)
Figure 4: Out-of-distributiio:: Analysis of Do-PEN's performance on 500 in-distribution datasets (IOD) compared various OOD settings. Do-PEN is robust to different noise distributions (left) and various forms of functional non-linearity (middle). Do-PEN's (v1)'s performance deteriorates on larger graph sizes (right), which recovered by Do-PEN (v1.1) via larger-scale pre-training.
more complex graphs. Furthermore, we find that Do-PFNs can effectively use additional data points to alleviate increasing levels of noise (Appendix E.2
Base rate treatment effect We also show in Figure 9 in Appendix E.2 that Do-PFN remains consistent in MSE across different base rate levels of the ATE. This result shows that Do-PFN is robust to different magnitudes of base rate ATE, which is also beneficial in cases of problem misspecification, for example when a specified treatment does not influence an outcome.
Uncertainty calibration An analysis of prediction interval coverage probability (PICP) curves (Appendix E.5) confirms that Do-PEN is overall well calibrated. In the unidentifiable "Unobserved Confounder" case study, the model's uncertainty increases (Appendix Figure 17), as expected due to unidentifiability (see Section Meanwhile, the PICP curve confirms that PFN learns to exactly capture this increase in uncertainty. We also observe that Do-PFN's slightly under-confident for theoretically identifiable case studies. Reasons for this include the discretization error resulting from using bar-distribution to parameterize the posterior and a training loss that is not perfectly minimized. Since Do-Pl Po-P minimizes the forward KL-divergence to the ground-trun posterior, this leads to underdispersion Murphy (2023): McNamara et al. (2024).
Duttor-distribution analysis We find that Do-PEN's performance is robust to different noise distributions, including noise from Laplace-, T- and Gumbel-distribution,, all the aforementioned distributions; while using a two-times larger variance of the Gaussian noise drastically hurts performance (leftmost plot in Figure 4 The performance is also relatively robust to using different non-action activation functions and to using post-non-linearites noise after applying nonlinearity). However, testing the pointwise average of the functions used during training does lead to a notable decrease performance. Furthermore, we sample 500 synthetic datasets from our prior with graphs of up 50 nodes. For larger graphs out of its prior-distribution, we observe a significant reduction in Do-PFN-v11s performance. Do-PFN (v1.1), however, which was trained graph sizes up to 60 nodes, achieves significantly better performance large graphs with 21-50 nodes.
well as mixture of
lincarities (instead of adding the
ACIC1000
CPS
Lalard PSIn
Do (r1
Dot-PPN
Ouoaal
Sunraer (24142201
T. TLaceer (TA12PN)
Semus (SPO)
Gonal Frow DN1 (1P0))
Drsporeet
01200
0100
01005
0.00
2,00
0.30
1500
10.2
Cull
0.00
0.00
5002
M85 (CATE)
M453 (CATE)
3852 (CATR)
SISR
(SATR)
Figure 5: Results on RealCause: Performance of Do-PFN and our causal baselines in conditional average treatment effect (CATE) estimation on the RealCause benchmark. Do-PFN provides compet- itive performance in these semi-synthetic, unconfounded settings.

---

4.5 Hybrid syathetic-real-worldd data
RealCause: In order to evaluate on datasets from the RealCause benchmark (Neal et al., 2020). we pre-train a version of Do-PEN (which we call vl.1) on larger graphs with up to 60 nodes. We observe that Do-PFN-vI. I's extended pre-training leads to strong performance among competitive baselines three out of four RealCause datasets. These datasets are simulated to perfectly satisfy the unconfoundedness assumption, which inherently advantages the baselines over Do-PEN. Nonetheless, Do-PFN remains competitive with this strong set of CATE estimators, demonstrating that its synthetic pre-training transfers effectively to complex, real-world data. Furthermore, Do-PEN achieves a Pareto-optimal prediction speed and performance compared to all baselines (Appendix E.I
Furthermore, we evalu- ate on the Amazon Sales and Law School Admissions datasets where the causal graph is known (refer to Appendix DD..22 for more details). We also evaluate against six highly competitive baselines detailed in Appendix [D.3] We find that Do-PFN (v1) achieves the strongest performance on Law School and is not significantly worse than the best method on Amazon Sales, a dataset with complex mediation structure.
Known graph:
Low School (Race)
Amnasa Sals
Dadt'
Orned First Eac
Sumner (EP0)
Coaaa Eopor DMil, (EPU
Drspehhr
60
012
0.4
MEE (CATE)
MSU (CATE)
Figure 6: Results on known graph scenarios: Perfor- mance of Do-PEN and our causal baselines in CATE estimation the Law School Admissions and Amazon Sales datasets. Do-PEN outperforms other methods in mediated scenarios.
Discussion
We introduced Do-PFN, pre-trained transformer leveraging ICL to learn to predict interventional outcomes from observational data. Our empirical results on carefully controlled synthetic setups suggest that Do-PFN performs better in different causal scenarios, and more robust to the violation of commonly made assumptions, such as unconfoundedness, than a strong set of tabular and causal machine learning baselines. We also perform numerous ablation studies within our distiiiitti to elucidate Do-PEN's predictive behavior, and assess the calibration of uncertainty estimates showing both empirically and thcoretically that Do-PFN effectively captures the uncertainty that arises from unidentifiability.
Moving outside of our prior distribution, we perform an OOD analysis which shows that Do-PEN is robust to many forms of noise and non-linearities not seen explicitly during pre-training Our evaluations on two semi-synthetic benchmarks show Do-PEN's competitive performance when unconfoundedness is satisfied, and especially strong performance in mediated scenarios when this assumption is not met. Nevertheless, we need to discuss a range of limitations:
First, Do-PFN's generalization capability critically depends on its synthetic prior over SCMs, p(V). adequately capturing real-world causal complexity. As our current validation based on synthetic data, Do-PFN's robustness to prior- praortreality mismatches and its performance on diverse real-world datasets require further systematic exploration, alongside principled methods for prior design and validation. What makes us optimistic are empirical findings providing strong evidence that synthetic
prior fitting can lead to strong real-world performance (Hollmann et al.) 2025 Hoo et al., 2024 Reuter et al. 2025). as well as first theoretical results (Nagler 2023).
Second, Do-PFN's amortized inference approach offers efficiency but has different theoretical underpinnings compared to traditional, non-amortized estimators under known causal structures. The statistical theory concerning guaranties for such amortized models is still developing.
Third, many configurations and types of causal challenges were necessarily not addressed in this initial work. These include, for instance, broader array of intervention types beyond binary interventions and counterfactuals, non-i.i.d. input data, and various observational data characteristics or modalities not explicitly part of our current experimental scope. Incorporating them into our generating prior can give us a completely new handle on some of these hard problems.
To conclude, the key contribution of Do-PFN is novel methodology for causal effect estimation. We are optimistic about its prospects to become part of the standard MI. toolkit, thus helping to give causal effect estimation the broad accessibility that its real-world relevance deserves.
10

---

Acknowledgements
We acknowledge funding by the European Union via ERC Consolidator Grant DeepLearning 2.0, grant no. 101045765 and via the Horizon Europe research and innovation programme under grant agreement no. 101214398 (ELLIOT). Funded by the European Union. Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union, the European Commission, or the European Research Council. Neither the European Union nor European Commission nor the European Research Council can be held responsible for them. Frank Hutter acknowledges the financial support of the Hector Foundation. This research was also funded by the Deutsche Forschugsgemmcnschattt (DFG, German Research Foundation) under grant number 539134284, through EFRE (FEIH 2698644) and the state of Baden- Württemberg.
S. elliot
Funded by the European Union
Baden-Würrtemberg
MULTIMODAL MADE EUXUME

---

References
Athey, and Imbens, G. (2016). Recursive partitioning for heterogeneous causal effects. Proceedings of the National Academy of Sciences, 113(27):7333-736)
Balazadeh, V., Kamkari, H., Thomas. V., Li, B.. Ma, J., Cresswell, J. C., and Krishnan, R. G. (2025). Causalpfn: Amortized causal effect estimation via in-context learning. arXiv preprint arXiv:2506.07918.
Barber, D. and Agakov, F. (2004). The IM algorithm: variational approach to information maxi- mization. Advances in neural information processing systems, 16(320):201.
Battocchi, K., Dillon, E., Hei, M., Lewis, G., Miruna, O., and Vasilis, S. (2019). Econml: A python package for ml-based heterogeneous treatment effects estimation. https///ittub..com/py why/EconMM. Version O.x.
Blöbaum, P., Götz, P., Budhathoki, K., Mastakouri, A. A., and Janzing, D. (2024). Dowhy-gem: An extension of dowhy for causal inference in graphical causal models. Journal of Machine Learning Research, 25 (1147))1-7
Breiman, I.. (2001). Random forests. Machine Learning, 45:5-32.
Bynum, L. E., Puli, A. M., Herrero-Quevede D., Nguyen, N., Fernandez-Granda, C., Cho, K., and Rangamath, R. (2025). Black box causal inference: Effect estimation via meta prediction. arXiv preprint arXiv:2503.05985.
Chernozhukov, V. Chetverikov, D., Demirer, M., Duflo, E., Hansen, C., Newey, W., and Robins, J. (2018). Double/debiased machine learning for treatment and structural parameters.
Curth, A., Svensson, D., Weatherall, J., and van der Schaar, M. (2021). Really doing great at estimating cate? critical look at ml benchmarking practices in treatment effect estimation.
Dhir, A., Ashman, M., Requeima, J., and van der Wilk, M. (2025). A meta-learning approach to Bayesian causal discovery. In The Thirteenth International Conference on Learning Representa- tions.
Doob, L. (1949). Application of the theory of martingales. Le calcul des probabilites et ses applications, pages 23-27.
Dooley, S., Khurana, G. S., Mohapatra, C., Naidu, S. V., and White, C. (2023). Forecastpfn: Synthetically-trained zero-shot forecasting. Advances in Neural Information Processing Systems, 36:2403-2426.
Garnelo, M., Rosenbaum, D.. Maddison, C., Ramalho, T., Saxton, D., Shanahan, M., Teh, Y. W., Rezende, D., and Eslami, S. A. (2018a). Conditional neural processes. In International conference machine learning, pages 1704-1713. PMLR.
Garnelo, M., Schwarz, J.. Rosenbaum, D., Viola, F., Rezende, D. J., Eslami, S., and Tch, Y. W. (2018b). Neural processes. arXiv preprint arXiv:/807.0166
Gloeckler, M., Deistler, M., Weilbach, C., Wood, F., and Macke, J. H. (2024). All-in-one simulation- based inference. In Proceedings of the 41st International Conference Machine Learning, pages 15735-15766.
Guo, S., Tóth, V., Schölkopf, B., and Huszár, F. (2023). Causal de finetti: On the identification of invariant causal structure exchangeable data. In Thirty-seventh Conference on Neural Information Processing Systems.
Guo, S., Zhang, C., Mohan, K., Huszár, F., and Schölkopf, B. (2024). Do finetti: On causal effects for exchangeable data. Advances in Neural Information Processing Systems, 37:127317-127345.
Herbold, S. (2020). Autorank: python package for automated ranking of classifiers. Journal of Open Source Software, 5(48):2173.
Hemán, M. A. and Robins, J. M. (2010). Causal inference.
12

---

Hollmann, N., Müller, S., Eggensperger. K., and Hutter, F. (2023). TabPFN: A transformer that solves small tabular classification problems in second. In The Eleventh International Conference on Learning Representations.
Hollmann, N., Müller, S., Purucker, L., Krishnakumar, A., Körfer, M., Hoo, S. B., Schirrmeister, R. T., and Hutter, F. (2025). Accurate predictions on small data with a tabular foundation model. Nature, 637(8045):31--326
Hoo, S. B., Müller, S., Salinas, D., and Hutter, F. (2024). The tabular foundation model TabPFN outperforms specialized time series forecasting models based on simple features. In NeurIPS Workshop Time Series in the Age of Large Models.
Hunter, J. D. (2007). Matplotlib: A 2d graphics environment. Computing in science & engineering, 9(3::90-95.
Imbens, G. W. (2024). Causal inference in the social sciences. Armual Review of Statistics and Its Application, 11.
Imbens, G. W. and Rubin, D. B. (2015). Causal inference in statistics, social, and biomedical sciences. Cambridge university press.
Ke, N. R., Chiappa, S., Wang, J., Goyal, A., Bornschein, J., Rey, M., Weber, T., Botvinic, M.. Mozer, M., and Rezende, D. (2022). Learning to induce causal structure. arXiv preprint arXiv:2204.04875.
Kennedy, E. H. (2023). Towards optimal doubly robust estimation of heterogeneous causal effects. Electronic Journal of Statistics, 17(2):3008-33) 3049.
Kingma, D. P. and Ba, J. (2014). Adam: A method for stochastic optimization. arXiv preprint arXiv: 1412.6980
Künzel, S. R., Sekhon, J. S., Bickel, P. J., and Yu, B. (2019). Metalearners for estimating heteroge- neous treatment effects using machine learning. Proceedings of the national academy of sciences, 116(10)4456-4166
Kusner, M. J., Loftus, J., Russell, C., and Silva, R. (2017). Counterfactual fairness. In Advances in Neural Information Processing Systems, volume 30, pages 4069 4079.
Lorch, L., Sussex, S., Rothfuss, J., Krause, A., and Schölkopf, B. (2022). Amortized inference for causal structure learning. Advances in Neural Information Processing Systems, 35:13104-13118
Madigan, D. and Raftery, A. E. (1994). Model selection and accounting for model uncertainty in graphical models using occam's window. Journal of the American Statistical Association,
193455).1535-154.
Manber, U. (1989). Introduction algorithms: creative approach, volume 142. Addison-Wesley Reading, MA.
McElfresh, D., Khandagale, S., Valverde, J., Prasad C, V., Ramakrishnan, G., Goldblum, M., and White, C. (2023). When do neural nets outperform boosted trees on tabular data? Advances in Neural Information Processing Systems, 36:76336-76369.
McNamara, D., Loper, J., and Regier, J. (2024). Sequential monte carlo for inclusive kl minimization in amortized variational inference. In International Conference on Artificial Intelligence and Statistics, pages
PMLR.
Miller, J. W. (2018). A detailed treatment of doob's 's theorem. arXiv preprint arXiv:1801.00
Müller, S., Feurer, M., Hollmann, N., and Hutter, F. (2023). Pfns4bo:
In-context learning for Bayesian optimization. In International Conference on Machine Learning, pages 25444-25470. PMLR.
Müller, S., Hollmann, N., Arango, P., Grabocka, J., and Hutter, (2022). Transformers can do bayesian inference. In International Conference on Learning Representations.
Murphy, K. P. (2023). Probabilistic machine learning: Advanced topics. MIT press.
13

---

Nagler, T. (2023). Statistical foundations of prior-data fitted networks. In International Conference on Machine Learning. pages 25660-256777 PMLR.
Neal, B., Huang, C.-W., and Raghupathi, S. (2020). Realcause: Realistic causal inference bench- marking, arXiv preprini arXiv:2011.15000
Nguyen, T. and Grover, A. (2022). Transformer neural processes: Uncertainty- awwae meta learning via sequence modeling. In ICML.
Nie, X. and Wager, S. (2021). Quasi-oracle estimation of heterogeneous treatment effects. Biometrika, 10992))299-319
Nilforoshan, H., Moor, M., Roohani, Y., Chen, Y., Surina, A., Yasunaga, M., Oblak, S., and Leskovec,
J. (2023). Zero-shot causal learning, Advances in Neural Information Processing Systems, 36:6862- 6901.
Paszke, A. (2019). Pytorch: An imperative style, seghoperformnnce deep learning library. arXiv preprint arXiv:1912.01703.
Pearl, J. (2009). Causality. Cambridge university press.
Peters. J., Janzing, D., and Schölkopf, B. (2017). Elements of causal inference: foundations and learning algorithms. The MIT Press.
Rakotoarison, H., Adriaensen, S., Mallik, N., Garibov, S., Bergman, E., and Hutter, F. (2024). In- context freeze-thaw Bayesian optimization for hyperparameter optimization. In Proceedings of the 41st International Conference on Machine Learning, pages 41982-42008.
Reuter, A., Rudner, T. G., Fortuin, V., and Rügamer, D. (2025). Can transformers learn full Bayesian inference in context? arXiv preprint arXiv:2501.16825
Robertson, J., Hollmann, N., Awad, N., and Hutter, F. (2024). FairPEN: Transformers can do counterfactual faimess. In /CML 2024 Next Generation of Al Safety Workshop.
Rosenbaum, P. R. and Rubin, D. B. (1983). The central role of the propensity score in observational studics for causal effects. Biometrika, 70(1):41-55.
Sauter, A., Salchkaleybar, S., Plaat, A., and Acar, E. (2025). Activa: Amortized causal ef- feet estimation without graphs via serafoommr-bbsedd variational autoencoder. arXiv preprint arXiv:2503.01290.
Sauter, A. W. M., Acar, E., and Plaat, A. (2024). Causalplayground: Addressing data-generation requirements in cutting-edge causality research.
Shalit, U., Johansson, F. D., and Sontag, D. (2017). Estimating individual treatment effect: general- ization bounds and algorithms. In International conference on machine learning, pages 3076-3088 PMLR.
Sharma, A. and Kiciman, E. (2020). Dowhy: An end-to-end library for causal inference. arXiv preprint arXiv:2011.04216
Shi, C., Blei, D., and Veitch, V. (2019). Adapting neural networks for the estimation of treatment effects. Advances neural information processing systems, 32.
Shpitser, I. and Pearl, J. (2006). Identification of conditional interventional distributions. In Proceed- ings of the Twenty-Second Conference on Uncertainty in Artificial Intelligence, pages 437-444.
Spirtes, P.. Glymour, C., and Scheines, R. (1993). Causation, prediction, and search. Soringer-Verlage (2nd edition MIT Press 2000).
Varian, H. R. (2016). Causal inference in economics and marketing. Proceedings of the National Academy of Sciences, 113(27):731--3315.
Vasist, M., Rozet, F., Absil, O., Mollière, P., Nasedkin, E.,, and Louppe, (2023). Neural posterior estimation for exoplanetary atmospheric retrieval. Astronomy & &Astrophysics, 672:A147.
14

---

Wager, S. and Athey, S. (2018). Estimation and inference of heterogeneous treatment effects using random forests. Journal of the American Statistical Association, 113(523):1228-12))
Wang, C., Wu, Q. Weimer, M., and Zhu, E. (2021). Flaml: fast and lightweight automl library. Proceedings of machine learning and systems, 3:434-447.
Waskom, M. L. (2021). seaborn: statistical data visualization. Journal of Open Source Software, 6(60):3021.
Wightman, L. F. (1998). LSAC national longitudinal bar passage study. ISAC Research Report Series.
Wildberger, J., Dax, M., Buchholz, S., Green, S., Macke, H., and Schölkopf, B. (2023). Flow matching for scalable simulation-base inference. Advances in Neural Information Processing Systems, 36: 6837-168644
Wu, X., Peng. S., Li, J., Zhang, J., Sun, Q.. Li, W., Qian, Q., Liu, Y., and Guo, Y. (2024). Causal inference in the medical domain: A survey, Applied Intelligence, 54(6):4911-4934
Xu, D. Q Cirit, F. O., Asadi, R., Sun, Y., and Wang, W. (2025). Mixture of in-context prompters for tabular PFNs. In The Thirteenth International Conference on Learning Representations.
15

---

A Proof of Proposition |
The risk for a single interventional data point when using the NLL loss, as in Algorithm takes the following form:
Rs | - log(re (do((())))) x Dp((at , ,"" " a ae dyd dx
-
)dD'
Let's consider p(Dob, ,"", y ). Then we can obtain by first marginalizing out the distribution p(1)) of Structural Causal Models (SCMs) and, second, utilizing the factorization of the joint distribution implied by the data generating process Algorithm
/ D(Dro, imm y0 x vdd
p(Dob pi x())
1 p(y" x , x "do(t")), (pp("" ()(("
(8)
Now, we can use that
(,)))))))))))))) p(y" x (do(t")),, xb) ) px do(t"m), ())((x" (do(t""), 2)).
Further:
p(x'"|do(("" (do((t)) ),b))((" ())(((") (D0 (b)()))))))))))))))))))))))))))) = p(Das, 15n x ,b).
(9)
This implies
(DDbb 1 - "x do((')), (DD enn x 1b)
(10)
Plugging this into equation 7 followed by using that the entropy between distributions and q is equal to the Kullback-Leible divergence between and plus the entropy of p. formally
H (p,9) H (p) D (pllg), fact used by Müller et al. (2022) and Barber and Agakov (2004) in analogous scenarios, yields:
Ra= = /
log(go (do(t")), x Dobl)
ply""| do do(("n), ", (DD" pr x "))D di di" "x"" dub
/ [ Br [x do(t")) 2) go (y (do (y""(((( x, tn Dobl]
DDD 1't xt" sb ) ai dx dub C (11)
This implies that minimizing Ro results in (forward) Kullback-Leibbe optimal approximation of ply" (do(tin), ), 1. x²) with the model go (0 |do(tin) ), x", Dob) in expectation over the data simulated from p(b, Dob, gin xi").
Please note that analogous to PFNs, the optimality only holds when the expectation is taken with respect synthetic data-generating process. However, theoretical results by Nagler (2023) and plethora of empirical findings regarding the transferability of PFNs to real-world scenarios, well as
related approaches (Hollmann et al. 2025) Hoo al, 2024, Reuter et al.) 2025), provide evidence that synthetic prior fitting can lead to strong real-world performance.
16

---

A.1 Alternative version of proposition Furthermore, when using
I
ply Aly""dook "do"td" x Dob) Ido(t"))
x""", ,v) p(ab) ydb.
(12)
as for instance in Madigan and Raftery 1994), we can rewrite:
RR - || -
log(go [do(t""), (dd(tt x Dob))
p(y""I"" do(("m), (DD em, x",, w))d dy dx dil
log(go (do(t x Dobl)
f ply do(tt)), })p( x de p p(Dob, tim x dDD dDDbdtt de'n dy dyd dx
|||l-log(co0 log(g0 ), Dobl)
p(y"")d(((")) xtn Dath p(Dah 1 x DdD ai "ndd"""".. (13)
which, using the same arguments as before implies an alternative version of Proposition T
Proposition 2. Performing stochastic gradient descent according to Algorithm 7 corresponds to minimizing the expected forward Kullback- Kulbback-Leibler divergence between p(y" |do(t (m)) xtn Dob) and the distribution qo (y"" do((tn)), ).
xxx , ,Drh) parameterized by the model,
Egin gin Dot [KL
), Dob )lga (y" |do(t")), xtt Dob)]]
(14)
Note that, in comparison to Proposition 1
the expectation no longer taken over the SCMs.
B On the consistency of Do-PFN
In the following section, will show that the posterior predictive interventional distribution P(Y do([""), X Dot) of Do-PFN consistent up observationally equivalent SCMs. Assuming the observational data Dob were generated by an SCM Vo. could also have been generated by set [Vo] have (informally) that when the number of observational data points goes to infinity |Dob + C0.
P(yin do(tt)). X Dob) + P(yrin do(T""), X {
(15)
This means that asymptotically recover the posterior P(Y do(T X (Wo) under the of all SCMs [Yo] that yield the same observational distribution ( Mo) (Dab vr) for all
V6 E [V)].
To prove this fact, proof by (Balazadeh et al. 2025) which pertains to the consistency posterior expectations.
Doob's theorem (Doob) 1949 Miller 2018) and ideas that also appear in
B.1 Mathematical framework
In the following, we depart from the previous convention of exclusively working with probability densities and use capital letters to denote random variables with capital P denoting probability distribution with, if it exists, density wrt. the Lebesgue measure counting measure (if p is probability mass function) respectively.
Further, let ()), B) be the outcome space for y Let v be the random SCM under prior P(V)
over SCMs, and let
D,
[V Toa x )=
17

---

be the first n observational i..d. samples from the observational distribution P(Y T " vo under true SCM 4h- We assume throughout that all random variables take values complete separable metric spaces equipped with the induced Borel o-algeeras. Note that this requirement is satisfied for the simple case of real-valued (multivariate) random variables which represent the only relevant case for Do-PFN.
X
Define observational equivalence on the domain of Y by
+ P(Y T X v v) P(y T X
Φ).
Choose measurable selector V R v that r(b) is the unique representative of [+t]. and set
R R= To = r(vo).
Please note that we use this set of representatives, and the distribution P(R) = rVP((V. which is the pushforward of (( under to avoid working with set-valued random variables.
We write P(Y im do(("T). X"n, [uol) to denote P(Y" dd((""), X R == Ro).
Finally, define for each B B:
Nn (B) P(Y E B do(T" X Dr); (Bx = P(yin E B do(T" X R ro). Proposition (Consistency of Do-PFN), Assume observational data Dn. drawn iidd drawn from an Xb v =o)) Then, for sample size + 0o.
SCM Vo, i.e. Dn itd P(y ,ooh
P(Y do(T""), x Dn) " P(y do((T"n), ;²))
where " denotes weak convergence of measures.
Proof. Fix any Borel B S and define
Mn(B) P(y'n E B do(T" X -
Dn, do(Ti xin].
First tower step. By the law of total expectation,
Ma b E " DR, do(T""), X v) Dn, do(Tm)) x
Since knowing & renders the past data Dn irrelevant for the conditional interventional distribution,
E [[ Dn, d x = E [1 [1 do
X V
Furthermore, since P(V) Dn, X do(Tin)) = P.))DD)) due to the conditional independencies in algorithm
E (ymEBB) Dny do(Tin), x w Dns do(T xin) = =E[ Dny do(Tin), xin wi Dm
When assuming that densities exist, the steps above just represent the fact that ply" (lo x Dob) p'y""""do xt v) Dob)DDb)dss 2. Second tower step. Conditioning next on the representative R R=r(V),
(a B[B[B[[ ( do(Tin)) X w] Dm R = B[ga(R) Dml,
where
(r(r) = | [(ying)) |o((Ti)) in R= P(rin E B do(TTin) X R r).
In terms of densitics, this can seen via:
p(b D,.) p(r Dn) (r) dr,
18

---

and noting by definition of observational equivalence ~ that p(eb r, D.a)) = p(()) r), we have p(vb)
D,,) = p(r
p(r "D p(sp r) dr. This can be plugged into f ply" m ) x , 1)) p(ab Dob dd to
get:
/ (dot") x,, v) plu r) db p(r D:th) dr
ply" (lo(")) xtn, Daai)
3. Martingale convergence. By construction, we have that P(y Tob X
V = r)
P(Y Toh Xin V/ r'). whenever * r'. Furthermore, since do(Tim), X in R r) is measurable since it can be written terms of conditional expecatations, we can apply Doob's martingale convergence theorem (Miller 2018) which yields:
P(y E B
Ma (B)
gB (ro) dB (r(w))))
i.e., (B) - 1o (B) a.s. for every measurable set B. This implies pn - Poo weakly.
B.2 Comments on the consistency theorem Previously, we have shown that when observational data Do generated by an SCM who
Pyy i do(t""), x Di.)) - P(y" do(ti"), xt [voo)..
(16)
Note that P(y"n do(tim),x x [Nbl]) has the following density:
p(y" do(tt"), [Vol) = P(y"M | do(ttm), x p((b)) dv.
(17)
where r denotes the representative the observational equivalence class of voo. Further, p(b|r) [[bb[v/)) specifies how likely the specific SCM vi is given that the SCM is in the observational
=
equivalence class [wol]-
Note that when [No] { i.c. there exists exactly one SCM do that could have generated the observational data Drob meaning that the is identifiable from P(De)), we get that P( = 8 and thus recover the conditional international distribution under the unique voo for infinite observational data:
P(y" dd((t"), x"" (ro) - P(y dd v0))
(18)
C Details on the prior-fitting procedure
In this section, provide the details of the data- tata-gennrattig process in Algorithm I that represents our modeling assumptions. From the perspective of PFNs, this data- getta-genrratng process represents Do-PEN's "prior". Concretely, prior-fitting procedure involves the following key steps:
Sampling the SCM: First, for every iteration i = 1, an SCM vi is sampled. This is achieved by first sampling DAG via topological sorting of vertices (Manber 1989). For each node k in the graph, we uniformly at random sample the nonlinearity y to be one of the following functions: the quadratic function SCM to take the form of an additive noise model (ANM) fu ( <x) = ((EEpp())) w)21)+66. The
x + ReLU(z), and x + tanh(x). We define the mechanisms in the
weights of the SCM sampled using Kaiming initialization wi Uniform( VIPA(R)) v [AA()))
for I = 1, 2, [A)], where PA(k)] denotes the number of parents of node k.
Sampling observational data: Next, observational data is sampled according to the SCM dh More specifically, dataset Dob is filled with Mas data points, where the number of data points is drawn uniformly between Mmin = and Myaz = 2, 200. Each element Dob generated by first sampling noise vector ej p(e) which is passed through the SCM to generate each element
3 t. x
19

---

Sampling interventional data: To sample an element in the interventional dataset D;", with M = Mmaz MM data points, first, a noise vector E ~ p(e) is sampled again. Subsequently covariate-vector X is sampled from p(x[{),,<)) ER). This ensures that the vector x characterizes the subject l prior the intervention. After sampling the value for the treatment & we perform the intervention " and sample y from the intervened-upon SCM using the same noise EK as beford
Gradient descent For each iteration = 1,...... NN, an observational dataset De and an interven- tional dataset n are generated. These datasets are utilized to compute the negative log-likelihood under our model q6- This loss is calculated with respect to predicting the interventional outcome yk" based on the value of the intervention th the covariates x;" and the observational dataset Dob. Subsequently, a gradient step is taken the negative log-likelihood. In practice, we perform mini-batch stochastic gradient descent using the Adam optimizer (Kingma and Ba 2014).
D Experimental Details
D.1 Details the synthetic case studies
In this section we provide the details on all considered case studies from Section 4.1 The standard deviation Gezo of the exogenous noise is sampled from dezo Uniform(11. ([,,3) For the standard deviation the additive noise terms, we sample 3 ~ Beta(1, 5), and then set a. = 0.3 B.
The functions Sz, take the form fa(zu, e) 7 (iepp())) wiz)) +e. The weights of the SCM are sam- pled using a Kaiming initialization wi ~ Uniform( v [PX(k]]
VIPACE]) for ( = 1, 2, ... [A)),), where PA(k)] denotes the number of parents of node k. The nonlinearities fa are sampled uniformly at random from the set {f1, f2, fs} where fi(z) = x², f2(2) = tanh(r) and fs = (e) =
max(0, z). Details on the case studies can be found in Table 1
'Because the noise is held constant to produce the pre-interventional covariate- x and interventional outcomes, y this process can also be seen simulating counterfactuals or single potential outcomes.
20

---

Setting
Equations
4, N((,
T1 Ne0,)
Observed Confounder
t f((z),))
fa(21,t,Eg)
Ex . cy NN(
t Uniform( (0, 1})
Observed Mediator
= Jay (mr)
y = (x
6z Er ~ o.)
12 (e0,
Confounder + Mediator
t = fe(
Iı Son (1, Eari)
y = (21, D22 t, Cy)
4. c N(0,oz)
T1 N(00 Taro)
Unobserved Confounder
t = fi(21,22,6) (2, Et)
D1 fa, (t, Cmi)
I2, 221, t, Ey)
Ex E2, Ey NOO, 0e)
I2 NN0,, Tezo)
Back-Door Criterion
I = f((21
I = fa (x2)
fz111, (ny
C, C2 (N0,
82 (ezo) t ())
Front-Door Criterion
81 = Say (12,
V 101, Ey)
Table 1: Structural equations for all causal case studies.
21

---

D.2 Evaluation metric
We evaluate our results in terms of normalized mean squared error (MSE), as it allows results to be compared across datasets. We define MSE below:
1
yi y1
MSE(y.9)
(19)
π
myx min(y)
D.3
Description of baselines
D.3.1
Conditional interventional distribution prediction
Do-PFN (v1) (ours): TabPEN regression model pre-trained for 48 hours varying graph structures of up to 10 nodes to approximate the CID p(y
|x DDb)
Dont-PFN: TabPFN regression model (Hollmann et al. 2025
pre-trained on our prior to D
approximate the posterior predictive distribution (PPD) p
Do-PEN-Short: TabPFN regression model pre-trained for 20 hours on varying graph structures of up to 5 nodes to approximate the CID "0 Dob)
Draph: TabPFN regression model pre-trained for 5 hours to approximate the CID p x Dob) on fixed graph structures from our case studies.
Do-PEN-Mixed: Do-PFN-Shar Short pre-trained for 24 hours varying whether additive noise terms are sampled from zero-mean Gaussian, Laplacian, Students- T, and Gumbel distribu- tions.
DoWhy (Int//ntff): a SCM fit observational samples Dob and the graph structure gg. The constructed SCM is used to predict interventional (Int.) and counterfactual (Cntf.) outcomes. Crucially, TabPFNClassifie and Teggeeorrr models (Hollmann al.) 2025) are used approximate binary and continuous structural equations.
TabPEN (v2): A TabPFNRegressor model proposed in (Hollmann et al.
2025 pretrained
to approximate the posterior predictive distribution (PPD) ply x De0
Random Forest: ensemble of decision trees Breiman 2001 trained D
D.3.2
Conditional average treatment effect estimation
Do-PFN (v1) (ours): Do-PEN (v1) applied to predict the specific quantity
== Eyinn (of [do(t))=)))( (eo)) [y E tn (y* =0) (Do [v]
(20)
where the model directly approximates the conditional interventional distribution.
Do-PEN (v1.1) (ours): version of Do-PFN pre-trained for 96 hours varying graph structures of up to 60 nodes applied to predict the CATE as described above.
Dont-PFN: Dont-PFR applied to predict the CATE,
S-Learner (TabPFN): meta-learning approach (Künzel al. 2019) that trains single TabPEN model jointly on treatment and covariates to predict outcomes, using the treatment additional input feature.
T-Learner (TabPFN): a two-model meta-learning approach (Künzel al. 2019) that fits separate TabPFN predictors for treated and control populations, estimating CATEs from their prediction difference.
X-Learner (TabPFN): an extension of the meta-learning framework (Künzel et al. 2019) where TabPFN serves the base model in cross-fitting procedure to reduce bias and improve finite-sample efficiency.
DML (TabPFN): a double machine learning (DML) baseline using TabPFN as the base learner, following (Nie and Wager 202T), which leverages orthogonalized score functions and flexible meta-leamed regressors estimate treatment effects.
22

---

S-Learner (HPO): htyprparameeter-optiiiiee optimized meta-learning baseline implemented using the EconML library (Battocchi et al. 2019). where single model jointly learns predict outcomes from covariates and treatment. HPO is performed using FLAMI. where we tune the the base-learner for 60 seconds (Wang et 202T).
two-model baseline implemented using EconML, fitting separate outcome models for treated and control units. HPO is performed using FLAML where we tune the each base-learner for 60 seconds (Wang et al. 202T
T-Learner (HPO): a tyyeeparaaeete-optimniis
X-Learner (HPO): ayperparrmeeee-optiiizeed variant of the X-Learner (Künzel et al. 2019), implemented with EconML, which combines treatment and control outcome models through cross-fitting to reduce estimation bias. HPO is performed using FLAML where we unne the the base-learners and the propensity model each for 60 seconds (Wang et al. (021).
TARNet:
the treatment aggnosiic representation network (Shalit et al. 2017), which learns shared covariate representations and separate outcome heads for treated and control groups; implemented using the CATENets library (Curth et al. 202T).
DragonNet: a neural network based CATE estimator (Shi al. 2019 that augments TARNet with an explicit propensity head and targeted regularization to improve covariate balance; also implemented using CATENets.
Causal Forest (DML): a double machine learning (DML) approach based on (Wager and Athey 2018) that combines multiple causal trees estimate conditional average treatment effects (CATEs), Hyperparameters are tuned using exhaustive search.
DoWhy-CATE (Int/Cntf.): DoWhy (Int/Cnttf) used as S-Learner (Künzel al. 2019) to estimate conditional average treatment effects (CATEs). When DoWhy (Cntf.) is used, noise terms inferred and held constant across forward passes.
D.3.3 Software
We use Pytorch (Paszke 2019) to implement our experiments. Our implementation of the causal prior is based on the Causal Playground library (Sauter et al. (2024) and the codebase used for TabPFN (Hollmann et al. 2023 2025). We use Matplotlib (Hunter 2007), Autorank (Herbold (2020)
and Seaborn (Waskom 2021) for our plots.
D.4
Hybrid synthetic-real--orrd data
D.4.1 RealCause
The RealCause benchmark (Neal al. 2020) provides realistic causal inference datasets by fitting generative models to real-world data, ensuring that the simulated data is similar to the real-world data. This approach enables access ground-truth causal effects while preserving characteristics of the real-world data. Crucially, all observational datasets are simulated in way that ensures that the unconfoundedness assumptions strictly hold, regardless of whether the actual real-world data supports this.
D.4.2 Known graph
We furthermore conduct experiments on two real-world datasets with agreed upon causal graphs depicted in Figure 7 Those causal graphs allow us to simulate gold-standard outcomes using the DoWhy library (Sharma and Kiciman 2020), which facilitates the evaluation of Do-PFN and our baselines. More concretely, we specify the agreed upon causal graph and use DoWhy to estimate the causal mechanisms from the real-world datasets. Subsequently, we simulate interventional data using these mechanisms and use the real-world observational data as input to all models.
Amazon Sales The Amazon sales dataset (Blöbaum et al. 2024) contains data on the effect of
special shopping events ("Shopping Event?") on the profit made from smartphone sales ("Profit"), further provides variables with information on the spending on ad campaigns ("Ad Spend"), the price of the device ("Unit Price"), the number of phones sold ("Sold Units"), the number of page view ("Page Views"), the revenue that day ("Revenue") and the operational cost ("Operational Cost").
23

---

Amazon Sales
Shupping
Unit Price
Page Views
Law School Admissions
Sold tiett
UGPA
Operational
Revenur
Pro
ESAT
Figure 7: Real-world case studies: Widely agreed-upon causal graphs for the Amazon Sales and Law School Admissions datasets.
Law School Admissions The law school admissions dataset (Figure was drawn from the 1998 LSAC National Longitudinal Bar Passage Study (Wightman) 1998) and was made popular in the realm of counterfactual fairness due to appearance in Kusner al. (2017) where the variable "Race" was treated as a protected attribute. We note that we do not address the topic of algorithmic fairness, but would like to highlight that the ability of Do-PEN to predict interventional outcomes on demographic information could be a fruitful application in model-bias assessment. We delve into the effect 32 of "Race" [on firs-year--veen ("FYA"), which is mediated by two variables undergraduate grade-piin--average ("UGPA") and "LSAT", a law school entrance exam in the United-States.
E
Supplementary results
E.1 Inference speed
While TabPFN's synthetic pre-traue pre-training paradigm offers benefits in terms of predictive performance and generalization, it also offers benefits in terms of training (.fit() and inference (.predict ()), as the cost of pre-training incurred once up front such that training examples can be passed into the transformer in-context and predictions can be obtained in single forward pass. We demonstrate on the RealCause benchmark, collection of up to 100 realizations of each of four datasets that range from 10-50 features and up to 10,000 samples, that Do-PFN achieves Pareto-Opting prediction speed and performance across all four dataset groups.
ACIC 2016 conieee
Lalenda
Lalonde PSID
PCN
PON (c1.1)
Doan PEN
Lounnet TARNN
4. 201-
2 23
XIEE (CATT)
AISE (CATE))
MSE (CATTE)
XSE (CATE)
HPO
Figure 8: Training and inference speed: Multi-objective scatter plot and Pareto Fronts depicting average inference speed and normalized mean squared error (MSE) of Do-PEN-CATE and our causal baselines conditional average treatment effect (CATE) estimation on the RealCause benchmark. Do-PFN is on the Pareto Front on all RealCause datasets.
'We note that Race in the lawschool dataset is typically treated binary variable. We very much disagree with this formulation, and acknowledge that the term "ethnicity" better describes this complex social construct.

---

E.2 In-distribution analysis
In order to better understand Do-PEN's behavior on different parts of its' prior, we conduct several ablation studies varying the base rate of the average treatment effect, dataset size, as well as graph size and complexity.
Base rate average treatment effect First, we evaluate Do-PEN's performance on datasets with various base rates of the average treatment effect (ATE). We observe in Figure 9 (left) that while Do-PEN and TabPFN (v2) perform similarly on datasets with small ATE, TabPFN's performance diverges as the base ATE grows. This divergence can be explained by either Do-PEN's pre-training objective or its ability to handle the implicit distribution shift between the context and query set created when predicting interventional outcomes. We do note, however, that Do-PFN's performance also seems diverge in the largest ATE group. This result is likely explained by the high probability of low-ATE datasets in our prior, caused by the combination of additive noise and non-linearities with squishing effect. This result highlights the need for future research in synthetic causal data cutt-generation that creates strong treatment effects that persist across long causal chains.
We also ablate upon Do-PEN's performance on datasets with between 5 and 2000 samples, observing in Figure 9 and converges in terms of variacee as dataset size increases. This result is consistent with the generally strong performance of TabPFN on small-data Hollmann et al. (2023), which stems from TabPFN's pre-training objective: obtain maximally accurate predictions from synthetic datasets with an varied number of samples.
Dataset size
(center left) that Do-PEN's MSE in predicting CATE values decreases
Do-PEN (vl)
Do-PEN (vl)
(v1)
Do<PPN (v1)
TabPEN (v2)
DoWby
DoWhy (Catf.)
DoWhy (Curf.)
MSE (CATE))
MSE (CATEE)
MSE (CATE)
MSE (CATE))
0.2
11.4
0.05
0.05
0.00
6.00
0.0
Q2
Q3
Q5
Q3
Q2
Q3
Q5
5.0
7.5
10.0
True XTE Quantile
Darasct Size Quantile
Graph Size (Bdiges
Graph Size (Nodee)
Figure 9: In-distribution analysis: Ablating the performance of Do-PFN across base rates of the average treatment effect (ATE), dataset sizes, and graph complexities. Do-PFN is robust to datasets with varied base rate ATE (left), improves in performance and consistency as dataset size grows (left center), and performs competitively with baselines that have access to the ground-truth graph structure as graph size and complexity grows (right).
Graph size and complexity We finally evaluate Do-PFN's performance across data generated from graphs of increasing complexity, sampling 500 datasets generated with graph structures consisting of 4 to 10 nodes and 2 to 43 directed edges. This result is visualized in Figure (right), which shows that Do-PFN performs competitively to DoWhy (Cntf.), a baseline that has access to the ground- truth DAG across graphs of varying size and complexity. We note that while our synthetic data- generating mechanisms are relatively simple from mathematical perspective, graph identification is combinatorially hard problem, with the number of unique Directed Acyclic Graphs (DAGs) of 10 nodes reaching 4.17 10.8.
Standard deviation of additive noise We also highlight in Figure (left) that the performance of Do-PFN decreases with an increase in the standard deviation of additive noise, which corresponds also observe in Figure 10 (center-right) that Do-PEN's
to larger irreducible error. However, performance for different levels of additive noise seems to increase with dataset size. This means that the NMSE for datasets with a certain amount of additive noise can be reduced up to certain extent with more data.
Causal criteria To investigate the robustness of Do-PEN and our CATE-estimation baselines with respect to the violation of the unconfoundedness assumption, compare the performances
25

---

Do-PPN (v1)
Do (onhf(Cutf.
DoWhy (Cutf.)
Q1 Q2 Q3 Q1 Qs Q6
Q2 Q3 Q5 Q6
MSE (CID)
-5.0
Datase Sizee
MSE (CATE)
Datoset Size
5 MSE (CATE)
-7.5
1
-10.0
0.0
0.1
0.2
Q1 Q2 Q3 Q24 Q5 Q6 Addicive Noise Sud. 5,
Q1 Q2 Q3 Q4 Q5 Q6 Additive Noise Std. o,
Additive Noise Sud. (o.)
Figure 10: In-distribution analysis (additive noise): Evaluation of Do-PFN's performance across different quantiles (QI-Q5) additive noise standard deviation. The density plot (left) shows that Do-PEN's performance decreases with (irreducible) additive noise. However, the heatmap (center) shows that for datasets with similar additive noise levels, Do-PFN's performance increases with dataset size. This effect is even stronger than for DoWhy (Cntf.).
synthetic case-studies that satisfy unconfoundedness (the "Observed Confounder" and "Backdoor- Criterion" case-studies) and that violate this assumption ("Observed Mediator", "Confounder Mediator", "Unobserved Confounder", "Front-door Criterion"). The results in Figure show that, while the performance of almost all methods degrades without unconfoundedness, Do-PFN maintains the strongest performance in both scenarios.
Unconfoundeedesss
No Unconfoundedness
Do-PFN (v1)
Causal Forest DML
Dont-PFN
$ Learner (TabPFN)
Learner
(TabPFN)
T-Laareer
TabPPNN)
DML (TabPPN)
TARNet
DragonNet
100
0
50
100
MSE (CATE)
MSE (CATE)
Figure 11: Robustness to unconfoundedness: Bar plots with 95% confidence intervals depicting the median MSE in CATE estimation when unconfoundedness (the treatment assignment is independent of the potential outcomes given the covariates) is fulfilled violated. We find that while the performance of all methods degrades without unconfoumdedness, Do-PEN maintains the strongest performance in both settings.
26

---

Obeerved Confornder
Observed Mediator
Confounder Mediator
Unobsrrved Confomder
Front- Door Criterion
Back Door
Critertion
DoWle (Cmd))
+
DoWhy (lar.)
Do-PPX-Croph
0.0 0.0000
0 0,055
0.000
0. 0.025
0000
002
0.00
0.05
0 0000
0 0.0025
0.000
0,002
MSR (CID)
MSE (CID)
MSE (CIL)
MSE (CID)
MSE (CID)
MSE (CID)
CD
Do-PEN-Gran DoWhy (Int.)
DoWlyy (Cul.. Do-PEX
Figure 12: Gold-standard comparison (CID): Box-plots and critical difference (CD) diagrams depicting distributions of normalized mean squared error (MSE) Do-PFN and our "gold-standard" baselines in conditional interventional distribution (CID) estimation on our six synthetic case studies. Do-PFN significantly outperforms Do-PEN- Graph and DoWhy (Int.).
E.3 Comparison to conceptual baselines
In this section, we compare Do-PEN to various conceptual baselines (i.e. baselines unlikely to be applied in the real-world) our synthetic case studies.
Gold standard baselines First, we compare Do-PEN to a set of gold standard baselines which have access to the ground-truth truth DAG in order to further explore Do-PFN's ability to perform causal inference in settings where graph knowledge required in order to correctly estimate causal effects, despite receiving information regarding the causal structure. In Figure 12 we visualize bar-plots and critical difference (CD) diagrams of normalized-mean waaplizzddmmaa--suur--------e results show that Do-PEN performs competitively with baselines Do-PFN-Grran and evaluated on fixed graph structures, and Do (Int.), in which a SCM is to the observational data and interventions are performed (without inferring noise terms, distinguising from Do-Why (Cntf.)). In Figure 13 we observe a similar trend, except that in CATE estimation, Do-PEN rather performs competitively with Do-Why (Cntf.). We explain this result in bias-decomposition (Figure 14 of the predictions of Do-PFN and Do- Wh-yhy (Cntf.), which shows that although Do-PFN slightly overpredicts CIDs, its bias is consistent the prediction of both potential outcomes and thus cancels out when predicting the CATE.
in CID prediction. The which is pre-trained
Do-PFN variants We also compare our proposed model with other variants, including Do-PFN- Short, Do-PFN-Graph, and Do-PFN-Minn Mixed. In Figure 15] we observe that Do-PFN outperforms Do-PFN-Grun Graph and Do-PFN-Short, which suggests that the number of pre-training datasets seen is a key factor in performance. We also observe that Do-PEN-Mixel despite being pre-trained for only 24 hours, performs on par with our proposed model, suggesting that more complex noise distributions can aid in model convergance. This result also mirrors our out-of-distribution analysis, which shows that our model, which is pre-trained using only Gaussian additive noise, can generalize other noise distributions.
E.4 Common effect case study
We also provide an additional case-study, "Common-Effese which perfectly represents a randomized controlled trial (RCT). We show in Figure 16 that only in this case, do traditional tabular predictors predict the CID with comparable performance. As discussed in Section 4.2, we hypothesize that this is largely due to the lack of distribution shift between the context and query set caused by the lack of a directed edge from the treatment to the covariate.
27

---

Ohserved Confounder
Observed Mediator
Confounder Mediator
Unobserrved Confounder
Pront-Door Czicerion
Back- Door Criterion
(v-P
DoWlyy (Cal)
DoWiyy (lu.)
0.000
0.025
0.00
0.02
0,000
0.020 MSB (CATE)
0.0
0.1 MSE (CATE)
0.0
0.2
MIS (CATE)
MIE (CATE)
MSB (CATE)
MSE (CATE)
CD
(lot) (o.PFN(v1)
DoWhy: (Cnsf)
Figure 13: Gold-standard comparison (CATE): Box-plots and critical difference (CD) diagrams depicting distributions of normalized mean squared error (MSE) of variants of Do-PEN and our baselines in conditional average treatment effect (CATE) estimation on our six synthetic case studies. Do-PEN-CATE outperforms DoWhy-CATE (Int.) and performs competitively with DoWhy-CATE
(Cntf.).
Obsrrood
Obsrrved Modicctor
Confemmer
Unoberved Confander
Front-Dow
Tack-
Confouder
Modlator
Chiberion
Chitzzion
Do-Per FeNe
De-PT
Do
Do-FNN
do23)
Dive 4007
0.00
cunz3
0.005
0.000
DeWlly (Call
600
(Chnf)
DeWCa
0.0
(Cual
8:00
DoWll (CatE
6.00
GU82
0.00
1056
ILIZ
JUCONI CUD25
0310
0J12
GU02
Biaa do(0)
Bio se())
Blu do00)
Biuo do(0)
Bap abo0)
Bia do10)
Figure 14: Bias decomposition (CID): The median of the bias of DoWhy (Cntf.) and Do-PFN across 100 synthetic datasets for the interventions do(0) and d1(.). The gold-standard DoWhy (Cntf.) maintains bias very close zero for all case-studies while Do-PFN has small positive bias that takes almost the same value for do(O) and do(1).
Obeerved Confounder
Obeerved Mediator
Confounder Mediator
Unchserved Confeander
Front-Doot Criterion
Back Door Criterion
PPN-Sett
PFN Graph
Do-PFN-Maod
MISE (CID)
MSE (CID)
MSE (CID)
MSE (CID)
MSE (CID)
MSE (CID)
CD
Do-PEN-Craph Do-PEN-Short
Do-PEN Do-PFN-Mixed
Figure 15: Comparison of Do-PEN variants (CID): Box-plots and critical difference (CD) diagrams depicting distributions normalized mean squared error (MSE) of Do-PEN variants in conditional interventional distribution (CID) estimation six synthetic studies. Do-PFN significantly outperforms other variants except Do-PFN-Mixed, which achieves statistically similar performance in half the pre-training time.
28

---

Cummon Effect
Commnon
Elfect
DoPPFN (v1)
TaPPFN (v2)
NO, i,))
Random Forest
X1 N(O, aaze) ((,
Doat:
Doar-PEN PFN
MBE (CID)
Figure 16: Common effect case study: Visualization of graph structure and structural equations (left) for our "Common-Effectt" case study, as well as box plots depicting distributions of normalized mean squared error (MSE) of Do-PFN compared regression baselines in conditional interventional distribution prediction. Regression baselines perform similarly to Do-PFN, as the intervention does not cause distribution shift between Do and Din
Obeerved Conforuderr
Observed Confounder
Unoleerved Confounder
Unobecrved Confounder
Observed Mediszor
Observed Mediator
Pront-Door Criterion
Front- Door Criterion
Back-Door Criterion
Back- Door Criterion
Observed Medistor Confounder
Obeerved Mediator Coufoundes
2.5
5.0
Entropy
Cross Entrogy (CF)
Figure 17: Uncertainty quantification: Cross-entropy (CE) loss (right) and entropy (left) of Do- PFN's bar distribution output. Do-PEN is highly uncertain on the "Unobserved Confounder" case study due unidentifiability. Do-PEN also shows high uncertainty case study, which we argue is due to its only exogenous term being binary variable, causing the continuous effect in the outcome only come from additive noise.)
the "Observed Mediator"
29

---

E,5
Uncertainty
Next, we explore Do-PEN's uncertainty calibration, by visualizing the prediction interval coverage A PICP curve equal to the 45-degree diagonal corresponds to a model consistently yielding prediction intervals with exactly the desired coverage. Being above the diagonal corresponds to under-confident and being below the diagonal to over-confident prediction intervals. First, we observe that Do-PEN's uncertainty is slightly under-confident for theoretically identifiable case studies (the full set of calibration plots can be found in Appendix E.5). In the "Unobserved Confounder" case study, the model's s high uncertainty reflected by a relatively large entropy in its output distribution However, our PICP results show that the model's uncertainty for this case study is correctly calibrated.
probability (PICP) in Figure 2
Observed
Confounder +
Backdoor
Mediator
Mediator
Criterion
1.0
1.0
Actual Coverage
Actund Coverage
Actund Coverage
0.5
0.5
Do<PEN
D&-FEN Optimal
Do-
Optimal
Optimal
0.0
0.0
0.0
0.0
0.5
Destred Coverage
Desired Coverage
Desized Coverage
Frontdoor
Observed
Unobserved
Criterion
Confounder
Confounder
1.0
1.0
1.0
Áctual Coverage
Actual Coverage
Actual Caverage
0.5
0.5
Do-PFN Optinal
Do-PFN Optimal
DFN
Optimal
0.0
0.0
0.0
0.0
Destrdd Coverage
Desired Coverage
Desived Coverage
Prediction interval coverage probability (PICP) plots for the "Observed Mediator", "Confounder Mediator", "Backdoor Criterion" and "Frontdoor Criterion" cases. The solid blue line shows the coverage and standard deviation achieved by Do-PFN, spanning desired probabilities from 0 to The dashed line represents the ideal calibration achievable with access to the ground-tre -tuth CID. Do-PFN is slightly under-confident for identifiable case studies, and, crucially, correctly unconfident for the "unobserved confounder" case.
Figure 18: Uncertainty calibration:
E.6 Critical Difference Plots
To compare model performances the autorank package (Herbold 2020), which performs a non- non-parametric Friedman test followed by pairwise post-hoc analysis using the Nemenyi test. The resulting critical difference (CD) plots visualize average ranks of models across datasets: models that are not significantly different at 0.05 are connected by horizontal bars. Lower ranks correspond better performance. This approach allows us to jointly assess both the magnitude and statistical significance of performance differences across all case studies.
multiple datasets in statistically rigorous way, employ
30

---

CD
Dout-PEN
Do-PEN TabPFN (v2)
Random Forest
Figure 19: Critical difference (CID): Critical difference (CD) diagram depicting distributions of normalized mean squared eπror (MSE) of Do-PFN and our regression baselines in conditional interventional distribution (CID) prediction. Across our six causal case studies, Do-PFN achieves statistically significant improeementsovvrrrgrrssss baselines, showing that our pre-training objective effective for predicting CIDs.
Dragon Not. TetNet [TaPPN]
Do-PFN (v1)
L.Lrre (TabPPN) Doal-PEX
T- T-Learzer DML
[ILPPN
Causel Forest
X-Learner
(TabPEN:
Figure 20: Critical difference (ATE): Critical difference (CD) diagram depicting distributions of normalized mean squared error (MSE) of Do-PEN and our regression baselines average treatment effect (ATE) estimation. Across our six causal case studies, Do-PEN significantly outperforms our meta-learner, double machine learning, and deep-learning based baselines.
TARNet Net [TaPPFN]
Do-PEN (v1) S-Learer (labPEN) DML TTbbF Causal Foret DML
T-Leamer
Deat-PFN
X-Learee (TabPEN)
Figure 21: Critical difference (CATE): Critical difference (CD) diagram depicting distributions of normalized mean squared error (NMSE) of Do-PEN-CATE and our causal baselines in conditional average treatment effect (CATE) estimation. Across our six causal case studies, Do-PFN significantly outperforms our meta-learner, double machine learning, and deep-learning based baselines.
CD
10
7
5
3
DragonNet TARNe
S-Learner (TbPPNN)
T- T-Learer (TabPFN)
Do-PEN (v1)
Do-PEN (v1.1)
Causol Forest DML DML (ТыВPEN)
X-Learner (TabPPN)
Dout-PEN
Figure Critical difference (RealCause): Critical difference (CD) diagram depicting distributions of normalized mean squared error (NMSE) of Do-PFN-CATE and our causal baselines in conditional average treatment effect (CATE) estimation on the RealCause benchmark. Across all four dataset groups, Do-PEN maintains competitive performance with our meta-learner, double machine learning. and deep-learning based baselines.
31

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: All claims made in the abstract and introduction are fully supported by our empirical and theoretical results.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the Limitations of the work performed by the authors?
Answer: [Yes]
Justification: Section Jexplicilly points out and extensively discusses four major limitations of our work.
Guidelines:
The answer NA means that the paper has ion while the answer No means that the paper has Limitations, but those are not discussed the paper.
The authors are encouraged to create a separate "limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-specification. asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
32

---

Answer: [Yes]
Justification: We provide full set of assumptions for Proposition sketch the proof main body of the manuseript and provide the full details of the proof in Appendix A We further critically discuss the assumptions and implications of this result.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proof's provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide the details to reproduce the experimental results of our paper in Section C and Section D.I of the Appendix. We also release the code for our prior, the pre-trained model, code to perform inference, and files containing all of our raw results.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
33

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: We release the code for our prior, the pre-trained model, code perform inference, and files containing all of our raw results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co
waw.aiiissiiioiicoooooooo for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
7 //nips. cc/ www.c/ppwwppppccccecceoeoooooeee http://wdiess/ee www.eeeuwisssoonnooooooooooooeee
for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We specify all details of our experiments necessary to understand the results in either the main body of the paper and in Appendix D
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
We communicate main results via boxplots, that also show the 25% quantile and 75% quantile and include a critical difference plot where we use post-hoc Friedman indicate statistical significance (Herbold (2020). In all line plots, we indicate 25% and 75% quantiles in our results via shaded areas.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
34

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (c.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We specify the type of GPU and the training time needed for all our models.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics http:////aaarpppccppcaaccccccaaa
Answer: [Yes]
Justification: The research conducted in this paper conforms, in every respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
35

---

Our paper is contribution to fundamental research in the field of machine learning for causal effect estimation. There are potentially various future consequences of our work, none of which we feel must be specifically highlighted here.
Justification:
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to partieular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out, For example, it is legitimate to point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our paper concerns the investigation of the principle of prior-liteddnettorks for the purpose of causal effect estimation. We therefore do not release models or data that have high risk for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes)
Justification: The creators original owners of all used assets are properly credited. We cite the authors more specialized softward packages in Appendix (D..33 Öther requirements can be found in the attached repository.
36

---

Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode. com camdsataeets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We document the code and model released with this paper. We note that our documentation is geared towards accessing and reproducing results and not towards end-users.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/cooe/mmodll as part of their submissions via structured templates. This includes details about training, license, Limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
37

---

Answer:
[NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs as any important, original, or non-standard components
Guidelines:
The answer NA means that the core method development in this research does not non
involve LLMs as any important, original, non- stanstandard components.
Please refer to our LLM policy for what should or should not be described.
38


Reviews:
[{"review_id": "lPMniKZdNQ", "reviewer": "Reviewer_Fczh", "summary": "Do-PFN is a PFN‐based transformer for zero‐shot causal inference via meta‐training on synthetic SCMs, with a theoretical guarantee of CID optimality under its prior. Empirically, it outperforms regression and causal‐forest baselines and rivals graph‐aware methods, showing robustness and calibrated uncertainty—though its real‐world success hinges on prior fidelity.", "questions": "1. How does Do-PFN perform on real or semi-synthetic causal datasets (e.g., healthcare or economics benchmarks), and what evidence can you provide that your synthetic prior’s function families, noise distributions, and DAG structures cover the diversity of real-world causal mechanisms?\n\n2. Can you quantify Do-PFN’s robustness when test data come from distributions not represented in your synthetic prior—i.e., under realistic distribution shift—and characterize how its “optimality” degrades compared to Proposition 1’s guarantees?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "### Weaknesses\n\n- **w1:** All experiments in the paper are based on carefully designed synthetic SCMs, but none are validated on any real causal datasets (e.g., semi-synthetic or pseudo-real data in healthcare or economics). A core question is whether the function families, noise distributions, and DAG structures in the synthetic prior can cover the extremely diverse causal mechanisms encountered in real-world scenarios.  \n- **w2:** When the test data **are not** generated from this synthetic prior but come from other real distributions, the “optimality” guaranteed by Proposition 1 no longer holds—closely related to w1.  \n- **w3:** There is no systematic comparison with widely used causal-inference tools (e.g., T-learner, IPW, CFRNet, TARNet) on the same synthetic tasks.  \n- **w4:** The SCM is configured with a relatively small number of random nodes. It remains unclear how the method performs on high-dimensional datasets.  \n\n### Strengths\n\n- **s1:** Proposes an end-to-end in-context causal inference approach that requires no pre-specified causal graph and does not rely on untestable no-confounding assumptions. It can automatically learn front-door and back-door adjustments, greatly simplifying the traditional causal-inference pipeline.  \n- **s2:** The same Do-PFN architecture can handle multiple tasks—such as CID (distribution prediction) and CATE point estimation—thereby reducing the overall complexity of the causal-inference pipeline.", "was_revised": true, "final_justification": "I have raised my score to 4 based on the real-world experimental results provided by the authors.", "rebuttal": "# General Response\n----\n\nAll reviewers acknowledge that our paper presents a promising approach, but also raise the question of prior mismatch, especially when analyzing real-world data. While we believe it is crucial for a new method to study synthetic data with known ground truth, we agree that analyzing real world data is important. Therefore, we have meanwhile performed an analysis of Do-PFN's performance on **six new datasets based on real-world data**, with promising results that we detail next: \n\n**Known Graph**: First, we evaluate on the Amazon Sales and Law School Admissions datasets where the causal graph is known (please refer to the supplementary material attached to the original submission as a .zip file for details). We also evaluate against six new baselines. \n\n## Median MSE (CATE) - Known Graph\n|Method|Law School|Amazon Sales|\n|:------------------------|:------------------|:------------------|\n|Causal Forest DML|0.157 ± 0.056|0.136 ± 0.279|\n|Causal Forest DML (HPO)|0.519 ± 0.080|0.579 ± 0.316|\n|DML (TabPFN)|0.283 ± 0.067|**0.037 ± 0.169**|\n|Do-PFN (v1)|**0.029 ± 0.010**|**0.038 ± 0.023**|\n|Do-PFN (v1.1)|**0.067 ± 0.043**|0.209 ± 0.135|\n|Dont-PFN|**0.103 ± 0.046**|0.313 ± 0.158|\n|DragonNet|0.371 ± 0.035|0.579 ± 0.316|\n|S-Learner (HPO)|0.123 ± 0.045|0.233 ± 0.246|\n|S-Learner (TabPFN)|0.161 ± 0.051|0.568 ± 0.303|\n|TARNet|0.372 ± 0.119|0.579 ± 0.316|\n|T-Learner (HPO)|0.151 ± 0.095|**0.017 ± 0.072**|\n|T-Learner (TabPFN)|0.161 ± 0.048|0.579 ± 0.316|\n|X-Learner (HPO)|0.519 ± 0.080|0.579 ± 0.316|\n|X-Learner (TabPFN)|0.134 ± 0.047|0.579 ± 0.316|\n\nWe find that Do-PFN (v1) achieves the strongest performance on Law School and is not significantly worse than the best method on Amazon Sales, a dataset with a complex mediation structure.\n\n**RealCause:** Furthermore, we have pre-trained a version of Do-PFN (which we call v1.1) on larger graphs with up to 60 nodes, in order to evaluate on datasets from the RealCause benchmark [5].\n\n##  Median MSE (CATE) - RealCause\n|Method|LalondeCPS|LalondePSID|IHDP|ACIC2016|\n|:-------------------|:------------------|:------------------|:------------------|:------------------|\n|Causal Forest DML|0.114 ± 0.014|0.016 ± 0.002|0.024 ± 0.001|0.005 ± 0.001|\n|DML (TabPFN)|0.067 ± 0.009|0.013 ± 0.001|0.025 ± 0.002|0.008 ± 0.004|\n|Do-PFN (v1)|0.067 ± 0.006|0.022 ± 0.003|0.031 ± 0.002|0.033 ± 0.004|\n|Do-PFN (v1.1)|**0.039 ± 0.004**|**0.008 ± 0.001**|0.021 ± 0.001|0.019 ± 0.011|\n|Dont-PFN|0.049 ± 0.004|0.011 ± 0.001|0.032 ± 0.001|0.036 ± 0.003|\n|DragonNet|0.064 ± 0.006|0.020 ± 0.003|0.543 ± 0.154|0.097 ± 0.035|\n|S-Learner (HPO)|0.068 ± 0.010|0.015 ± 0.001|0.022 ± 0.002|0.003 ± 0.001|\n|S-Learner (TabPFN)|0.053 ± 0.007|0.013 ± 0.003|**0.019 ± 0.001**|**0.002 ± 0.001**|\n|T-Learner (HPO)|**0.044 ± 0.003**|**0.007 ± 0.001**|0.023 ± 0.002|0.002 ± 0.001|\n|T-Learner (TabPFN)|**0.042 ± 0.002**|**0.007 ± 0.001**|**0.020 ± 0.001**|**0.002 ± 0.001**|\n|TARNet|0.064 ± 0.006|0.020 ± 0.003|0.517 ± 0.147|0.092 ± 0.032|\n|X-Learner (HPO)|0.069 ± 0.008|0.020 ± 0.003|0.155 ± 0.033|0.048 ± 0.016|\n|X-Learner (TabPFN)|0.083 ± 0.010|0.015 ± 0.005|**0.019 ± 0.001**|**0.001 ± 0.000**|\n\nWe observe that Do-PFN-v1.1’s extended pre-training leads to strong performance among competitive baselines on three out of four RealCause datasets. We note that these datasets perfectly satisfy unconfoundedness, giving our baselines a fundamental advantage against Do-PFN. \n\nThese results show Do-PFN to nevertheless be competitive with a strong set of CATE baselines, suggesting the strong transfer of Do-PFN's synthetic pre-training to complex, real-world data.\n\n# Response to Reviewer Fczh\n----\nThank you for your constructive feedback!\n\n> There is no systematic comparison with widely used causal-inference tools (e.g., T-learner, IPW, CFRNet, TARNet) on the same synthetic tasks.\n\nIn response, we have evaluated Do-PFN on our synthetic case studies against a strong set of meta-learning and double machine learning (DML) baselines with TabPFN (v2) as a base learner, following [1], who use only meta-learners and [2], who show TabPFN to be a strong base learner for meta-learning CATE estimators. We also include TARNet [3] and DragonNet [4] implemented in the CATENets library.\n\n## Median MSE (CATE) - Synthetic\n|Method|Back-Door Criterion|Confounder + Mediator|Front-Door Criterion|Observed Confounder|Observed Mediator|Unobserved Confounder|\n|:-------------------|:-----------------------|:-------------------------|:------------------------|:-----------------------|:---------------------|:-------------------------|\n|Causal Forest DML|0.820 ± 0.329|1.508 ± 0.873|0.851 ± 1.035|0.278 ± 0.059|200.944 ± 109.848|4.325 ± 0.754|\n|DML (TabPFN)|0.667 ± 0.148|0.717 ± 0.277|1.093 ± 0.538|0.435 ± 0.086|258.944 ± 210.589|2.161 ± 0.821|\n|Do-PFN (v1)|**0.159 ± 0.108**|**0.223 ± 0.154**|**0.186 ± 0.104**|**0.017 ± 0.014**|10.900 ± 9.947|**0.291 ± 0.074**|\n|Dont-PFN|3.529 ± 1.817|1.923 ± 0.750|2.884 ± 7.457|0.475 ± 0.247|313.138 ± 383.839|5.116 ± 0.887|\n|DragonNet|35.127 ± 44.199|18.362 ± 10.619|50.792 ± 47.052|6.123 ± 1.598|3817.618 ± 2773.768|7.526 ± 2.265|\n|S-Learner (TabPFN)|1.343 ± 0.765|0.542 ± 0.188|1.205 ± 0.963|0.329 ± 0.131|14.988 ± 11.032|3.760 ± 0.896|\n|T-Learner (TabPFN)|26.049 ± 10.711|6.592 ± 3.319|55.158 ± 59.264|3.034 ± 0.840|**3.434 ± 2.537**|4.430 ± 0.500|\n|TARNet|35.756 ± 26.782|19.177 ± 8.814|52.404 ± 48.647|7.172 ± 2.349|4229.619 ± 3540.554|8.003 ± 2.861|\n|X-Learner (TabPFN)|6.280 ± 3.720|3.831 ± 2.562|9.344 ± 6.165|0.921 ± 0.465|5.175 ± 4.380|4.141 ± 0.602|\n\n\nAcross our six synthetic case studies, we find that Do-PFN achieves the best performance on all cases except for “Observed Mediator” in which the two-model T- and X-learner strategies are best able to capture this highly binary treatment effect, due to the treatment being the root cause.\n\n> Can you quantify Do-PFN’s robustness when test data come from distributions not represented in your synthetic prior [..]?\n\nRobustness to prior mismatch is an important topic for PFNs in general, but no theoretical results of the form suggested by the reviewer exist (also not for the non-causal TabPFN - this is a very hard problem). However, in addition to the strong real-world performance we observe above, we also perform several ablation studies regarding distribution mismatch where we vary core characteristics of the SCM-generated data Do-PFN is evaluated on.\n\n## Median MSE - Out of Distribution\n\n|OOD Type|Do-PFN (v1)|% Change|\n|:--------------------|:--------------------|:-----------|\n|In Distribution|**0.0053 ± 0.0008**|0.0%|\n|ELU Non-Linearity|0.0033 ± 0.0004|-38.2%|\n|Post Non-Linearity|0.0057 ± 0.0006|7.0%|\n|Sin Non-Linearity|0.0066 ± 0.0007|24.6%|\n|Student-t Noise|0.0050 ± 0.0006|-4.6%|\n|Laplacian Noise|0.0060 ± 0.0008|13.3%|\n|Gumbel Noise|0.0059 ± 0.0007|10.9%|\n|High Gaussian Noise|0.0212 ± 0.0006|301.4%|\n\n\nWe find that Do-PFN’s performance is robust to different noise distributions, and using post-non-linearities (instead of adding the noise after applying a nonlinearity). Using different nonlinearities can even improve performance while increasing the variance of the Gaussian noise drastically hurts performance. \n\nWhile not a general form of Proposition 1, one can make the following argument regarding the mismatch between training and testing data. For the posterior $p(y|x, do(t), D)$ denote $x = (x,t, D)$, such that under abuse of notation $p(y|x, do(t), D) = p(y|x)$. Denote Do-PFN’s error as a function of $p$ as $E_{p(x)} D_{KL}(p(y|x)||q(y|x)) =: e(p)$. Assume that $e \\in F$ for some function class $F$. Now let the testing data come from a different distribution $p’$ and write $e’(p) = E_{p’(x)} D_{KL}(p(y|x)||q(y|x))$. \nBy definition of the integral probability metric $d_F$ wrt. $F$, we get: \n$|e(p) - e’(p)| \\leq d_F(p, p’) $. In case $F$ is the space of 1-Lipschitz functions, $d_F(p, p’) = W_1$ by the dual representation of the 1-Wasserstein distance $W_1$. \n\nIntuitively, when the loss of Do-PFN is well behaved and the real-world data distribution $p’$ has a difference of $C$ to the synthetic training distribution $p$, the same constant $C$ bounds how much the loss increases when taking inputs from $p’$ instead of $p$. \n\nHaving said that, while the (surprisingly) strong generalization of PFNs from synthetic training data to real-world evaluation datasets has been extensively empirically confirmed, the theory behind this phenomenon is still developing [6]. \n\n> It remains unclear how the method performs on high-dimensional datasets.\n\nWe refer to a new experiment, in which we sample 500 synthetic datasets from our prior with graphs of up to 50 nodes.\n\n### Median MSE (CID) - OOD Graph Size\n|Number of Nodes|Do-PFN (v1)|Do-PFN (v1.1)|\n|:------------------|:----------------|:---------------------|\n|3-10|**0.0044 ± 0.0009**|0.0045 ± 0.0029|\n|11-20|**0.0083 ± 0.0012**|0.0091 ± 0.0026|\n|21-30|0.0107 ± 0.0012|**0.0083 ± 0.0011**|\n|31-40|0.0119 ± 0.0017|**0.0076 ± 0.0013**|\n|41-50|0.0140 ± 0.0020|**0.0051 ± 0.0018**|\n\nFor larger graphs out of its prior-distribution, we observe a significant reduction in Do-PFN-v1’s performance, which decreases as the graph size goes further out of distribution. Do-PFN (v1.1), however, which was trained on graph sizes up to 60 nodes, achieves significantly better performance on large graphs with 21-50 nodes. This provides evidence towards Do-PFN’s ability to scale up to more complex and high-dimensional problems. \n\n## References \n[1] Vanderschueren et al. \"AutoCATE: End-to-End, Automated Treatment Effect Estimation.\" ICML 2025. \n\n[2] Zhang et al. \"TabPFN: One Model to Rule Them All?.\" arXiv 2025.\n\n[3] Shalit et al. \"Estimating individual treatment effect: generalization bounds and algorithms.\" ICML 2017.\n\n[4] Shi et al. \"Adapting neural networks for the estimation of treatment effects.\" NeurIPS 2019. \n\n[5] Neal et al. \"Realcause: Realistic causal inference benchmarking.\" arXiv 2020.\n\n[6] Nagler. \"Statistical foundations of prior-data fitted networks.\" ICML 2023."}, {"review_id": "eUrPVgnrNb", "reviewer": "Reviewer_HtXo", "summary": "This paper extends Prior-data Fitted Networks (PFNs) to causal effect estimation on tabular data, introducing Do-PFNs. The key innovation is training PFNs on synthetic datasets spanning diverse causal structures. Do-PFNs achieve performance comparable to methods with access to ground-truth causal graphs while substantially outperforming baselines that lack this information.", "questions": "I might increase the scores based on authors response to my questions below:\n- Intuitions on why the model is underconfident in identifiable cases?\n- Can you run experiment estimating ATE and compare with the same baselines?\n- Possibility of including more variables (e.g. number of confounders in your experiment)?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "*Strengths*:\n- The theoretical analysis provides valuable insights into Do-PFN's empirical performance, offering principled understanding of why the approach works.\n- The experiments convincingly demonstrate that Do-PFN learns to estimate causal effects rather than merely fitting observational outcome distributions, as evidenced by comparisons with Dont-PFN.\n- The experimental evaluation is comprehensive, featuring: (1) diverse causal scenarios covering various structural assumptions, (2) comparison against strong baseline methods, and (3) thorough ablation studies that isolate key design choices.\n\n*Weaknesses*:\n- The experimental evaluation is limited to simulated datasets with at most 5 variables, which may not capture the complexity and dimensionality of real-world causal inference problems.\n- The exclusion of Average Treatment Effect (ATE) estimation from the main experimental results is unexplained, despite ATE being a fundamental quantity of interest in causal inference.\n- The computational efficiency appears questionable, deploying a 7.3 million parameter model for datasets with only 5 variables raises concerns about scalability and whether such model complexity is warranted for these problem sizes.\n- The paper lacks theoretical or empirical analysis explaining two key behavioral patterns: why the model exhibits underconfidence in identifiable cases and why it performs well specifically on small datasets.", "was_revised": true, "final_justification": "The authors have addressed my questions properly and the additional experiments showed evidence of strength of their method.", "rebuttal": "# General Response\n----\n\nAll reviewers acknowledge that our paper presents a promising approach, but also raise the question of prior mismatch, especially when analyzing real-world data. While we believe it is crucial for a new method to study synthetic data with known ground truth, we agree that analyzing real world data is important. Therefore, we have meanwhile performed an analysis of Do-PFN's performance on **six datasets based on real-world data**, with promising results that we detail next: \n\n**Known Graph:** First, we evaluate on the Amazon Sales and Law School Admissions datasets where the causal graph is known (please refer to the .zip file in the supplementary material attached to the original submission for details): \n\n## Median MSE (CATE) - Known Graph\n|Method|Law School|Amazon Sales|\n|:------------------------|:------------------|:------------------|\n|Causal Forest DML|0.157 ± 0.056|0.136 ± 0.279|\n|Causal Forest DML (HPO)|0.519 ± 0.080|0.579 ± 0.316|\n|DML (TabPFN)|0.283 ± 0.067|**0.037 ± 0.169**|\n|Do-PFN (v1)|**0.029 ± 0.010**|**0.038 ± 0.023**|\n|Do-PFN (v1.1)|**0.067 ± 0.043**|0.209 ± 0.135|\n|Dont-PFN|**0.103 ± 0.046**|0.313 ± 0.158|\n|DragonNet|0.371 ± 0.035|0.579 ± 0.316|\n|S-Learner (HPO)|0.123 ± 0.045|0.233 ± 0.246|\n|S-Learner (TabPFN)|0.161 ± 0.051|0.568 ± 0.303|\n|TARNet|0.372 ± 0.119|0.579 ± 0.316|\n|T-Learner (HPO)|0.151 ± 0.095|**0.017 ± 0.072**|\n|T-Learner (TabPFN)|0.161 ± 0.048|0.579 ± 0.316|\n|X-Learner (HPO)|0.519 ± 0.080|0.579 ± 0.316|\n|X-Learner (TabPFN)|0.134 ± 0.047|0.579 ± 0.316|\n\nWe find that Do-PFN (v1) achieves the strongest performance on Law School and is not significantly worse than the best method on Amazon Sales, a dataset with a complex mediation structure.\n\n**RealCause:** We also evaluate on four new datasets from the RealCause benchmark [5], using a version of Do-PFN (which we call v1.1) pre-trained on larger graphs with up to 60 nodes. \n\nWe observe that Do-PFN-v1.1’s extended pre-training leads to strong performance among competitive baselines on three out of four RealCause datasets. We note that these datasets perfectly satisfy unconfoundedness, giving our baselines a fundamental advantage against Do-PFN. Due to space constraints we refer the Table “**Median MSE (CATE) - RealCause**” in our general response to reviewer **Fczh** for the precise MSE values.\n\nThese results show Do-PFN to nevertheless be competitive with a strong set of CATE baselines, suggesting the strong transfer of Do-PFN's synthetic pre-training to complex, real-world data. \n\n# Response to Reviewer HtXo\n----\n\nThank you very much for your feedback and comments!\n\n> Possibility of including more variables (e.g. number of confounders in your experiment)?\n\nWe are happy to provide more high-dimensional synthetic cases! We refer to a new experiment in our out-of-distribution (OOD) analysis, in which we sample 500 synthetic datasets from our prior with graphs of up to 50-nodes.\n\n## Median MSE (CID) - OOD Graph Size\n\n|Number of Nodes|Do-PFN (v1)|Do-PFN (v1.1)|\n|:------------------|:----------------|:---------------------|\n|3-10|**0.0044 ± 0.0009**|0.0045 ± 0.0029|\n|11-20|**0.0083 ± 0.0012**|0.0091 ± 0.0026|\n|21-30|0.0107 ± 0.0012|**0.0083 ± 0.0011**|\n|31-40|0.0119 ± 0.0017|**0.0076 ± 0.0013**|\n|41-50|0.0140 ± 0.0020|**0.0051 ± 0.0018**|\n\nFor larger graphs out of its prior-distribution, we observe a significant reduction in Do-PFN-v1’s performance, which decreases as the graph size goes further out of distribution. Do-PFN (v1.1), however, which was trained on graphs up to 60 nodes, achieves significantly better performance  on large graphs with 21-50 nodes. This provides evidence towards Do-PFN’s ability to scale up to more complex and high-dimensional problems. \n\n> The computational efficiency appears questionable, deploying a 7.3 million parameter model for datasets with only 5 variables raises concerns about scalability and whether such model complexity is warranted for these problem sizes.\n\nFirst, we show, via the performance of Do-PFN (v1.1), that using the same architecture (and number of parameters) scales up to achieve strong performance on more complex tasks, as shown in the **General Response** provided above. \n\n## Median Train/Optimize + Inference Time (s)  - RealCause\n\n|Method|LalondeCPS|LalondePSID|IHDP|ACIC2016|\n|:-----------------------|:------------------|:------------------|:------------------|:------------------|\n|Causal Forest DML|8.075 ± 0.090|8.014 ± 0.083|6.839 ± 0.051|19.070 ± 0.239|\n|DML (TabPFN) GPU|12.376 ± 0.035|12.441 ± 0.094|11.740 ± 0.029|22.807 ± 0.271|\n|Do-PFN (v1) CPU|2.699 ± 0.047|2.324 ± 0.131|**0.699 ± 0.009**|**2.734 ± 0.179**|\n|Do-PFN (v1) GPU|**1.029 ± 0.098**|**1.311 ± 0.162**|1.270 ± 0.143|**1.060 ± 0.160**|\n|Do-PFN (v1.1) CPU|3.010 ± 0.201|2.188 ± 0.013|**0.655 ± 0.004**|4.474 ± 2.042|\n|Do-PFN (v1.1) GPU|**1.011 ± 0.002**|**0.990 ± 0.004**|0.962 ± 0.001|**1.035 ± 0.036**|\n|Dont-PFN GPU|2.916 ± 0.169|2.437 ± 0.097|**0.769 ± 0.005**|3.258 ± 0.125|\n|DragonNet GPU|67.469 ± 14.605|124.642 ± 10.753|6.791 ± 0.013|6.378 ± 0.249|\n|S-Learner (HPO) |60.160 ± 0.072|60.373 ± 0.117|60.066 ± 0.012|60.498 ± 0.206|\n|S-Learner (TabPFN) GPU|**2.001 ± 0.004**|**1.927 ± 0.012**|1.791 ± 0.002|4.694 ± 0.105|\n|T-Learner (TabPFN) GPU|3.473 ± 0.009|3.573 ± 0.168|3.309 ± 0.005|7.378 ± 0.202|\n|TARNet GPU|58.258 ± 21.274|119.821 ± 14.448|6.804 ± 0.008|6.728 ± 0.293|\n|X-Learner (HPO) |120.240 ± 0.094|120.172 ± 0.102|120.060 ± 0.007|120.201 ± 0.050|\n|X-Learner (TabPFN) GPU|13.555 ± 0.239|13.469 ± 0.216|12.886 ± 0.037|31.158 ± 2.123|\n|Shape|2500,5|1340,5|375,7|2405,50|\n\nSecond, in our runtime analysis, Do-PFN (run on CPU and GPU) has a faster inference speed than all other baselines. Do-PFN outperforms other baselines because there is no conventional model-fitting, and is slightly faster than the TabPFN meta-learning approaches (when both methods use a GPU), since Do-PFN has less parameters and does not do ensembling. \n\n> [...] why does the model exhibit underconfidence in identifiable cases? \n\nThis is a very interesting question, which we think can be best explained by two factors: \n\nFirst, the training objective for Do-PFN minimizes a **forward** KL divergence $D_{KL}[P||Q]$ between the true posterior $P$ and Do-PFN’s approximation $Q$ (refer to Proposition 1 in our paper and [6]). This encourages $Q$ to be mass-covering when it cannot approximate $P$ perfectly, meaning $Q$ will be overdispersed in cases where there is a difference between $Q$ and $P$ [2,3]. \n\nSecond, in the identifiable case, the posterior $P$ will be more concentrated (no need to account for unidentifiability), which means that our TabPFN-like approximation of the posterior, based on a bar-distribution [4], inevitably incurs more discretization error than in the unidentifiable case. This causes a larger KL-divergence relative to the unidentifiable case and thus, by the first point, leads to a slightly more conservative estimated distribution $Q$.\n\nWe also note that underconfidence in case of approximation error is typically considered a very desirable property [3]. \n\n> [...] why it performs well specifically on small datasets\n\nWhile performance on small datasets is a quality inherent to basically all PFNs and has been extensively empirically confirmed [4,5], from a theoretical perspective, this hasn’t been extensively explored. We believe that the best explanation is that PFNs are trained across millions of samples of small datasets, to minimize the CE loss on the test split of the dataset, which directly penalizes overfitting.\n\n> Can you run experiment[s] estimating ATE and compare with the same baselines?\n\nWe have evaluated Do-PFN and an extended set of baselines (please see our first response to reviewer **Fczh** for their descriptions) on the task of ATE estimation.\n\n## Relative Error (ATE) - Synthetic\n\n|Method|Back-Door Criterion|Confounder + Mediator|Front-Door Criterion|Observed Confounder|Observed Mediator|Unobserved Confounder|\n|:-------------------|:-----------------------|:-------------------------|:------------------------|:-----------------------|:---------------------|:-------------------------|\n|Causal Forest DML|1.256 ± 0.308|0.872 ± 0.068|1.039 ± 0.204|0.550 ± 0.105|1.000 ± 0.000|2.332 ± 0.247|\n|DML (TabPFN)|1.143 ± 0.156|1.000 ± 0.000|1.067 ± 0.118|0.853 ± 0.064|1.000 ± 0.000|1.760 ± 0.222|\n|Do-PFN (v1)|**0.369 ± 0.086**|**0.178 ± 0.038**|**0.271 ± 0.080**|**0.064 ± 0.010**|0.099 ± 0.031|**0.623 ± 0.089**|\n|Dont-PFN|2.270 ± 0.689|0.786 ± 0.053|1.063 ± 0.403|0.334 ± 0.054|0.627 ± 0.108|2.755 ± 0.441|\n|Dragon Net|4.166 ± 1.177|1.142 ± 0.093|3.340 ± 0.697|1.029 ± 0.064|1.019 ± 0.091|1.371 ± 0.174|\n|S-Learner (TabPFN)|1.572 ± 0.390|0.485 ± 0.061|1.039 ± 0.265|0.378 ± 0.067|0.285 ± 0.047|2.128 ± 0.200|\n|T-Learner (TabPFN)|9.195 ± 1.756|1.756 ± 0.264|10.692 ± 3.516|1.256 ± 0.254|**0.061 ± 0.020**|2.332 ± 0.251|\n|TARNet|3.731 ± 1.341|1.117 ± 0.072|3.706 ± 0.988|1.104 ± 0.095|1.015 ± 0.100|1.224 ± 0.155|\n|X-Learner (TabPFN)|4.603 ± 2.072|0.956 ± 0.111|2.793 ± 0.938|0.776 ± 0.160|0.139 ± 0.033|2.386 ± 0.277|\n\nWe find Do-PFN to have the best performance in 5/6 case-studies, except for in the “Observed Mediator” case, in which the T and X-Learner baselines (with TabPFN v2 as a base model) is best suited to encapsulate the binary nature of this case study due to its two-model approach. \n\n## References\n\n[1] Neal et al. \"Realcause: Realistic Causal Inference Benchmarking.\" arXiv 2020.\n\n[2] Murphy. \"Probabilistic Machine Learning: Advanced topics. MIT press.\" 2023.\n\n[3] McNamara, et al. \"Sequential Monte Carlo for Inclusive KL Minimization in Amortized Variational Inference.\" AISTATS 2024.\n\n[4] Hollmann, Noah, et al. \"TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second.\" NeurIPS 2022. \n\n[5] McElfresh, et al. \"When Do Neural Nets Outperform Boosted Trees on Tabular Data?.\" NeurIPS 2023."}, {"review_id": "gioAdPp92g", "reviewer": "Reviewer_bczT", "summary": "This paper proposes Do-PFN, a transformer-based architecture trained via meta-learning to perform causal effect estimation from observational data. Building upon the prior-data fitted network (PFN) framework, the authors pre-train Do-PFN on a large number of synthetic datasets simulated from structural causal models (SCMs), including both observational and interventional data. The model is trained to predict interventional outcomes given only observational data and a target intervention. The authors evaluate Do-PFN across a suite of synthetic benchmarks reflecting canonical causal inference challenges (e.g., back-door, front-door, and unobserved confounding), demonstrating that it accurately predicts conditional interventional distributions (CIDs) and conditional average treatment effects (CATEs). Notably, Do-PFN achieves competitive or superior performance compared to baselines that rely on access to the ground-truth causal graph, despite requiring only observational data.", "questions": "Have you considered evaluating Do-PFN on real-world benchmarks for causal effect estimation (e.g., IHDP, Twins)? Even if ground truth is not perfect, such comparisons would help assess real-world transfer and robustness.\n\nHow sensitive is Do-PFN to mismatches between the synthetic prior used during training and the test-time SCMs?\n\nThe paper claims Do-PFN performs implicit adjustment akin to back-door or front-door criteria. Could the authors include examples or diagnostics to confirm that the model performs appropriate adjustment?\n\nDo-PFN is pre-trained for up to 48 hours on a single GPU. How does the training time and inference speed compare to existing causal inference methods, especially in settings with larger datasets or real-time requirements?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The paper's main strengths lie in its originality, empirical rigor, and the clarity of its contributions. It presents a compelling case for applying PFNs to causal inference, offering a novel approach to a long-standing challenge: estimating causal effects in the absence of a known causal graph. The proposed method is grounded in a well-defined meta-learning framework and makes use of amortized inference via synthetic SCMs. The authors offer both theoretical motivation and empirical validation. The paper is overall well-written\n\nHowever, the paper has a few limitations. While the use of synthetic data is well-justified in this context, it is unclear how well Do-PFN generalizes to real-world settings. The paper briefly acknowledges this issue, but does not provide empirical results or strong evidence of robustness under real-world distribution shifts or misspecified priors. The lack of real data experiments and the reliance on synthetic priors with assumed coverage over plausible real-world SCMs is the most substantial weakness. While the paper claims Do-PFN learns to “implicitly” adjust for confounding (e.g., back-door/front-door), it would benefit from deeper analysis.", "was_revised": true, "final_justification": "This is overall a good submission. The author(s) engaged during rebuttal, so I am happy to recommend this paper for acceptance.", "rebuttal": "# General Response\n----\n\nAll reviewers acknowledge that our paper presents a promising approach, but also raise the question of prior mismatch, especially when analyzing real-world data. While we believe it is crucial for a new method to study synthetic data with known ground truth, we agree that analyzing real world data is important. Therefore, we have meanwhile performed an analysis of Do-PFN's performance on **six datasets based on real-world data**, with promising results that we detail next: \n\n**Known Graph:** First, we evaluate on the Amazon Sales and Law School Admissions datasets where the causal graph is known (please refer to the .zip file in the supplementary material attached to the original submission for details): \n\n## Median MSE (CATE) - Known Graph\n|Method|Law School|Amazon Sales|\n|:------------------------|:------------------|:------------------|\n|Causal Forest DML|0.157 ± 0.056|0.136 ± 0.279|\n|Causal Forest DML (HPO)|0.519 ± 0.080|0.579 ± 0.316|\n|DML (TabPFN)|0.283 ± 0.067|**0.037 ± 0.169**|\n|Do-PFN (v1)|**0.029 ± 0.010**|**0.038 ± 0.023**|\n|Do-PFN (v1.1)|**0.067 ± 0.043**|0.209 ± 0.135|\n|Dont-PFN|**0.103 ± 0.046**|0.313 ± 0.158|\n|DragonNet|0.371 ± 0.035|0.579 ± 0.316|\n|S-Learner (HPO)|0.123 ± 0.045|0.233 ± 0.246|\n|S-Learner (TabPFN)|0.161 ± 0.051|0.568 ± 0.303|\n|TARNet|0.372 ± 0.119|0.579 ± 0.316|\n|T-Learner (HPO)|0.151 ± 0.095|**0.017 ± 0.072**|\n|T-Learner (TabPFN)|0.161 ± 0.048|0.579 ± 0.316|\n|X-Learner (HPO)|0.519 ± 0.080|0.579 ± 0.316|\n|X-Learner (TabPFN)|0.134 ± 0.047|0.579 ± 0.316|\n\nWe find that Do-PFN (v1) achieves the strongest performance on Law School and is not significantly worse than the best method on Amazon Sales, a dataset with a complex mediation structure.\n\n**RealCause:** We also evaluate on four new datasets from the RealCause benchmark [5], using a version of Do-PFN (which we call v1.1) pre-trained on larger graphs with up to 60 nodes:\n\n##  Median MSE (CATE) - RealCause\n|Method|LalondeCPS|LalondePSID|IHDP|ACIC2016|\n|:-------------------|:------------------|:------------------|:------------------|:------------------|\n|Causal Forest DML|0.114 ± 0.014|0.016 ± 0.002|0.024 ± 0.001|0.005 ± 0.001|\n|DML (TabPFN)|0.067 ± 0.009|0.013 ± 0.001|0.025 ± 0.002|0.008 ± 0.004|\n|Do-PFN (v1)|0.067 ± 0.006|0.022 ± 0.003|0.031 ± 0.002|0.033 ± 0.004|\n|Do-PFN (v1.1)|**0.039 ± 0.004**|**0.008 ± 0.001**|0.021 ± 0.001|0.019 ± 0.011|\n|Dont-PFN|0.049 ± 0.004|0.011 ± 0.001|0.032 ± 0.001|0.036 ± 0.003|\n|DragonNet|0.064 ± 0.006|0.020 ± 0.003|0.543 ± 0.154|0.097 ± 0.035|\n|S-Learner (HPO)|0.068 ± 0.010|0.015 ± 0.001|0.022 ± 0.002|0.003 ± 0.001|\n|S-Learner (TabPFN)|0.053 ± 0.007|0.013 ± 0.003|**0.019 ± 0.001**|**0.002 ± 0.001**|\n|T-Learner (HPO)|**0.044 ± 0.003**|**0.007 ± 0.001**|0.023 ± 0.002|0.002 ± 0.001|\n|T-Learner (TabPFN)|**0.042 ± 0.002**|**0.007 ± 0.001**|**0.020 ± 0.001**|**0.002 ± 0.001**|\n|TARNet|0.064 ± 0.006|0.020 ± 0.003|0.517 ± 0.147|0.092 ± 0.032|\n|X-Learner (HPO)|0.069 ± 0.008|0.020 ± 0.003|0.155 ± 0.033|0.048 ± 0.016|\n|X-Learner (TabPFN)|0.083 ± 0.010|0.015 ± 0.005|**0.019 ± 0.001**|**0.001 ± 0.000**|\n\nWe observe that Do-PFN-v1.1’s extended pre-training leads to strong performance among competitive baselines on three out of four RealCause datasets. We note that these datasets perfectly satisfy unconfoundedness, giving our baselines a fundamental advantage against Do-PFN. These results show Do-PFN to nevertheless be competitive with a strong set of CATE baselines, suggesting the strong transfer of Do-PFN's synthetic pre-training to complex, real-world data.\n\n\n# Response to Reviewer bczT\n----\n\nWe thank you for your positive feedback and very meaningful questions! \n\n> How sensitive is Do-PFN to mismatches between the synthetic prior used during training and the test-time SCMs?\n\nTo address this,  in addition to the strong real-world performance we observe above, we also perform several ablation studies regarding distribution mismatch. We also refer to our response to reviewer **Fczh**, where we provide OOD results in terms of graph size, and a theoretical argument regarding prior mismatch.\n\nOur new out-of-distribution analysis shows the performance of Do-PFN on in-distribution samples from its prior, compared to out-of-distribution samples in terms of 1) graph size, 2) non-linearity type, and 3) noise distributions. \n\n## Median MSE - Out of Distribution\n\n|OOD Type|Do-PFN (v1)|% Change|\n|:--------------------|:--------------------|:-----------|\n|In Distribution|**0.0053 ± 0.0008**|0.0%|\n|ELU Non-Linearity|0.0033 ± 0.0004|-38.2%|\n|Post Non-Linearity|0.0057 ± 0.0006|7.0%|\n|Sin Non-Linearity|0.0066 ± 0.0007|24.6%|\n|Student-t Noise|0.0050 ± 0.0006|-4.6%|\n|Laplacian Noise|0.0060 ± 0.0008|13.3%|\n|Gumbel Noise|0.0059 ± 0.0007|10.9%|\n|High Gaussian Noise|0.0212 ± 0.0006|301.4%|\n\nWe find that Do-PFN is most robust to changes in the noise distribution and with respect to post-non-linearities, with its MSE in CID prediction changing by a magnitude of less than 0.001. Interestingly in some cases, such as the “ELU Non-Linearity” ablation, going out of distribution results in an increase in performance, possibly since ELU shares functional similarities to the ReLU activation which is represented in our prior.\n\n> How does the training time and inference speed compare to existing causal inference methods in settings with larger datasets or real-time requirements?\n\nWe thank you for the opportunity to discuss the relationship between training- and inference time of Do-PFN. Do-PFN is a foundation model of 7.3 million parameters (small compared to current LLMs, but large, compared to traditional causality methods). It is pre-trained for up to 48 hours, but its pre-training time is incurred **once** up-front. When applying DoPFN to a new dataset, predictions can be obtained in a single forward pass. \n\nWe now provide inference time results, calculating the median time in seconds of the *.fit* and *.predict* functions of Do-PFN and our baselines. We also provide versions of several baselines with HPO for 60 seconds per underlying base model, and run deep-learning-based methods on an RTX-2080 GPU. \n## Median Train/Optimize + Inference Time (s)  - RealCause\n\n|Method|LalondeCPS|LalondePSID|IHDP|ACIC2016|\n|:-----------------------|:------------------|:------------------|:------------------|:------------------|\n|Causal Forest DML|8.075 ± 0.090|8.014 ± 0.083|6.839 ± 0.051|19.070 ± 0.239|\n|DML (TabPFN) GPU|12.376 ± 0.035|12.441 ± 0.094|11.740 ± 0.029|22.807 ± 0.271|\n|Do-PFN (v1) CPU|2.699 ± 0.047|2.324 ± 0.131|**0.699 ± 0.009**|**2.734 ± 0.179**|\n|Do-PFN (v1) GPU|**1.029 ± 0.098**|**1.311 ± 0.162**|1.270 ± 0.143|**1.060 ± 0.160**|\n|Do-PFN (v1.1) CPU|3.010 ± 0.201|2.188 ± 0.013|**0.655 ± 0.004**|4.474 ± 2.042|\n|Do-PFN (v1.1) GPU|**1.011 ± 0.002**|**0.990 ± 0.004**|0.962 ± 0.001|**1.035 ± 0.036**|\n|Dont-PFN GPU|2.916 ± 0.169|2.437 ± 0.097|**0.769 ± 0.005**|3.258 ± 0.125|\n|DragonNet GPU|67.469 ± 14.605|124.642 ± 10.753|6.791 ± 0.013|6.378 ± 0.249|\n|S-Learner (HPO) |60.160 ± 0.072|60.373 ± 0.117|60.066 ± 0.012|60.498 ± 0.206|\n|S-Learner (TabPFN) GPU|**2.001 ± 0.004**|**1.927 ± 0.012**|1.791 ± 0.002|4.694 ± 0.105|\n|T-Learner (TabPFN) GPU|3.473 ± 0.009|3.573 ± 0.168|3.309 ± 0.005|7.378 ± 0.202|\n|TARNet GPU|58.258 ± 21.274|119.821 ± 14.448|6.804 ± 0.008|6.728 ± 0.293|\n|X-Learner (HPO) |120.240 ± 0.094|120.172 ± 0.102|120.060 ± 0.007|120.201 ± 0.050|\n|X-Learner (TabPFN) GPU|13.555 ± 0.239|13.469 ± 0.216|12.886 ± 0.037|31.158 ± 2.123|\n|Dataset Shape|2500,5|1340,5|375,7|2405,50|\n\nIn our runtime analysis, Do-PFN (run on CPU and GPU) has a faster inference speed than all other baselines. Do-PFN outperforms other baselines because there is no conventional model-fitting; and is slightly faster than the TabPFN meta-learning approaches (when both methods use a GPU), since Do-PFN has less parameters and does not do ensembling. \n\n> While the paper claims Do-PFN learns to “implicitly” adjust for confounding (e.g., back-door/front-door), it would benefit from deeper analysis.\n\nWhile we empirically find that Do-PFN works very well across diverse causal challenges, on both carefully controlled synthetic setups (including front-door and back-door cases) as well as real world data, the precise inner workings of PFNs in general, and thus also Do-PFN, are rather unclear. We consider it unlikely that Do-PFN works analogously to classical multi-step approaches that would, e.g., perform causal discovery, then determine estimands, and finally carry out the appropriate adjustment, but the precise in-context-learning dynamics of Do-PFN remain to be explored. \n\nIn the field of LLMs, mechanistic interpretability tools have become popular and their applications to PFNs, and Do-PFN in particular, might allow to systematically trace specific procedures or algorithms Do-PFN utilizes internally. \nBased on your question, we will expand on this as a key area of future work in the discussion section. \n\n## References \n\n[1] Neal et al. \"Realcause: Realistic Causal Inference Benchmarking.\" arXiv 2020.\n\n[2] Bereska, Leonard, and Stratis Gavves. \"Mechanistic Interpretability for AI Safety-A Review.\" TMLR 2024."}, {"review_id": "D75xdE1b4c", "reviewer": "Reviewer_Ww3g", "summary": "This paper proposes a novel meta-learning approach for causal effect estimation that pre-trains a Prior-data fitted network (PFN) on synthetic observational and interventional data drawn from various common structural causal models. Differing from the inspired work of TabPFN, designed for standard prediction tasks on tabular datasets given in-context training sets, Do-PFN performs amortized causal inference by outputting a conditional intervention distribution given an observational dataset, interventional covariates and an intervened treatment variable. The trained model is evaluated on synthetic data from six common graph structures requiring Do-PFN to perform front door and back door adjustment. It is shown that the learned method performs comparably to baseline algorithms that access the true causal graph in terms of CID and CATE and that it outperforms other baselines without this assumption. Ablation demonstrates the method performs well even on small datasets and is robust to magnitude of ATE and graph sizes.", "questions": "Some questions were raised above. \n\n1. The ACTIVA method presented in Sauter et al. seems to have many similarities with Do-PFN. While I understand that the output of ACTIVA produces the shift in distribution for all variables after performing and intervention, wouldn’t this also inherit the shift in outcome variables distribution which Do-PFN is predicting? I am curious what are the primary differences that makes Do-PFN a stronger approach than ACTIVA, and accordingly where your paper's main source of novelty lies.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. Applying the popular PFN to the task of causal inference is an attractive extension.\n2. Presents a reasonable framework for performing causal inference without knowledge of the causal graph or prior interventions, increasing practical utility.\n3. Quite interesting that Do-PFN can capture uncertainty arising from unidentifiability of causal effects.\n4. Paper is very well-written\n\nWeaknesses:\n1. Do-PFN is entirely pretrained on synthetic data from a very constrained set of possible SCMs as explained in Appendix B. Given evaluation is performed on the exact same class of synthetic SCMs, it is not surprising that the performance would be quite strong. However, since there is no discussion of why this synthetic prior over SCMs would capture real-world causal systems, it is unclear whether this model would at all useful in practice. Additional experiments demonstrating that this method of prior fitting can lead to good real-world performance seems essential.\n2. Paper lacks a clear understanding of where and to what extent uncertainty arises in the model’s predictions. Ie. why is it the case that uncertainty is under confident for theoretically identifiable cases?\n3. This work considers only the class of single-node binary interventions, which is restrictive in practice. A consideration of multi-node continuous interventions would be much stronger.", "was_revised": true, "final_justification": "I am much more confident following the rebuttal from the authors that Do-PFN can be used as a practical tool for the accurate estimation of causal effects without prior knowledge of the causal graph. As far as I am aware, using in-context learning approaches for causal effect estimation is quite novel and quite attractive, and while the work may have some limitations in terms of the intervention types they accept and still questionable prior matching, I believe it is a great starting place for considering a new look on causal inference.", "rebuttal": "# General Response\n----\n\nAll reviewers acknowledge that our paper presents a promising approach, but also raise the question of prior mismatch, especially when analyzing real-world data. While we believe it is crucial for a new method to study synthetic data with known ground truth, we agree that analyzing real world data is important. Therefore, we have meanwhile performed an analysis of Do-PFN's performance on **six datasets based on real-world data**, with promising results that we detail next: \n\n**Known Graph:** First, we evaluate on the Amazon Sales and Law School Admissions datasets where the causal graph is known (please refer to the .zip file in the supplementary material attached to the original submission for details): \n\n## Median MSE (CATE) - Known Graph\n|Method|Law School|Amazon Sales|\n|:------------------------|:------------------|:------------------|\n|Causal Forest DML|0.157 ± 0.056|0.136 ± 0.279|\n|Causal Forest DML (HPO)|0.519 ± 0.080|0.579 ± 0.316|\n|DML (TabPFN)|0.283 ± 0.067|**0.037 ± 0.169**|\n|Do-PFN (v1)|**0.029 ± 0.010**|**0.038 ± 0.023**|\n|Do-PFN (v1.1)|**0.067 ± 0.043**|0.209 ± 0.135|\n|Dont-PFN|**0.103 ± 0.046**|0.313 ± 0.158|\n|DragonNet|0.371 ± 0.035|0.579 ± 0.316|\n|S-Learner (HPO)|0.123 ± 0.045|0.233 ± 0.246|\n|S-Learner (TabPFN)|0.161 ± 0.051|0.568 ± 0.303|\n|TARNet|0.372 ± 0.119|0.579 ± 0.316|\n|T-Learner (HPO)|0.151 ± 0.095|**0.017 ± 0.072**|\n|T-Learner (TabPFN)|0.161 ± 0.048|0.579 ± 0.316|\n|X-Learner (HPO)|0.519 ± 0.080|0.579 ± 0.316|\n|X-Learner (TabPFN)|0.134 ± 0.047|0.579 ± 0.316|\n\nWe find that Do-PFN (v1) achieves the strongest performance on Law School and is not significantly worse than the best method on Amazon Sales, a dataset with a complex mediation structure.\n\n**RealCause:** We also evaluate on four new datasets from the RealCause benchmark [5], using a version of Do-PFN (which we call v1.1) pre-trained on larger graphs with up to 60 nodes. \n\nWe observe that Do-PFN-v1.1’s extended pre-training leads to strong performance among competitive baselines on three out of four RealCause datasets. We note that these datasets perfectly satisfy unconfoundedness, giving our baselines a fundamental advantage against Do-PFN. Due to space constraints we refer the Table “**Median MSE (CATE) - RealCause**” in our general response to reviewer **Fczh** for the precise MSE values.\n\nThese results show Do-PFN to nevertheless be competitive with a strong set of CATE baselines, suggesting the strong transfer of Do-PFN's synthetic pre-training to complex, real-world data. \n\n\n# Response to Reviewer Ww3g\n----\n\nThank you for your feedback and interesting questions!\n\n> Paper lacks a clear understanding of where and to what extent uncertainty arises in the model’s predictions.\n\nWe thank you for encouraging us to further develop our theory regarding uncertainty! \n\nThe posterior predictive distribution that Do-PFN approximates can be factorized as follows: $p(y|x, do(t), D) = \\int p(y|x, do(t), \\psi) p(\\psi|D) d\\psi$, where $y$ denotes the outcome, $x$ a covariate vector, $\\psi$ an SCM,  and $D$ an observational dataset. \n\nLet’s also consider the observational equivalence relation $\\psi_1 \\sim \\psi_2 \\iff p(y,x,t|\\psi_1) = p(y,x,t|\\psi_2)$, where a selector $r(\\psi)$ maps any SCM $\\psi$ onto the representative of its equivalence class $[\\psi]$. Subsequently, this yields: \n\n$p(y|x, do(t), D) = \\int \\int p(y|x, do(t), \\psi) p(\\psi|r) p(r|D) dr d\\psi$\n\nThis factorization allows to distinguish three types of uncertainty: \n\n- First, $p(y|x, do(t), \\psi)$ represents the uncertainty in the conditional interventional distribution of the outcome $y$ assuming a **fixed** SCM $\\psi$; all uncertainty about $y$ in $p(y|x, do(t), \\psi)$ is caused by the noise distributions in the SCM $\\psi$ and thus represents a form of aleatoric uncertainty. \n\n- Second, $p(\\psi|r)$ is the distribution over observationally equivalent SCMs assuming we are in equivalence class $[r]$, and thus represents uncertainty due to potential unidentifiability of $\\psi$ from observational data. \n\n- Finally, the distribution $p(r|D)$ quantifies the epistemic uncertainty about the Markov equivalence class of a ground truth SCM caused by a finite observational dataset $D$. \n\nWe show, in a newly added section of our paper, that for infinite observational data the uncertainty in $p(r|D)$ vanishes completely, but the other sources of uncertainty remain. More specifically, when the observational data $D$ comes from a ground-truth SCM $\\psi_0$, the posterior conditional interventional distribution $p(y|x, do(t), D)$ converges to the posterior $p(y|x,do(t),[\\psi_0])$ where $[\\psi_0]$ denotes the set of all observationally equivalent SCMs. \n\nMore formally:\n\n$$p(y|x, do(t),D) \\rightarrow p(y|x,do(t),[\\psi_0]) =  \\int p(y|x, do(t), \\psi) p(\\psi|[\\psi_0]) d\\psi, \\quad \\text{for}\\  |D| \\rightarrow \\infty.$$\n\nWe formulate all those arguments in a rigorous way using measure theory and Doob’s theorem in a new theoretical section in the appendix of our manuscript. \n\n> Why is it the case that uncertainty is under-confident for theoretically identifiable cases?\n\nThis is a very interesting question! \n\nAs we show in Proposition 1, we minimize the **forward** KL-divergence $D_{KL}[p(y|x, do(t), D)||q(y|x,do(t), D)]$ between the posterior $p(y|x, do(t), D)$ and the model $q(y|x,do(t), D)$. This leads to $q$ being mass covering, i.e. overdispersed—especially when $q$ is a suboptimal approximation of $p$ [2,3]. We parametrize $q$ as a bar distribution, with fixed positions of the bars following TabPFN [4]. In the identifiable case the posterior is more concentrated leading to a slightly larger discrepancy between the posterior $p$ and the model $q$ potentially due to discretization error. This difference manifests itself in Do-PFN being **slightly** underconfident (please see our calibration curves), rather than overconfident due to the nature of the forward KL-divergence. \n\nPlease also note that underconfidence in case of approximation error is typically considered a very desirable property [3]. \n\n> While I understand that the output of ACTIVA produces the shift in distribution for all variables after performing and intervention, wouldn’t this also inherit the shift in outcome variables distribution which Do-PFN is predicting? I am curious what are the primary differences that makes Do-PFN a stronger approach than ACTIVA, and accordingly where your paper's main source of novelty lies.\n\nThank you for the opportunity to point out the differences between Do-PFN and the contemporaneous ACTIVA method. We summarize some of the major differences below: \n\n- **Task:** DoPFN is trained to learn how to predict the effect of a binary intervention on a univariate outcome variable, which is a fundamental yet practically relevant and widely researched task in causal inference with a broad set of available baselines and benchmarks [5]. ACTIVA considers the problem of learning the effect of multi-variable interventions on the multivariate distribution of all variables in an SCM simultaneously. \n\n- **Training data:** ACTIVA is trained on data from fixed-sized, purely linear additive causal models (without any nonlinearities) and the SERGIO simulator (with 11 nodes) with knockout interventions. Causal sufficiency is assumed, i.e. covariates in an SCM are observed. In contrast, we vary the nonlinearities, graph sizes and noise distributions and also include unobservable variables. Conceptually, it would be possible to marginalize the output distribution of an ACTIVA-model after conditioning on only a single intervention in order to obtain predictions for a single variable. However, this would not bridge the gap between vastly different pre-training setups. \n\n- **Model:** Do-PFN learns a univariate posterior predictive and employs the same architecture as TabPFN [4], as well as the same histogram-based posterior approximation, which has been found to be highly effective for predictive tasks. ACTIVA learns a multivariate posterior and employs a $\\beta$-VAE with Vamp prior plus a mixture-of-Gaussian parametrization of the conditional interventional distributions and is based on transformer encoders and decoders with alternating row- and column attention.\n \n- **Evaluation:** We compare Do-PFN against established methods for causal effect estimation besides observational baselines. ACTIVA is evaluated in terms of similarity of the predicted distribution compared to the ground truth interventional distribution (for instance evaluated in terms of MMD); we investigate the **predictive performance** of Do-PFN on specific causal graphs and semi-synthetic data. \n\n> This work considers only the class of single-node binary interventions, which is restrictive in practice. A consideration of multi-node continuous interventions would be much stronger. \n\nWe focus on this specific task since it is one of the most researched problems in causal inference, see e.g. [5]. We believe that, due to its relevance, this warrants studies that fully evaluate the performance of different methods on synthetic and semi-real benchmarks, as well as comparisons against numerous existing baselines. It’s a good starting point.\n\n## References \n\n[1] Neal et al. \"Realcause: Realistic Causal Inference Benchmarking.\" arXiv 2020.\n\n[2] Murphy, Kevin P. \"Probabilistic machine learning: Advanced topics.\" MIT press, 2023.\n\n[3] McNamara, et al. \"Sequential monte carlo for inclusive kl minimization in amortized variational inference.\" AISTATS 2024.\n\n[4] Hollmann, Noah, et al. \"TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second.\" NeurIPS 2022. \n\n[5] Yao, Liuyi, et al. \"A survey on causal inference.\" ACM Transactions on Knowledge Discovery from Data (TKDD) 15.5 (2021): 1-46."}]

Author rebuttal:

Meta-review: This paper presents a new approach named Do-PFN for zero‐shot causal inference via meta‐training, which extends the prior-data fitted networks (PFNs) to dealing with treatment effect estimation task on tabular data. Extensive experiments with ablation studies demonstrate the effectiveness of this approach. 

Reviewers recognized the novelty and technical contributions of this work. This paper provides an innovative approach for applying PFNs to causal inference. Theoretical analysis provides valuable insights on why the proposed method works. Moreover, the experimental results and analysis are comprehensive and convincing. 

Meanwhile, reviewers found that the paper has some limitations, such as lack of evaluations on real-world datasets, missing baselines, computational efficiency, etc. The authors have provided detailed responses with additional results that help address the concerns from reviewers. 

Overall, this paper is a solid work that extends PFNs to dealing with causal inference tasks. Considering the novelty and technical contributions of this work, I recommend it for Spotlight presentation.
Decision: Accept (spotlight)