CausalPFN: Amortized Causal Effect Estimation via In-Context Learning
Authors: Vahid Balazadeh
Hamidreza Kamkari
Valentin Thomas
Junwei Ma
Bingru Li
Jesse C. Cresswell
Rahul Krishnan
Venue: NeurIPS 2025 spotlight ()
Keywords: Causal Inference
Prior-Data Fitted Networks
Bayesian Inference
Ignorability
In-Context Learning
Meta-Learning
Automated Causal Inference
Primary area: probabilistic_methods
TL;DR: CausalPFN is a pre-trained transformer that amortizes causal effect estimation: trained once on simulated data-generating processes, it outputs calibrated effects for new observational datasets with zero tuning.

Abstract: Causal effect estimation from observational data is fundamental across various applications. However, selecting an appropriate estimator from dozens of specialized methods demands substantial manual effort and domain expertise. We present CausalPFN, a single transformer that *amortizes* this workflow: trained once on a large library of simulated data-generating processes that satisfy ignorability, it infers causal effects for new observational datasets out of the box. CausalPFN combines ideas from Bayesian causal inference with the large-scale training protocol of prior-fitted networks (PFNs), learning to map raw observations directly to causal effects without any task-specific adjustment. Our approach achieves superior average performance on heterogeneous and average treatment effect estimation benchmarks (IHDP, Lalonde, ACIC). Moreover, it shows competitive performance for real-world policy making on uplift modeling tasks. CausalPFN provides calibrated uncertainty estimates to support reliable decision-making based on Bayesian principles. This ready-to-use model requires no further training or tuning and takes a step toward automated causal inference (https://github.com/vdblm/CausalPFN/).

CausalPFN: Amortized Causal Effect Estimation via In-Context Learning
Vahid Balazadeh
Hamidreza Kamkari*3 Valentin Thomas Jesse C. Cresswell
Benson Li
Junwei Ma
Rabul G. Krishnan
vahidêcs. toronto edu, hamid0layer6. University of Toronto
Vector Institute
*Layer 6 Al
Abstract
Causal effect estimation from observational data fundamental across various applications. However, selecting an appropriate estimator from dozens of special- ized methods demands substantial manual effort and domain expertise. We present CausalPFN, single transformer that amortizes this workflow: trained once on large library of simulated seta-generating processes that satisfy ignorability, it infers causal effects for new observational datasets out of the box. CausalPFN combines ideas from Bayesian causal inference with the large-scale training proto- col of prior-fitted networks (PFNs), learning to map raw observations directly causal effects without any task-specific adjustment. Our approach achieves superior average performance heterogeneous and average treatment effect estimation benchmarks (IHDP, Lalonde, ACIC). Moreover, it shows competitive performance for worldw policy making uplift modeling tasks. CausalPFN provides cali- brated uncertainty estimates to support reliable decision- -erisiin-mmaing based on Bayesian principles. This ready-to-use model requires further training tuning and takes a step toward automated causal inference (https://itthub. wwwplm/aauaallllPiccaaaaa
Introduction
75th percentile)
CausalPEN
Deep Leaming
DA-Learner
2
EccaML
BART, GRF
Causal inference inference-essimaaiing the effects of interventions from data-is fundamental across numerous domains, including pub-
CausalPFN
DragonNet
Median Rank (25th- (25th-75th
T-Leamer
Lernnrr
lie policy, economics, and healthcare [75, 5,
47]. The central challenge lies in estimat- ing causal quantities from observational data: records collected without explicit interven- tions, where confounding factors can obscure true causal effects. Various causal identifi- cation settings have emerged to address this challenge [45, 6, 10, 71]. Perhaps the most common one is to assume no unobserved con- founding (ignorability or backdoor) [98, 87].
8
TarNet RA-Net
10
BART
S Learmer
GRF
Forest DML/ DR-Learer
10a
10a
10a
1
Median Time per 1,000 Samples (seconds, log scale)
Figure Time vs. Performance. Comparison across 310 causal inference tasks from IHDP, ACIC, and Lalonde. CausalPFN achieves the best average rank (by precision in estimation of heterogeneous effect) while being much faster than other baselines.
Even within the conceptually straightforward ignorability framework, researchers have de- veloped dozens specialized causal estima- tors over the past four decades. Prominent examples include Meta-Learners [57]. doubly robust methods [30, 52], double machine learning
'Equal Contribution
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Traditional Paradigm
CausalPFN Training & Inference
T
Gather identifiable DGPs
X
Estimated Effects
Observational Data
New Observations
Conceptualize Different DGPs
Simulate Observations
Simulate Effects
Y
Estimate Effects
Train to Match
Design or Select Estimator
Estimated Effect
Figure 2: Traditional Causal Inference vs. CausalPFN, (Left): A domain expert manually builds or selects an estimator for DGP that they deem appropriate for the given data. (Right): The domain expert simulates diverse DGPs for pre-training, and transformer learns to amortize causal inference automatically.
(DML) [16, 29]. and neural network approaches [104, 106, 19-22], among others [97, 56, 70, 65]. This large number of estimators creates practical challenges as domain expertise is required to select, tune, or design the most appropriate estimator for each application [107, 103, 27, 2, 81, 73].
The Bayesian paradigm offers an elegant framework to address these challenges [99, 46, 47, 40, 9]: rather than manually designing or selecting the best estimator, one can: (1) parameterize an appropriate prior distribution over plausible underlying causal mechanisms, i.e., the data-generating processes (DGPs), (2) define the causal estimand functional of the DGP parameters, (3) compute posterior distribution over DGPs conditioned on observed data, and (4) derive the posterior predictive distribution (PPD) of the causal estimand, However, the practical adoption of Bayesian methods remains limited. Computing posterior distributions typically requires expensive sampling methods [84, 40], which often leads researchers make specific assumptions about the DGPs or priors that are not necessarily reflective of the complexity of the downstream tasks [36, 62].
Meanwhile, an emerging area in deep in-context learning suggests using large models that can approximate PPDs by taking the entire list of observations as context and amortize the expensive process of posterior inference [32, 54]. A successful example is the prior-fitted network (PFN) [80] that achieved remarkable performance in tabular prediction tasks [42, 69, 37, 43, 118, 76, 66). PFNs employ transformer architectures trained on large-scale simulated DGPs, representing rich prior, to perform posterior predictive inference via in-context learning; given dataset of input-output examples as context, they can predict outputs for new inputs. PFNs shift the computational burden from inference time to (pre-)training and accurate predictions on unseen datasets. However, they are only designed for regression and classification, not for causal inference.
producing single set of model parameters that can make fast
We propose to bridge the large-scale training of amortized models with Bayesian causal inference and introduce CausalPFN, transformer model for causal effect estimation via in-context learning. Our framework leverages general-purpose prior, based on the ignorability assumption, generate vast collection of simulated DGPs. By training on these diverse DGPs, our method learns to infer the causal estimands directly from observational data. While our approach requires an expensive pre-training phase, once complete it is ready for inference on new datasets with no further training, fine-tuning, or hyperparameter optimization. Hence. CausalPFN is casy-to-use, efficient for inference, and shows remarkably strong performance as an estimator. Figure 1 illustrates the relative performance and efficiency of our method compared to standard baselines. For inference on an unseen dataset, CausalPFN requires only forward passes, whereas baseline methods have additional costs including hyperparameter tuning or validation. We therefore report the computational time for all of these stages for the baselines to reflect the total costs of predicting
new dataset.
We show CausalPFN's workflow compared traditional causal inference in Figure 2. Our key contributions are: (i) To our knowledge, for the first time, we demonstrate that a single transformer- based model trained on a diverse library of simulated DGPs can match or surpass specialized estimators across multiple datasets without task-specific tuning. Specifically, CausalPFN achieves the best average rank on CATE across IHDP, ACIC, and Lalonde, and competitive ATE performance, without task-specifi tuning. (ii) We highlight CausalPFN's competitive out- out-of-the-bor the-theexxxx performance for real-world policy making on various uplift modeling tasks. (ii) We theoretically characterize the assumptions under which CausalPFN's estimates are asymptotically consistent. (iv) We develop principled uncertainty quantification framework for CausalPFN to produce finite-sample calibrated credible intervals for the estimates. (v) Finally, we release our model's weights with user-friendly API, streamlining the adoption of CausalPFN as a capable estimator. CausalPFN is fast, ready-to-use, and does not require any further training hyperparameter tuning.

---

2 Background
Causal Effect Estimation. We adopt the potential-outcomes framework for causal inference [100]. Let E T denote the treatment from finite treatment set T, and X E X the observed covariates. For every E T, Y E R is the potential outcome under treatment t. while the observed (factual) outcome is Y == Yr. We call the joint distribution P(X, T, {Yi Y) the data- sattageneraating process (DGP), and denote by Pose the marginal distribution over observed triples (X, XXTT.). Y). Given samples from P obs central goal is recover the conditional expected potential outcomes (CEPOs):
He (x) == Y X x].
Vt E T, X
(I)
For binary treatments, two common estimands, average treatment effect (ATE), and conditional average treatment effect (CATE) follow directly from the CEPOs. We refer CEPOs, CATE, and ATE collectively as causal effects.
ATE: : x= - Yol = (X) po()),,
CATE:
x(x) = :x E Yo x [1) (x()).
Estimating causal effects from observational data is impossible without further assumptions: different DGPs can induce the same Poss but have different causal effects [87, 39, 47J. We thus define: Definition 1 (CHPO-Identfiabillty). For each & 'T CEPO-identifability holds when fe can be written as functional of the observational distribution Pots-
Throughout, we assume strong ignorability, standard sufficient assumption that makes CEPOs identifiable. Strong ignorability posits that, conditional on observed covariates, treatment assignment has positive probability for all E TT and is independent of all potential outcomes [98, 97, 89]:
Assumption (Strong Ignorability). (i) Y, T for all E (Unconfoundeednes)) and (ii)
P(T = X)
> a.e. for all t E T (Positivity).
Bayesian Causal Inference. A Bayesian formulation of causal inference considers explicit likelihood model for the DGP [99, 84, 62]. Let &b be the parameter that indexes the DGPs pd (X, T, (Vi).eT Y). A prior (1b) encodes domain knowledge on parameters v/. Given i.i.d.
observations Dohs = ((x((n),(t c,n)), ((m),y/ (n())) } coming from the observational distribution Pal Bayes' rule yields the posterior T Donss. For any functional for ATE-the posterior predictive distribution (PPD)
for example g( = B Yol
/I(O(t) E B)T(U Dobe) deb
(- Dobe)= B
B B,
(4)
is induced by the posterior distribution n(th Dobs) (B denotes the Borel o-algebra over R). Point estimates (posterior means) and credible intervals therefore arise automatically from these induced posteriors. Because the posterior is rarely available in closed form, one resorts approximate inference such as Markov-chain Monte-Carde (MCMC) [40] or variational inference [68, 48J. Such techniques have been applied with flexible priors including nonparametric BART models [40, 36), Dirichlet processes [64] and Gaussian processes [3]. In summary, the Bayesian paradigm offers uni- fied framework for inference on causal estimands and provides automatic uncertainty quantification.
Amortizing Posterior Predictive Inference with Prior-Fitted Networks. Running a new posterior inference for every dataset is computationally demanding, especially with high-dimensional covari- ates [36, 62]. Recent work shows that in-context transformers can amortize Bayesian prediction: instead of sampling from the posterior test time, a single network trained map a context set directly to the PPD [31, 32, 54, 80, 37J. PFNs instantiate this idea for supervised learning [42].
Consider a supervised dataset DSL. {(x("), (")) )> and prior S1, on parameters ø indexing (X Y). The Bayesian approach to predict the output for a new input x is to use the PPD
PPD(Y X x, DSL) / ( X x) (Ø DDL) dó.
Rather than approximating the posterior distribution n. = inference [49, 4, 82), PFNs directly parameterize the PPD using a single transformer model go (Y DSI) by minimizing the data-prion loss
DSLL with MCMC or variational
Eo = ESrxs DSL((XY))-P) [- [-log x, x.
(6)
3

---

Crucially, training requires only prior samples (6. D no posterior sampling is needed. With suitably rich prior, single PFN can be applied off the-thelff diverse predictive problems [69, 43].
3 The Mathematical Framework of CausalPFN
Our primary estimands of interest the CEPOs from (1). As shown in (2) and (3), CEPOs directly enable estimation of both ATE and CATES. Therefore, we focus on developing an estimator that can accurately infer these quantities from the observational data. Specifically, paradigm for causal inference, as introduced in Section 2, and parameterize CEPOs as μ v)) Given suitably rich prior distribution over the DGPs, which we will explicitly design in Section we define our target as the posterior predictive distribution of CEPOs:
follow the Bayesian
Definition (CEPO-PPD). For each E T and covariate vector x, the CEPO-PPD is
(- |-1x, X, Doba) = B [++/ f /Ilu(x: 4) E B)
Doba) dı
BEB.
(7)
Consistent Estimation of CEPOs. The CEPO-PPD captures the epistemic uncertainty about the CEPO encoded in the posterior. A concentrated distribution x" indicates that the observations Dobs are informative and enough samples are available to accurately pin down the true CEPO, whereas a high-variance distribution implies that the data is insufficient for estimation. With that in mind, we now study under which conditions increasing the size of the observations Dobs allows us to accurately recover the true CEPO from the CEPO-PPD. This is given through the following informal result (re-stated and proven formally in Appendix B) which provides necessary and sufficient conditions on the prior π under which the CEPO-PPDs enable consistent estimation of the CEPOs:
Proposition 1 (Informal). Under mild regularity assumptions, for almost all x and any set of i..dd samples Dobs P we have that as (Dobsl - 00,
Bio (-px, [p]] 0.5 pe(x; b²), VT, and almost all x E X, if and only if the prior is CEPO-identifiable, that is for almost all 1p n, the CEPOs pel: v)) only depend on the observational distribution Poim (Definition 1).
(8)
(Proof sketch) We group all DGPs tb that share the same observational distribution equivalence class and induce prior obtained from on the resulting quotient space. By Doob's theorem [26]-a classical result from Bayesian consistency theory-the posterior on this new prior almost surely concentrates on the true equivalence class once asymptotically many observations are given. Consequently, for any functional of the observations that is constant within each equivalence class, its posterior predictive converges almost surely to its true value. Importantly, the causal functional of interest, Pt, can be written as functional of the observations if and only if the corresponding DGP has identifiable CEPOs. Thus, identifiability is both necessary and sufficient to ensure that μt is constant throughout the equivalence class, and for the consistency result to hold.
abs into an
(Remark /) While the algorithms in our paper use strong ignorability. Proposition 1 itself is entirely general result and can be extended to DGPs that are not necessarily ignorable, but whose CEPOs satisfy identifiability in Definition 1. Importantly for our practical setting, when the prior x enforces strong ignorability, Proposition suggests that the CEPO-PPDs consistently recover the true CEPO.
(Remark 2) Proposition 1 highlights two key design principles for the prior (i) must rule out con-identifiable cases, and, once identifiability is secured, (ii) broadening и increases the chance that particular 5** lies within its support, thus enabling consistent recovery of the true CEPO for that x!*.
Learning the CEPO-PPD Having shown that CEPO- are useful for estimating the true CEPOs, we now describe how to learn them. Inspired by PFNs, we train a single transformer qo to approximate the full predictive distribution nh, To fit this model, we introduce the following loss:
Definition 3 (Causal Data- Loss). For any t T, define the causal data prior loss
(0) EE E (am ~ p. |- log qo(pu v) x, Doba))
In Appendix C, we show that minimizing (0) also minimizes the KI--divergence between the true CEPO-PPD and qe, leading to go(- x, Dobs) xe TH x, Dobs) for all t E T. This entire training

---

π
Sampled DGP
Transformer
Implicit Posterior
Simulate Dobe
Sample Query (x, t)
p (x, I, (Y
p
pvs
p
Prior
Simulate CEPO: (x: ().)
log qe (- | x, ł, Datm)
Figure 3: Causal Data-Prior Training. At each iteration index vi, is sampled (left), yielding the DGP
poi po:(X, T, (Yi).e,, Y). From this DGP we simulate an observational context Doie and a query (x, f) with its true (x (22) (center). Passing (x, t, Data) through the transformer predicts the CEPO-PPD qo(- x, t, Dobs) (in yellow), which is derived from an implicit posterior s(- Pota) that is never explicitly computed (right). We train 0 to minimize the causal data-prios loss (bottom).
process shifts the computational burden from inference pre-training: rather than evaluating the posterior π ( Doha) at test time, the model learns to map observational data directly to the corre- sponding predictive distribution. When the model is well-fitted, the prior satisfies the assumptions of Proposition 1, and Dobs is sufficiently large, the predicted Qe accurately pins down the true CEPO.
Figure 3 visually illustrates optimizing the causal data-prior loss using stochastic gradient descent: at each iteration, we sample a DGP dh ~ ग,, generate an observational dataset Dobs from this DGP, and select query point (x, t). We compute (simulate) the ground- -ruth CEPO μ4l (x : 4/5) and feed both the observational data and query to the model. The model outputs a CEPO-PPD, and we update using gradient descent to increase the probability assigned the true CEPO value. Through training, 0 minimizes the data-prior loss and implicitly learns to perform posterior predictive inference, and estimate the predictive distribution n without ever explicitly computing the posterior.
Point Distributional Estimation Causal Effects. Given observational data Dobs from an underlying v²,, natural point estimate for CEPOs is the mean of the predicted CEPO-PPD, Exo-go (- (ot,, [u] se [x(x; )* These CEPO estimates can also form point estimates for CATEs using (3), and for ATES using (2) by empirical averaging across units in Dobs
Beyond point estimation, the estimated CEPO-PPDs can also capture the epistemic uncertainty about the causal effects. We can use to construct credible intervals around CEPOs, CATEs, and ATEs via sampling from go(- x, 1, Dohs) and qo(- x, = 0, Dohs). We can then use these intervals to quantify the uncertainty of our estimated causal effects.
4 Implementing CausalPFN
While Section presents the framework in general form (arbitrary finite T and identifiability). for implementation we focus on binary treatments T = {0, 1} under strong ignorability. These assumptions reflect the most common settings encountered by practitioners and serve as a natural starting point. Extending the implementation and algorithms to more general settings is left for future work.
Retrieve or Synthesize Base Tobles
0
X
Ho P1
Yo
Y1
Here, we focus designing an appro- priate prior и over DGPs that satisfies the theoretical re- quirements established in Proposition 1. This prior must balance two factors: First, it should contain a rich set DGPs with sufficient coverage to approximate real-world ecenario-simile to the priors used in successful tabular predictive models like TabPFN [42, 43], TabDPT [69], and TabICL [92]. Second, and uniquely for causal inference, all DGPs in our prior must satisfy strong ignorability which directly implies identifiability of the prior. Moreover, the generated DGPs must allow us to access the ground-truth CEPOs, as required by the causal data-prior loss in Definition for training.
A Scalable Prior.
random propensity zeero-mean noise
Figure 4: Prior construction. Sample di- verse base tables (OpenML or synthetic TabPFN), select covariates X, draw treat- ment T with random propensity model, select columns /lo fi and add zero-mean noise to form Yo, Yi, and Y.
To address these requirements, we develop a procedure that can transform any base table from standard tabular priors into valid causal dataset, illustrated by Figure 4: (i) retrieve base table
5

---

with NN rows from either a large library of tabular data" or synthesize it (details in Appendix D.1); (ii) randomly select columns with a varying number of covariates as X: (ûii) pick two other columns. relabel them as no (X). 1,14 (X); (ir) optionally add zero-mean noise to n (X) and (X) to obtain Yo and Yi, or simply set Yg = (40 (X) and Y1 = M41 (X); these four steps simulate samples from the joint distribution (X, Y)): (v) generate random function f. leveraging similar synthetic functions as in Hollmann et al. [42] to map covariates to their treatment logits; (vi) sample binary treatments T ~ Bernoulli (f
(vit) finally, form the observed outcomes Y ;= YT.
The procedure above "simulates" collection (,(7)x("))
Mo ,440 DGP that can be used to sample the observational data and obtain CEPOs necessary for training (recall Figure 3). This approach guarantees strong ignorability by design: since treatment T's is determined solely from X, it is conditionally independent from the potential outcomes Yo, Y1. Additionally, by applying the sigmoid function, we ensure 0 < P(T = X) < 1, satisfying positivity. While this procedure primarily targets binary treatments, it can naturally extend to finite discrete treatments.
(1)) ((()) yy(")
J==1 from an underlying
For the diversity aspect of T, we rely on the empirical success of existing tabular foundation models and the deliberate design in our generation process. Sampling covariates directly from a mix of real and synthetic tables yields data that is more likely to reflect the scenarios the model will face at inference. We assume no distributional assumptions on covariates and potential outcomes. Appendix D.I details additional mechanisms for controlling treatment effect heterogeneity and positivity in our synthetic DGPs, as well as the detailed configurations of the prior-generation process.
Model Architecture & Parallel Training. We model qo using a PFN-style transformer encoder that receives a sequence of row tokens as context (i.e., Doba), where each token embeds a triplet (("),x()) () At every iteration, we embed BQ batched query tokens (t. x). We then apply 20 layers of self-attention and MLP layers, followed by a final projection layer to get ge x, t, Pobs) ) for all the (t, x) pairs in the batched query. The transformer uses the asymmetric masking used in PFNs: both context and query tokens attend only to the context tokens, ensuring that the predicted CHPO-PPDs are mutually independent.
To model each CEPO-PPD, we approximate it with quantized histogram. We discretize the outcome axis into L = 1024 bins and let the network project the query tokens into logits. We then apply SoftMax to turn the logits into a quantized distribution qo x, l, Dons) [(], Ve E [Z]. At each round of gradient update, we place a Gaussian with a small a at the true CEPO (x(x) and integrate it over bins to obtain Gaussian quantized probabilities N N( []]
and minimize the histogram loss:
HL|u ||
go] = EN(((()) o²) [ log [e.
(10)
E=1
This loss is an approximation to the causal data-prior loss in (9); it coincides in the limit o + 0 and L + 0o. The histogram loss formulation affords tractable proxy for the continuous CEPO-PPD.
A more detailed overview of the architecture and procedures for point and interval estimates is illustrated in Figure 5; further details (c.g., parameter counts, compute, inference-time techniques, number of prior datasets, scalability, and speed) are available in Appendices D.2, D.3, and D.4.
Experiments
Baseline Causal Effect Estimators. We compare to broad suite of baselines. This includes double machine learning (DML) [16, 7, 29], doubly robust learner (DR-Leerrne) [2), as well as the T-,
S- X-, and domain adaptation learner (DA-Learner), all part of the EconML package [11]. Moreover, we include seep-learning-basee methods such as TarNet [104], DragonNet [106], and RA-Net [20], implemented via the CATENets library [19]. Finally, we compare to inverse propensity weighting (IPW) [97], Bayesian regression trees (BART) [40, 15], and generalized random forests (GRF) [7].
All the baselines, except for IPW, provide both CATE and ATE estimates.
Importantly, we tune most of the baselines with cross-validation via grid search. The set of hyperpa- rameter. along with the results with default hyperparaaneters are all detailed in Appendix D.5.
Benchmarks with Ground- Truth Effects. A handful of benchmarks provide ground-trute -tuut causal effects, allowing us to directly measure estimation errors. Given a dataset of N units with covariates
We use 337 OpenML tables [12]. checked to avoid leakage, totaling over 108 feature values.

---

Observational Data
Predicted CEPO and Credible Interval
Y
Context Tokens
Transformer Output
Inference with Sampling
Tokens
SoftMax
Attention Mask
True CEPOS
Batched Queries
No
1
X
Query Tokens
Quantized CEPO-PPDs
Training with Histogram Loss
Adding Gaussian Noise
Figure 5: Architecture, Training, and Inference Details. (Left): An observational data, and a batch of queries along with their true CEPO values are sampled from the prior. Each observational row forms context token, while query tokens consist of only the treatment and covariates. (Middle): The context and query tokens are fed into transformer encoder with an asymmetric attention masking, where both context and query tokens attend only to the context tokens. (Bottom- (Bottom-Right): The output tokens are projected into a 1024- 1224-dimesiioal logit vector and softmaxed to form a discretized CEPO-PPD. Then, the true CEPO value corresponding to cach output token is smoothed hy adding nidth Gaussian, and training is done by minimizing the cross-entropy (histogram) loss. (Top-Right): At inference time, the CEPO-PPD mean is used as the point estimate.
Table I: CATE & ATE results. Columns correspond to benchmark suites: IHDP, ACIC 2016, Lalonde cPs/sIn. (left half) mean PEHE and the average rank when pooling all tasks. (right half) mean ATE relative errrr and its average across all tasks. Lalonde PEHE is in thousands. The best and second best columns are highlighted.
Cells with "_" indicate that the method is applicable.
Meun PRIE Standard Error (1 better;
Relutive Error Standard Rroor (1 better)
Method
HHDP
ACIC 2016
Lalonde cs
Lalondo so (x10")
Avg.
IHDP
ACIC 2016 Lalonde os
Lalendo im
Avg.
(x100)
Rank
Rank
CausalPEN
0.58-0.07
0.92=00111
14.40-0.20 8.96
0.20
0.100.11
0.20=0.04
0.05:0.01
0.13:-0.11
0.22=0002
4.4520.19
L.3-0.30
0.76+0.07
0.22+0.04
15.16-0.46
0.21+0004
003+0.01
0.34+0022
4.31+0.18
DA-Leamer
2.07=0.36
0.72:10.8
9.3910.06
14.55
0.24
3.60:20.16
0.23::0.04
0.03:0001
0.27:0.02
0.20::0.03
4.3320.19
DragonNet
2.162:025
2.11
10.9310.15
16.45-0.29
5.9910.18
0.2011:04
0.0610.02
0.5510.03
0.4710.03
6.2610.17
IPW
0.24:20.4
0.21:0:05
0.17:0001
0.10::001
4.41::0.21
RA-Neet
2.35+0.19
2.35=0:25
11.74+0.09
18.33-0.43
7,15-0016
0.20==004
0.07+003
0,74+0022
0.50=0004
6.78+0.17
X-Leureer
331-00.51
0.60-0.08
12.15:4..15
20.28-0.49
7.46:10.19
0.16:00.4
0.03-+0.01
0.84:0033
0.72-0003
7.31
TaNNet
1822:0.14
2.20=0021
12.8810.02
19.19-0.18
8.38.10.4
0.20
0.0510.02
1.0010.00
0.781:0.01
8.8310.15
S-Leamer
2.57:0.4
0.85=0.13
12.661:0.05
21.80=0.18
8.11:10.18
0.20:0004
0.03:00.1
0.9710.01
0.90:0002
8.8510.18
BART
2.50==0.39
0.68-0.11
12.81=0005
31.36=0,16
8.55==0.16
0.44=00.9
0.04:0001
0.99:0.01
0.86::0.11
8.99:0.18
3667+0.61
1.32-0030
12.33+00.6
22.91-0.17
8.82+-018
0.18+0003
0.0700022
0,820002
0.85-0022
8.02+0.18
Forest DML
4.53:10.73
1.4810.31
12.95110.04
12.99=0.15
9.83:10.17
0.08 0..0
0.05:0.01
10330001
1.05
9.6010.21
Forest Learner
4.0220067
1.3420.29
15.9810.68
22.78_0.54
10.00:0.117
0.17
0.0110.02
1.2010.23
3.6422.78
8.3810.18
and ground-truth CATE values (x((1, 7 ((x())))) >= and ground-truth ATE X. evaluate models
using the relative ATE error and the precision in estimation of heterogeneous effects (PEHE) [40]:
N
(A)
PEHE
Σ (T(x(m) - =
(11)
N
Here, f and Å denote the estimated CATE and ATE, respectively. Table 1 compares CausalPFN to all baselines on four standard set of datasets: 100 realizations of IHDP [94, 40]. 10 realizations of ACIC 2016 [27], and the Lalonde cps and Lalonde rSID cohorts [58] with their causal effects provided by RealCause (each with 100 realizations) [8]]. Our model demonstrates superior performance on both CATE and ATE tasks, remaining within the top models across most benchmarks. To assess the overall performance of each method for CATE estimation, we calculate the average rank of each method across all 310 realizations based on PEHE. For ATEs, calculate the average rank of each method based on relative errors. CausalPFN outperforms all basclines in terms of average CATE rank, while being competitive for average ATE rank. Notably, our model is trained entirely on simulated data and never sees the evaluation data during pre-training. While some baseline estimators in Table perform well on specific datasets, they underperform others. In contrast, the consistent performance of CausalPFN suggests that amortized approaches can potentially eliminate the manual burden of task-specific estimator design.
Policy Evaluation on Marketing Randomized Trials. Ground-truth CATES are only available for synthetic or semi-synthetic datasets. However, if randomized controlled trial (RCT) is available, we
7

---

Table 2: Normalized Qini scores (1 better). All datasets 50k stratified subsamples, except Hill( and Hill which use the full 64k rows. Columns are normalized to 1.0 for the best model.
Qini Curve (Q(q))
O)ll
Hil
0.5
CausaIFNN
Method
Hill
Hill (2)
Criteo
X5
Lenta
Mega
Avg.
X Leamer
Random
CausalPFN
0.992
0.968
0.859
0.922
1.000
0.970
0.952
0.0
X Learner
0.975
0,980
1.000
0.937
0.771
1.000
0.944
S Learner
1.000
1.000
0.881
1.000
0.651
0,941
0.912
0.00
0.25
0.50
0.75
1.00
DA Learner
0.985
0.964
0.626
0.929
0.781
0.998
0.881
Treated Fraction (q)
T Learner
0.991
0.972
0.701
0.964
0.644
0.986
0.876
Figure 6: (i & (ill Qini curves.
can still evaluate the quality of CATE estimator by assessing the performance of policies derived from it. A common tool for evaluating such policies is the Qini curve [93], which plots the cumulative treatment effect when units are ranked in descending order of their predicted CATE.
Formally, let (y((), corresponding CATE estimates, ordered so that fi 2 2 FN. Define
denote outcomes and binary treatments from an RCT, and let in be the
LgN]
\(q)= = Σ
(m) r(g)
(1-1"")(("))
Q = Xgg)A(L))
0S9SL,
(12)
where r(4) 1gN] Σ [ {(m) is the empirical treatment rate for the first g-quantile of units. Because
the data comes from an RCT, X(q) unbiasedly estimates the ATE for the top q-quantile of units ranked by predicted CATEs. Plotting ((z) against the treated fraction yields the (normalized) Qini curve, and the area under this curve is called the Qini score. A random ranking produces baseline curve as a straight line from (0, 0) to (1, 1). The higher the Qini curve lies above this line, the better the model prioritizes high-impact units with larger CATE values, leading greater lift and policy benefit.
We benchmark CausalPFN on five large marketing RCTs from the scikit-uplift library [74]. The first dataset, Hillstrom [41], includes 64,000 customers randomly assigned to one of three treatments: no e-mail, an e- e-mail advertising men's 's merchandise, or an e-mail advertising women's 's merchandise. The outcome is whether a website visit occurred within two weeks (binary). We consider two causal tasks: Hill Men's m-mrrchandise e-mail (treatment) vs. no e-mail (control), and (2ll Women's 's m-mrrchandise e-mail vs. no e-mail. We estimate CATES using CausalPFN (five-fold honest splitting) and X Learner. Figure 6 shows Qini curves where CausalPFN closely matches X Learner across the targeting range. Notably, Hill(2) shows much greater gains, suggesting focusing on women's s-merchandise ad campaigns, compared to men't 's can drive more gains in the number of website visits. We also evaluate CausalPFN on four larger campaigns-Lentts (X5), Megafon (Mega), and Criteo [61, 95, 78, 1 with 100 rows. For tractability, we compute Qini scores on stratified 50k subsamples; Table 2 shows CausalPFN achieves the best mean performance. However, when we run it on full tables (see Table 7 of Appendix D.6), we observe a drop in performance, which aligns with known context-length limitations of PFN-style transformers on large tables [109]. Still, the strong subsample results highlight the potential of scaling CausalPFN to longer contexts, which remains important future direction.
Retail Hero
Uncertainty & Calibration. Recall from Section 3 that for each unit covariate x, CausalPFN can produce both point estimates and credible intervals for the CATE and CEPOs. We do so by drawing 10,000 samples from the quantized distributions qo( (- x, l, Pnbs) and construct credible intervals at any desired significance level a. Here, we evaluate these intervals, focusing on the model's 's calibration. We also assess a key assumption from Proposition 1-whether the inference-time DGP ui* lies within the support of the prior æ, and how the model behaves when this assumption is violated.
We define families of synthetic DGPs simulate both in-distribution and out- -iittri-iitiittiinn (OOD) scenarios. Each DGP samples covariates x from a uniform distribution, defines treatment logit func- tion f and CEPO functions b4 for E {0, 1}. assigns treatment via T CernouliiSigmood (f (x))). and generates potential outcomes as y (x) + c, where c is drawn from a standard Uniform, Gaussian, or Laplace We consider two DGP families; Sinusoidal, where I and μ4, are functions with sinusoidal components, and Polynomial, where the functions ʃ and μ4 are polynomials of varying degree (see Appendix D.7 for detailed configurations). CausalPFN trained either on the same family it is tested on, different one (OOD).
8

---

Polynomial Model
Sinusodial Model
In Distribution
Ou-of-Distriuttion
Bayesian Coverage (cov: (xx)
1.0
0.4
Model-DGP Pairs
Bayesiun Coverage (cov (a))
1.0
0.8
ICE, ICE, Line
0.8
0.2
ICR
0.0
0.6
0.4
0.4
0.2
0.2
0.2
0.4
0.0
0.0
0.0
0.5
1.0
-0.50
-0.25
0.00
0.25
0.3
1.0
Credibility Level (1 a)
ICE,
Credibility Level (1
x))
Figure 7: Calibration. (Left): CATE coverage vs. nominal credibility. In-distribution DGPs (bluc) lie on or above while OOD DGPs (orange) fall below it (overconfident). (Middle): Across model- DGP pairs, CATE ICE (x-axis) exceeds regression ICE (x-axis). (Right): Temperature scaling based on
the diagonal (calibrated/conservatvee
regression ICE ensures the model is either calibrated or conservative for both in- and out-of-distribttion DGPs.
For a unit with covariates x and significance level a, we say the true CATE is covered (xx) lies within the predicted 100(1 a)% interval obtained using samples from qe- Plotting Bayesian coverage against nominal levels of a yields the CATE calibration curve. As shown in Figure 7 (left), CausalPFN is reliably calibrated under in-distribution settings but becomes severely overconfident when evaluated on OOD DGPs (20* d π). This aligns with prior observations that neural models often exhibit pathological overconfidence under distribution shift [35, 86].
To correct this, we apply temperature parameter Or to the SoftMax that outputs the quantized CEPO-PPD from the logits of the model. We aim to tune Or to minimize the calibration error. However, direct CATE calibration impossible because T(x) is never observed at test-time. Instead, we introduce the regression calibration based on observational data: an observed triple (t, x,, v) covered by the predicted credible interval when y lies inside the model's predicted interval for the CEPO-PPD [x(x; *²") With that in mind, we let (ox) and cov-(a) denote the Bayesian coverage at level o for the regression and CATE calibration curves, respectively, and define
ICE, L." (cov a) da, and ICE, //" (cov, (a) - a) da, (13)
the integrated coverage error (ICE) for regression and CATE (negative values overconfidence).
Note that we do not expect COV, to calibrated: regression intervals combine epistemic uncertainty of the CEPO with the irreducible (aleatoric) noise in Y, so ICE, is biased. Still, it holds useful signal. Across all model-DGi pairs in Figure 7 (middle), we consistently observe ICE, S ICE,: the regression curve sits at below the CATE curve. While ICE, is inaccessible without having the
the true CATE, ICE,, is computable from observational data. Consequently, cumperature-scalinn logits lift , to the diagonal also calibrates the CATE intervals makes them conservative. We thus tune 8 by grid search to drive ICE,, to zero using a 5-fold calibration on the observational data. The calibrated curves in Figure 7 (right) confirm that, after temperature scaling, CausalPFN's overconfidence on the OOD test-sets disappears. Additional synthetic train-/test-DGP pairs and real-world data experiments appear in Appendix D.7.
Comparison to TabPEN. We also compare against the latest version of TabPFN [43], plugging its regression output proxy for CEPO. As Table 3 shows, TabPFN is surprisingly competitive without any causal tuning, yet CausalPFN outperforms on every benchmark except ACIC 2016. To isolate the benefit of training TabPFN, we fine-tune it on our prior for 48 hours on an H100 GPU. This causal fine-tuning boosts the performance and confirms the added value of identifiable priors for causal effect estimation.
causal prior, compared to the predictive non-identifiable prior in
6 Related Work
Single-Dataset Estimators. Common methods for causal effect estimation are trained and applied on a single dataset. Representative examples include the X-, S-, DR-, and RA-Learners, as well as IPW and DML. [1]]. Alongside these approaches, several neural variants such as TARNet [104], DragonNet [106]. CEVAE [68], and NCMs [114, 115] have been proposed; however, all of them still require dar-datasct training and do not amortize across various datasets.

---

Table 3: TabPFN Comparison. PEHE (left half) alongside ATE relative error (right half). TabPFN' is the latest TabPEN model [43] tuned with our prior. Best numbers are highlighted.
PEHE + Standard Error better)
ATE Relative Error =
Standard Error (+ better)
Method
IHDP
2016
Lalonde crs Lalonde rum
IHDP
ACIC 2016 Lalonde crs Lalonde esm
(x10")
(x10")
CausalPEN (Ours)
0.58=0.07
0,92+0.11
8.96=0.02
14.40=0.20
0.20
0.05+1.001
0.13::0.01
0. 0.22:1:0.2
TabPEN (Ours)
0.90=0.16
0.47:-0.05
8.97=0.06
14,90=0.95
0.21
0.03-10.01
0.17::0.02
0.22-10.08
TabPFN
0.95
0.5410.08
9.45=0.19
18.7=0.83
0.21
0.03:0.01
0.32+10.05
0.60:10.07
Amortized Causal Inference. Amortized methods train a single network that maps observational data to causal quantities across multiple DGPs, Existing approaches fall into two groups: (i) methods that first recover a causal graph from observational data and then compute interventions on that graph [102, 72]. following ideas from causal discovery [88, 121, 53, 67, 51, 50]; and (ii) methods that infer causal effects end-to-end [83, 120, 14]. Amortization has also been explored in decision-making. where the aim is to learn policies that generalize across environments or tasks [60, 59]. While closely related, none of these methods provides a ready-to-use estimator that consistently surpasses specialized single-dataset estimators on standard benchmarks. In contrast, our method is trained once and produces causal effects without any access to or adaptation on the test-time DGPs. Through large-scale training, CausalPFN delivers out-of-the-box performance that exceeds specialized single- dataset estimators. Recently, concurrent work by Robertson et al. [96] also applies PFNs to causal effect estimation but lacks a procedure to guarantee the identifiability of the prior data; additionally. we observe relatively poor empirical performance compared to CausalPFN. For further discussion and comparison with this method, refer to Appendix E.
Scaling In-Context Transformers. In-context learning with transformers has shown impressive results across range of domains [13, 116, 18, 25, 110]. Although the underlying mechanisms
responsible for this success remain an active area of research [1, 23, 85, 111, 63, 117, 1 112, 8, 90], increasing model size and training data have consistently and undoubtedly led stronger performance. This success has recently extended to tabular prediction with models such as TabPFN [42, 43), TabDPT [69]. and TabICL [92], which are trained on broad prior distributions and perform well on real-world data without fine-tuning. CausalPFN complements these works, demonstrating with sufficient scale and training- meining-in-conten in-context learning can also be effectively adapted causal inference.
7 Conclusions, Limitations, and Future Work
In this paper, we introduced a practical paradigm for amortized causal effect estimation that combines Bayesian causal inference with large-scale tabular training. Despite learning solely from simulated data, CausalPFN matches, and often outperforms, specialized causal estimators across diverse real- world domains. Through amortization, we significantly reduce the burden of estimator selection at inference time, and to foster adoption, we have open-sourcee the code and presets.
That said, several important limitations remain: (i) Our approach fundamentally assumes strong ignorability, which is untestable assumption in practice. Without this condition, CausalPFN has no guarantees of validity. Domain expertise still remains essential to determine whether this method is appropriate or whether alternative approaches should be used. (ii) Our theoretical guarantees rely on idealistic assumptions: a well-specified prior and asymptotically large datasets. We lack finite-sample theory characterizing the estimator's behavior in practical settings. Investigating robustness to prior misspecification and developing finite-sampe sample guarantees remain open problems. Recent work on theory of valid adjustment sets [17] may offer promising directions for addressing these challenges. (iii) Performance degradation is evident on the largest marketing tables (Table 7), reflective of the known size-scalability trade-off inherent to PFN-style models [43]. (iv) While CausalPFN already supports multi-arm discrete treatments with a finite set T, we have only implemented it for the binary T. Additionally, extending to the continuous treatment setting where T not finite remains fully unexplored. (v) Finally, our entire implementation relies on the strong ignorability or backdoor assumption. Extending our framework to richer
priors like instrumental variables can broaden the framework's 's reach, although designing scalable priors for such cases is non-tri non-trivial.
10

---

Acknowledgements
We would like to thank Mouloud Belbahri for his suggestions regarding the uplift modelling ex- periments. RGK gratefully acknowledges support from the Canada Research Chairs Program (CRC-2022-00049) and the Canada CIFAR Al Chairs Program, This research was funded in part by NFRF Special Call Award (NFRER-2022-00S2) and NSERC Discovery Grant (RGPIN-2022-04446), Resources used in preparing this research were provided, in part, by the Province of Ontario, the Govermment of Canada through CIFAR, and companies sponsoring the Vector Institute.
References
[1] Ekin Akyürek, Dale Schuurmans, Jacob Andreas, Tengyu Ma, and Denny Zhou. What learning algorithm is in-context learning? investigations with linear models. In The Eleventh International Conference on Learning Representations, 2023.
[2] Ahmed Alaa and Mihaela Van Der Schaar. Validating causal inference models via influence functions. In International Conference on Machine Learning, pages 191-201. PMLR, 2019.
[3] Ahmed M. Alaa and Mihaela van der Schaar. Bayesian inference of individualized treatment effects using multi-task gaussian processes. In Advances Neural Information Processing Systems, volume 30, 2017.
[4] Christophe Andrieu, Nando De Freitas, Arnaud Doucet, and Michael I Jordan. An introduction to MCMC for machine learning. Machine Learning, 50:5 43, 2003.
[5] Joshua Angrist and Jörn-Steffen Pischke. Mastering 'Metrics: The path from cause to effect. Princeton University Press, 2014.
[6] Joshua D Angrist, Guido W Imbens, and Donald B Rubin. Identification of causal effects using instrumental variables. Journal of the American statistical Association, 91(434):444 455, 1996.
[7] Susan Athey, Julie Tibshirani, and Stefan Wager. Generalized random forests. The Annals of Statistics, 47(2): 148-1178, 2019. doi: 10. 11214/8-AAS17700
[8] Yu Bai, Fan Chen, Huan Wang. Caiming Xiong, and Song Mei. Transformers as statisticians: Provable in-context learning with in-context algorithm selection. In Advances in Neural Information Processing Systems, volume 36, pages 77125-57211, 2023.
[9] Vahid Balazadeh, Keertana Chidambaram, Viet Nguyen, Rahul G Krishnan, and Vasilis Syrgka- nis. Sequential decision making with expert demonstrations under unobserved heterogeneity. Advances in Neural Information Processing Systems, 37:65476-65444 65498, 2024.
[10] Alexander Balke and Judea Pearl. Bounds on treatment effects from studies with imperfect compliance. Journal of the American Statistical Association, 1771-1177 1997.
[11] Keith Battocchi, Eleanor Dillon, Maggie Hei, Greg Lewis, Paul Oka. Miruna Oprescu, and Vasilis Syrgkanis. EconML: A Python Package for ML-Based Heterogeneous Treatment 2019. Version 0.15.0.
Effects Estimation. http://githbb..om///xwwwyf/oooot
[12] Bernd Bischl, Giuseppe Casalicchio, Matthias Feurer, Pieter Gijsbers, Frank Hutter, Michel Lang, Rafael Gomes Mantovani, Jan van Rijn, and Joaquin Vanschoren. OpenML benchmark- ing suites. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, 2021.
[13] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry. Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. Advances in Neural Information Processing Systems, volume 33, pages 1877-1901, 2020.
11

---

[14] Lucius EJ Bynum, Aahlad Manas Puli, Diego Perrero-Quevedo Nhi Nguyen, Carlos Seraandz--Grandae Kyunghyun Cho, and Rajesh Ranganath. Black box causal inference: Effect estimation via meta prediction. arXiv:2503.05988 2025.
[15] Stephanie Chan, Adam Santoro, Andrew Lampinen, Jane Wang, Aaditya Singh, Pierre Richemond, James McClelland, and Felix Hill. Data distributional properties drive emergent in-context learning in transformers. In Advances in Neural Information Processing Systems, volume 35, 2022.
[16] Victor Chernozhukov, Denis Chetverikov. Mert Demirer, Esther Duflo, Christian Hansen, Whitney Newey, and James Robins. Double/debiased machine learning for treatment and structural parameters. The Econometrics Jourmal, 21(1), 2018. doi: 10.1
[17] Davin Choo, Chandler Squires, Arnab Bhattacharyya, and David Sontag. Probably approx- imately correct high- sigh-dimessional causal effect estimation given a valid adjustment set. In Proceedings of the Fourth Conference Causal Learning and Reasoning, volume 275 of Proceedings of Machine Learning Research, 2025.
[18] Julian Foda-Forno, Marcel Binz, Zeynep Akata, Matt Botvinick, Jane Wang, and Eric Schulz. Meta-in-context learning in large language models. Advances Neural Information Processing Systems, 36:65188 189 89-65201, 2023.
[19] Alicia Curth. CATENets: Sklearn-style Implementations of Neural Network-based Conditional Average Treatment Effect (CATE) Estimators. https://github. http://tttuuu..omAAmt Coom/AAlcciCCttt/CCTEEEEe 2021. GitHub repository, commit 821bfb0. Accessed: 2025-05-11.
[20] Alicia Curth and Mihaela der Schaar. On inductive biases for heterogeneous treatment effect estimation. In Advances in Neural Information Processing Systems, volume 34, pages 15883-15894, 2021.
[21] Alicia Curth and Mihaela van der Schaar. Nonparametric estimation of heterogeneous treat- ment effects: From theory to learning algorithms. In Proceedings of the 24th International Conference on Artificial Intelligence and Statistics (AISTATS). PMLR, 2021.
[22] Alicia Curth, David Svensson, James Weatherall, and Mihaela van der Schaar. Really Doing Great Estimating CATE? A Critical Look at ML Benchmarking Practices in Treatment Effect Estimation. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, volume 1, 2021.
[23] Damai Dai, Yutao Sun, Li Dong, Yaru Hao, Shuming Ma, Zhifang Sui, and Furu Wei. Why Can GPT Learn In-Context? Language Models Secretly Perform Gradient De- scent as Meta-Optimizers. In Findings of the Association for Computational Linguis- tics: ACL 2023, pages 4005-4015 Association for Computational Linguistics, 2023. doi: 10. 11111/6/1/2222..llg 2.2.2222...nninng-a1....7.
[24] Aaron Defazio, Xingyu Yang, Ahmed Khaled, Konstantin Mishchenko, Harsh Mehta, and Ashok Cutkosky. The road less scheduled. Advances in Neural Information Processing Systems, 1119977--1007,,0 2024.
[25] Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Jingyuan Ma, Rui Li, Heming Xia, Jingjing Xu, Zhiyong Wu, Baobao Chang, al. survey on in-context learning. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 1107-1128, 2024.
[26] Joseph L Doob. Application of the theory of martingales. Le calcul des probabilites et ses applications, pages 23-27, 1949.
[27] Vincent Dorie, Jennifer Hill, Uri Shalit, Mare Scott, and Dan Cervone. Automated versus do-it-yourself methods for causal inference: Lessons learned from data analysis competition. Statistical Science. 34(1):43-68, 2019.
[28] Sebastian Felix Fischer, Matthias Feurer, and Bernd Bischl. OpenMI-CTRR2 curated tabular regression benchmarking suite. In AutoML Conference (Workshop), 2023,
12

---

[29] Dylan J Foster and Vasilis Syrgkanis. Orthogonal statistical learning. The Annals of Statistics, 51(3):879-908, 2023.
[30] Michele Jonsson Funk, Daniel Westreich, Chris Wiesen, Til Stürmer, M Alan Brookhart, and Marie Davidian. Doubly robust estimation of causal effects. American Journal of Epidemiology. 173(7):761-677 2011.
[31] Marta Garnelo, Dan Rosenbaum, Christopher Maddison, Tiago Ramalho, David Saxton, Murray Shanahan, Yee Whye Teh, Danilo Rezende, and S. M. Ali Eslami. Conditional neural processes. In Proceedings of the 35th International Conference on Machine Learning. volume 80, pages
1713, 2018.
[32] Marta Garnelo, Jonathan Schwarz, Dan Rosenbaum, Fabio Viola, Danilo J Rezende, SM Es- lami, and Yee Whye Teh. Neural processes. 18X.008007.0162.,, 2018.
[33] Pieter Gijsbers, Marcos LP Bueno, Stefan Coors, Erin LeDell, Sébastien Poirier, Janek Thomas, Bernd Bischl, and Joaquin Vanschoren. AMLB: An AutoML benchmark. Journal of Machine Learning Research, (510)))11-65, 2024.
[34] Léo Grinsztajn, Edouard Oyallon, and Gaël Varoquaux. Why do tree-based models still out- perform deep learning on typical tabular data? In Advances in Neural Information Processing Systems, 2022.
[35] Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International Conference on Machine Learning, pages 1321-1330. PMLR, 2017.
[36] P Richard Hahn, Jared Š Murray, and Carlos Carvalho. Bayesian regression tree models for causal inference: Regularization, confounding, and heterogencous effects (with discussion). Bayesian Analysis, -555 2020.
[37] Kai Helli, David Schnurr, Noah Hollmann, Samuel Müller, and Frank Hutter. Drift-resilient TabPFN: In-context learning temporal distribution shifts on tabular data. Advances in Neural Information Processing Systems, 37:98742-98781. 2024.
[38] Alex Henry, Prudhvi Raj Dachapally, Shubham Shantaram Pawar, and Yuxuan Chen. Query- key normalization for transformers. In Findings of the Association for Computational Linguis- tics: EMNLP 2020, pages 4246-4253 Association for Computational Linguistics, November 2020. doi: 2..18653/v1/2221 http://........nseeeeeeee findings-emmll.379.
[39] M.A. Hernan and J.M. Robins. Causal Inference: What If. Chapman & Hall/CRC monographs on statistics & applied probability. Taylor Francis, 2023. ISBN 9781315374932.
[40] Jennifer L Hill. Bayesian nonparametric modeling for causal inference. Journal of Computa- tional and Graphical Statistics, 20( 1::11722,, 2011.
[41] Kevin Hillstrom.
Minethatdata e-mail analytics and data mining challenge dataset. www.mmmttttttaacooo.... htt.aa/aa.....oaooooo/o////aaaaa -al-----illaaaalll---dd-d----dd html, 2008 Accessed: 2025-05-11.
[42] Noah Hollmann, Samuel Müller, Katharina Eggensperger, and Frank Hutter. TabPFN: A transformer that solves small tabular classification problems in a second. In The Eleventh International Conference on Learning Representations, 2023.
[43] Noah Hollmann, Samuel Müller, Lennart Purucker, Arjun Krishnakumar, Max Körfer, Shi Bin Hoo, Robin Tibor Schirmeesterr, and Frank Hutter. Accurate predictions small data with tabular foundation model. Nature, 63778005) 2025.
[44] Ehsan Imani and Martha White. Improving regression performance with distributional losses.
In International conference machine learning, pages 2157-2166. PMLR, 2018.
[45] Guido W. Imbens and Joshua D. Angrist. Identification and estimation of local average treatment effects. Econometrica, 62(2):467-475 1994. ISSN 00129682, 14680262.
[46] Guido Imbens and Donald B Rubin. Bayesian inference for causal effects in randomized experiments with noncompliance. The Annals of Statistics, pages 305-327, 1997.
13

---

[47] Guido W Imbens and Donald B Rubin. Causal Inference in Statistics, Social, and Biomedical Sciences. Cambridge University Press, 2015.
[48] Andrew Jesson, Sören Mindermann, Uri Shalit, and Yarin Gal. Identifying causal-effect inference failure with uncertainty-aware models. Advances in Neural Information Processing Systems, 33:1 1637-11649, 2020.
[49] Michael I Jordan, Zoubin Ghahramani, Tommi S Jaakkola, and Lawrence K Saul. An introduction to variational methods for graphical models. Machine Learning, 37:183-233, 1999.
[50] Hamidreza Kamkari, Vahid Balazadeh, Vahid Zehtab, and Rahul G Krishnan. Order-based structure learning with normalizing flows. arXiv:2308.0748) 2023.
[51] Nan Rosemary Ke, Silvia Chiappa, Jane X Wang, Jorg Bornschein, Anirudh Goyal, Melanie Rey, Theophane Weber, Matthew Botvinick, Michael Curtis Mozer, and Danilo Jimenez Rezende. Learning induce causal structure. In International Conference on Learning Representations, 2023.
[52] Edward H. Kennedy. Towards optimal doubly robust estimation of heterogeneous causal effects. Electronic Journal of Statistics, 17(2):3008-3049, 2023.
[53] Ilyes Khemakhem, Ricardo Monti, Robert Leech, and Aapo Hyvarinen. Causal autoregressive flows. In International Conference on Artificial Antelligence and Statistics, pages 3520-3523 PMLR, 2021.
[54] Hyunjik Kim, Andriy Mnih, Jonathan Schwarz, Marta Garnelo, Ali Eslami, Dan Rosenbaum, Oriol Vinyals, and Yee Whye Teh. Attentive neural processes. In International Conference on Learning Representations, 2019.
[55] Diederik P Kingma. Adam: A method for stochastic optimization.
arXiv preprint
arXiv:1412.6980, 2014.
[56] Andrei Konstantinov, Stanislav Kirpichenko, and Lev Utkin. Heterogeneous treatment effect with trained kernels of the nadaraya watson regression. Algorithms,, 16(5):226, 2023.
[57] Sören R. Künzel, Jasjeet Sekhon, Peter Bickel, and Bin Yu. Metalearners for estimating heterogeneous treatment effects using machine learning. Proceedings of the National Academy of Sciences, 116(10)4456-44155 2019.
[58] Robert J LaLonde. Evaluating the econometric evaluations of training programs with experi- mental data. The American Economic Review, pages 604 620, 1986.
[59] Allison Lau, Younwoo Choi, Vahid Balazadeh, Keertana Chidambaram, Vasilis Syrgkanis, and Rahul G Krishnan. Personalized adaptation via in-context preference learning. arXiv preprint arXiv:2410. 14001, 2024.
[60] Jonathan Lee, Annie Xie, Aldo Pacchiano. Yash Chandak, Chelsea Finn, Ofir Nachum, and Emma Brunskill. Supervised pretraining can learn in-context reinforcement learning. Advances in Neural Information Processing Systems,
2023.
[61] Lenta LLC. Lenta uplift dataset. http:///gtaabbbcommmmmmmsssssshp 2020. Accessed: 2025-05-11.
[62] Fan Li, Peng Ding, and Fabrizia Mealli. Bayesian causal inference: a critical review, Philo- sophical Transactions of the Royal Society A, 3811(22477222221155, 2023.
[63] Yingcong Li, Muhammed Emrullah Ildiz, Dimitris Papailiopoulos, and Samet Oymak. Trans- formers as algorithms: Generalization and stability in in-context learning. In International Conference on Machine Learning, pages 19565-19585 19594. PMLR, 2023.
[64] Antonio R Linero and Joseph I. Antonelli. The how and why of bayesian nonparametric causal inference. Wiley Interdisciplinary Reviews: Computational Statisties, 15(1):e1883, 2023.
14

---

[65] Manqing Liu, David R Bellamy, and Andrew L Beam. DAG-aware transformer for causal effect estimation. arXiv:2410.1004 10044, 2024.
[66] Siyang Liu and Han-Jia Ye. TabPEN Unleashed: A Scalable and Effective Solution to Tabular Classification Problems. In Forty-second International Conference on Machine Learning. 2025.
[67] Lars Lorch, Scott Sussex, Jonas Rothfuss, Andreas Krause, and Bernhard Schölkopf. Amor- tized inference for causal structure learning. Advances in Neural Information Processing Systems, 35:13104-133 131188, 2022.
[68] Christos Louizos, Uri Shalit, Joris Mooij, David Sontag, Richard Zemel, and Max Welling. Causal effect inference with deep attet-varrable models. In Advances in Neural Information Processing Systems, volume 30, 2017.
[69] Junwei Ma, Valentin Thomas, Rasa Hosseinzadeh, Hamidreza Kamkari, Alex Labach, Jesse Cresswell, Keyvan Golestan, Guangwei Yu, Anthony I. Caterini, and Maksims Volkovs. TabDPT: Scaling Tabular Foundation Models on Real Data. In Advances in Neural Information Processing Systems, volume 38, 2025.
[70] Yuchen Ma. Valentyn Melnychuk, Jonas Schweisthal, and Stefan Feuerriegel. DiffPO: A causal diffusion model for learning distributions of potential outcomes. Advances Neural Information Processing Systems, volume 37, pages 43663 2024.
[71] David P MacKinnon, Amanda J Fairchild, and Matthew S Fritz. Mediation analysis. Annu. Rev. Psychol. 58(1):593-614, 2007.
[72] Divyat Mahajan, Jannes Gladrow, Agrin Hilmkil, Cheng Zhang, and Meyer Scethon. Zero-shot learning of causal models. arXiv:2410.06128, 2024.
[73] Divyat Mahajan, Ioannis Mitliagkas, Brady Neal, and Vasilis Syrgkanis. Empirical analysis of model selection for heterogeneous causal effect estimation. In The Twelfth International Conference on Learning Representations, 2024.
[74] Irina Elisova Maksim Shevchenko. User guide for uplift modeling and casual inference. http://www..pplit-mmodiiggg httpg////////eeesuuuueeeeieiiite
2020.
[75] Charles Manski. Identification problems in the social sciences. Sociological methodology, pages 1-56, 1993.
[76] Calvin McCarter. What exactly has TabPFN learned to do? In The Third Blogpost Track at ICLR 2024, 2024.
[77] Duncan McElfresh, Sujay Khandagale, Jonathan Valverde, Vishak Prasad C, Ganesh Ramakr- ishnan, Micah Goldblum, and Colin White. When do neural nets outperform boosted trees on tabular data? In Advances Neural Information Processing Systems, 2023.
[78] Megafon PJSC, Megafon uplift dataset. https///ggttubbooommmmmmmsssssss 2020. Ac- cessed: 2025-05-11.
[79] Jeffrey W Miller. A detailed treatment of Doob's theorem. arXiv: 2011
[80] Samuel Müller, Noah Hollmann, Sebastian Pineda Arango, Josif Grabocka, and Frank Hut- ter. Transformers Can Do Bayesian Inference. In International Conference on Learning Representations, 2022.
[81] Brady Neal, Chin-Wei Huang, and Sunand Raghupathi. RealCause: Realistic causal inference benchmarking. arXiv:2011. 15000 2020.
[82] Radford M Neal. Bayesian learning for neural networks, volume 118. Springer Science Business Media, 2012.
[83] Hamed Nilforoshan, Michael Moor, Yusuf Roohani, Yining Chen, Anja Surina, Michihiro Yasunaga, Sara Oblak, and Jure Leskovee. Zero-shot causal learning. In Advances Neural Information Processing Systems, volume 36, pages 6862-6901, 2023.
15

---

[84] Arman Oganisian and Jason A Roy. A practical introduction to bayesian estimation of causal effects: Parametric and nonparametric approaches. Statisties in Medicine, 40(2))518-55 2021.
[85] Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hatfield- -oddds, Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. In-context learning and induction heads. arXiv:2209.1 1895, 2022.
[86] Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, D. Sculley, Sebastian Nowozin, Joshua Dillon, Balaji Lakshminarayanan, and Jasper Snock. Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift. In Advances in Neural Information Processing Systems, volume 32, 2019.
[87] Judea Pearl. Causality. Cambridge University Press, 2009.
[88] Jonas Peters, Joris M Mooij, Dominik Janzing, and Bernhard Schölkopf. Causal discovery with continuous additive noise models. The Journal of Machine Learning Research, 15(1): 2009-2053, 2014.
[89] Jonas Peters, Dominik Janzing, and Bernhard Schölkopf. Elements of causal inference: Foundations and learning algorithms. The MIT Press, 2017.
[90] Maxime Peyrard and Kyunghyun Cho. Meta-statistical learning: Supervised learning of statistical inference. arXiv:2502. (tti::5522..2888.. /2088, 2025.
[91] Chris Preston. A note standard Borel and related spaces. Journal of Contemporary Mathematical Analysis, 44(1):63-7,, 2009.
[92] Jingang Qu David Holzmüller, Gaäl Varoquaux, and Marine Le Morvan. TabICL: A Tabular Foundation Model for In-Context Learning on Large Data. In Forty-second International Conference on Machine Learning, 2025.
[93] Nicholas Radcliffe. Using control groups to target on predicted lift: Building and assessing uplift models. Technical report, Stochastic Solutions, 2007.
[94] Craig T Ramey, Donna M Bryant, Barbara H Wasik, Joseph Sparling, Kaye Fendt, and Lisa M La Vange. Infant health and development program for low birth weight, premature infants: Program elements, family participation, and child intelligence. Pediatrics, 89(3): 454-465, 1992.
[95] Retail Hero. Retail hero (x5) uplift dataset. http://gtttbb..oommmanaassslkkkh 2020. Accessed: 2025-05-11.
[96] Jake Robertson, Arik Reuter, Siyuan Guo, Noah Hollmann, Frank Hutter, and Bernhard Schölkopf. Do-pfn: In-context learning for causal effect estimation. arXiv preprint urXiv:2506.06034 2025.
[97] Paul R Rosenbaum and Donald B Rubin. The central role of the propensity score in observa- tional studies for causal effects. Biometrika, 70(1):41-55 1983.
[98] Donald Rubin. Estimating causal effects of treatments randomized and nonrandomized studies. Journal of Educational Psychology. 66(5):688, 1974.
[99] Donald B Rubin. Bayesian inference for causal effects: The role of randomization. The Annals of Statistics, pages 34-58, 1978.
[100] Donald B Rubin. Causal inference using potential outcomes: Design, modeling, decisions.
Journal of the American Statistical Association, 100(469):322-331) 2005.
[101] Neela Sawant, Chitti Babu Namballa, Narayanan Sadagopan, and Houssam Nassif, Contextual multi-armed bandits for causal marketing, arXiv preprint 1rx1v:1810.01859, 2018.
16

---

[102] Meyer Scetbon, Joel Jennings, Agrin Hilmkil, Cheng Zhang, and Chao Ma. A fixed-pon point approach for causal generative modeling. In Proceedings of the 41st International Conference Machine Learning, volume 235, pages 43504 43541, 2024.
[103] Alejandro Schuler, Michael Baiocchi, Robert Tibshirani, and Nigam Shah. A comparison of methods for model selection when estimating individual treatment effects. arXiv:1804.05146, 2018.
[104] Uri Shalit, Fredrik Johansson, and David Sontag. Estimating individual treatment effect: generalization bounds and algorithms. In Proceedings of the 34th International Conference Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 3076-3085, 2017.
[105] Noam Shazeer. GLU variants improve transformer. arXiv:2002.05202, 2020.
[106] Claudia Shi, David Blei, and Victor Veitch. Adapting neural networks for the estimation of treatment effects. In Advances in Neural Information Processing Systems, volume 32, 2019.
[107] Yishai Shimoni, Chen Yanover, Ehud Karavani, and Yaara Goldschmnidt. Benchmarking framework for performance-evaluation of causal inference analysis. arXiv: /rXiv:/802.005446 2018.
[108] Sashi Mohan Srivastava. course Borel sets. Springer, 1998.
[109] Valentin Thomas, Junwei Ma, Rasa Hosseinzadeh, Keyvan Golestan, Guangwei Yu, Maksims Volkovs, and Anthony Caterini. Retrieval & fine-tuning for in-context tabular models. In Advances in Neural Information Processing Systems, volume 37, pages 198439-108400 2024.
[110] Julius Vetter, Manuel Gloeckler, Daniel Gedon, and Jakob H Macke. Effortless, simulation- efficient bayesian inference using tabular foundation models. arXiv:2504.17660, 2025.
[111] Johannes Von Oswald, Eyvind Niklasson, Ettore Randazzo, João Sacramento, Alexander Mordvintsev, Andrey Zhmoginov, and Max Vladymyrov. Transformers learn in-context by gradient descent. In International Conference Machine Learning, pages 35151-35174. PMLR, 2023.
[112] Johannes von Oswald, Maximilian Schlegel, Alexander Meulemans, Seijin Kobayashi, Eyvind Niklasson, Nicolas Zucchet, Nino Scherrer, Nolan Miller, Mark Sandler, Blaise Agüler y Arcas, Max Vladymyrov, Razvan Pascanu, and João Sacramento. Uncovering mesa-optimization algorithms in transformers. arXiv:22309.m 05858, 2023.
[113] Chi Wang, Qingyun Wu, Markus Weimer, and Erkang Zhu. FLAML: Fast and Lightweight AutoML Library. In Proceedings of Machine Learning and Systems, volume 3, 2021.
[114] Kevin Xia, Kai-Zhan Lee, Yoshua Bengio, and Elias Bareinboim. The causal-neural connection: Expressivenesss, learnability, and inference. Advances in Neurul Information Processing Systems, 34: 08833100866 2021.
[115] Kevin Muyuan Xia, Yushu Pan, and Elias Bareinboim. Neural causal models for counter- factual identification and estimation. In The Eleventh International Conference on Learning Representattions, 2023.
[116] Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. An Explanation of In-context Learning Implicit Bayesian Inference. In International Conference Learning Representations, 2022.
[117] Steve Yadlowsky, Lyric Doshi, and Nilesh Tripuraneni. Pretraining data mixtures enable narrow model selection capabilities in transformer models. arXiv:2311.0087/ 2023.
[118] Han-Jia Ye, YinYang Liu, and Wei Chao. A closer look TabPFN v2: Strength, limitation, and extension. arXiv:2502.17361. 2025.
[119] Shuai Yuan, Jun Wang, and Xiaoxue Zhao. Real-time bidding for online advertising: measure- ment and analysis. Proceedings of the seventh international workshop on data mining for online advertising, pages 1-8, 2013.
17

---

[120] Jiaqi Zhang, Joel Jennings, Agrin Hilmkil, Nick Pawlowski, Cheng Zhang, and Chao Ma. Towards causal foundation model: on duality between causal inference and attention. 1rXiv:2310.008000 2023.
[121] Xun Zheng, Bryon Aragam, Pradeep K Ravikumar, and Eric P Xing. DAGs with NO TEARS: Continuous Optimization for Structure Learning. In Advances in Neural Information Process- ing Systems, volume 31, 2018.
[122] Émilie Diemert, Olivier Teytaud, Guillaume Oblé, and Florent Meynet. large scale bench- mark for uplift modeling. In Proceedings of the AdKDD Workshop, 2018.
18

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer:
[Yes]
Justification:
We provide a concrete summary of contributions the end of the introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: We summarize our limitations, alongside future durections, in Section 7. We are also fully transparent in the limitations of our theory, and also some of the practical limitations of our method detailed in Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-tex system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
19

---

Answer: [Yes]
Justification: We provide all the assumptions and the complete proof for Proposition in Appendix B. We also provide proof sketch in the main paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide the code, along with the model checkpoints and Jupyter Notebooks replicate all the experiments in the main paper and appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
20

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes)
Justification: We publish CausalPFN as a standalone PyPI package (https://pypi org/ along with the instructions to reproduce all the results in the paper. The training data is fully public and is sufficiently reproducible from the given implementation details provided.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
guudes/Codeeuumii
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
//ppp.cccpppppp.ppcc wwtpccpppppcccccct.ccccccccc.lct guides CooeeSoomiisseeee wwwbbbimsssnooooiiioiioo
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The paper specifies all of the important details in the main text, in addition to extra details Appendix D. Moreover, the package we release contains all the necessary hyperparameters used for inference.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We provide the 1-sigma standard errors of the mean values in the CATE & ATE results table, across different realizations of each benchmark dataset. We also demonstrate error bars in the calibration plots, which show 1-sigma standard errors of the mean calibration curves, across multiple samples of the synthetic datasets.
Guidelines:
The answer NA means that the paper does not include experiments.
21

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We use single A100 GPU for days to train the base CausalPFN. We also use an H100 GPU for days to produce the TabPFN fine-tuned results in Table Apart from that, all of the other experiments are run on either desktop RTX6000, A100, or an H100.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics atttps://eerre cc http//uulccc/Bebeee httpai///hhhishieellealaeeee
Answer: [Yes]
Justification: All the datasets used the paper were either synthetically generated or publicly available. The authors confirm that the research conducted in the paper complies with the NeurlPS Code of Ethics, to the best of their knowledge.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative socictal impacts of the work performed?
22

---

Answer: [Yes]
Justification: Causal effect estimation is a fundamental problem with many societal benefits across public policy, healthcare, and economies. While we do not directly try solve any critical societal issues, by developing strong causal estimator, we believe it may lead to positive impact in the future.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymentt of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to partieular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: We believe our model poses such risks, as it is method for causal effect estimation for tabular observational datasets, with reliable credible intervals.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All of the project developers are authors in the paper and are properly credited for their contribution.
23

---

Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We attach and release all of the assets and code related to this document. All the code is well-documented and transparent.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our research did not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

---

Answer:
[NA]
Justification:
Our research did not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification:
We did not use LLMs for any important and original contributions in this research.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non con-standard components.
Please refer to our LLM policy (https /rtee////neuuee ////eerrip....// wwt..//ceennnccccccccooooo /LLM) for what should or should not be described.
25

---

Appendix Contents
A Notation, Definitions, and Assumptions
27
Consistency Result
28
B.1 Re-Statement of Proposition 1
28
B.2 Preliminaries for the Proof of Proposition
28
B.3 Proof of Proposition
29
C Validity of the Causal Data-Prior Loss
30
Experimental Details
31
D.1 Prior Generation & Simulating DGPs
D.2 Model Details
33
D.3 Sensitivity Dataset Size
34
D.4 Discussion on Inference Speed
D.5 Baseline Hyperparameters and Results without Hyperparameter Tuning
35
D.6 Marketing Experiments
35
D.7 Calibration, Coverage, and Credible Intervals
36
E Concurrent Work on PFNs for Causal Inference
40
26

---

A Notation, Definitions, and Assumptions
Sample Space. Let B denote the Borel o-algebra on R. Let Z - (X,,T collect the observed variables, taking values in standard Borel space (2, In particular, E X, T T where T is finite, and Y E TR. To reason about counterfactuals, define the augmented variable 2 (X, T, (YHeT,, Y) on a standard Borel space (2, 2z)
Dete-Gengg Parameters. Let (V, Bv) be standard Borel parameter space. For s E Vr a data-genee gettreenerainng process (DGP) is a probability measure p on (2, Bg), which induces the observational marginal Pu on (Z, Bz). We use & denote both the random parameter (when distributed according to prior) and its realized value, when clear from context.
We impose a mild regularity condition to ensure measurability of parameter-to-law maps. Let P(ž) and P)Z) denote the spaces of probability measures on 2 and 2, respectively, endowed with the Borel o-algebras generated by the weak topologies.
Assumption 2 (Measurabilitty. The map vi - Pv E P(2) is measurable, in the sense that & + P B) is By-meauuralle for each B E Bz. Similarly, 1b + Pois E P(3) is meamsuuuall and its image set {P & E ) is Borel subset of P(Z).
Prior and Posterior Distributions. Let be prior on By). Define the joint law pa of and then, conditional v, sampling (Zi)121 iid. from p We use Px to denote its marginal distribution on X.
((Z)))2,,))
vb) by first sampling
Let D obs (Z1, Zz,.... 2n) denote the first n observed variables (the Z-marginals of the corre- sponding 7,). We write r- (ibs)) for the posterior on T induced by ps.
Parametric CEPOs and CEPO Posterior Predictive. For each & T and r-almost every v regular conditional distributions for (Y, X) exist because all relevant spaces are standard Borel. Thus, there
a Borel version of the conditional expectation x + P Y X = x]. We fix a version pe(:: : 1)) that is jointly measurable in (X, ) and call it the conditional expected potential outcome (CEPO):
(x vb)
E [Y Y
X = x], for P every x.
(14)
Assumption 3 (Integrability). For every & E T and Pa-almost every
E" [x, +)] < co.
(15)
For any query x) and dataset De the CEPO posterior predictive distribution (CEPO-PPD). a probability measure R, is the pushforward of the posterior T(t D Dots)) through + Me v):
Th: (B x, Dire) = [ur(x: eb) E (d) Do
BEB.
(16)
Model. Given a query x) and context Di a model with parameters 0 yields predictive distribution (o x, t, ba on ER for the CEPO values.
Observational Quotient Space. Standard consistency results such as Doob's consistency theorem [79] concerned with the parameters of the observational distribution. To leverage such results, characterize the set of DGPs with the same observational distributions as follows:
Detinition 4 (Observational Quotient Space). Let &- / be the set of equivalence classes under: 4/1 ~ 2/2 iff p p Let [ & + be the quotient map and equip with the quotient o-algebra Bg = (A C & H (A) E B. We call (Ø, Ba) the observational quotient space.
We write @ [w]] to denote the equivalence class corresponding to the parameter & and may interchangeably use p(o] ph P to denote the corresponding observational distribution. Note (0) B4) is also standard Borel. Indeed, identify & with the image Rt == { t E v S P(Z) via the measurable bijection [v*] ps obs. By Assumption 2, R is Borel subset of the standard Borel space P(2) (the space probability measures on 2 with the weak topology is standard Borel whenever (2, B2) is), hence R and thus &b are standard Borel [108].
27

---

Identifiability. Finally, we re-state the definition of identifiability from Section more formally. This will be necessary condition to prove our results in the next sections.
(V, Be) CEPO-identifiable, if for each value of E T, there exists a map ft X' x + + IR, such that for 7-almost all parameters vb and -allmmost all values of x, the CEPO value Jer (x: 1)) fx(, [v]]--
Definition 1 (CEPO-Ldentifabilltty We call prior
Note that the above definition is compatible with the standard identifiability in the literature [89]. since there is bijection between each equivalence class [+}] and the observational distribution obs
B Consistency Result
B.1 Re-Statement of Proposition 1
Proposition (Formal), Under Assumptions and 3, there exist sets No S. X and V C with P (X)) = and ((+)) L, such that for all t E T,x E No. and i E v if 11, Zz.... P i.ddd, then
lim E p (1x.P [p] = mi(x; v/*) n00
a.s. a.s.
(17)
and only the prior is CEPO-identifiable
B.2
Preliminaries for the Proof of Proposition
Here, we introduce some concepts to simplify the statement of the proof. We start by presenting a corollary of Doob's consistency theorem without proof (Corollary 2.3 of Miller [79]), which will heavily leverage for the proof of Proposition The result is re-stated to match our parallel notation:
Theorem 2 (Corollary of Doob's Consistency Theorem). Suppose (2,Bz) and (Φ, BB) are two
standard Borel spaces. Let v be probability measure on (4, BB). For each ø E 4, let pe be probability measure on (2, Bz). Consider a measurable map Φ- -> R and assume:
(i) Measurability. 6 + (B) is measurable for every B E Bz.
(ii) No Redundancy. o + = pe & pe
(iii) Integrability. E" [l)((()) <c.
Moreover, define the extended joint probability measure Viot on ((Z1, Zzy....), Q) by first drawing d ~ v, and then, conditioned on ó. sampling ZA, Zz,... i..dd from Po, Then, there exists & C $ with vt ()))) = 1, such that for any do E to and Z1, 7o,, pdo i.d.d we have
lim 12 100
thr (g( Z1
Zn] g( pt..a.s
(18)
The Joint Measure IL. For technical convenience, define joint measure II variables b, (Z.)221 [V], and (Zi)22 as the pushforward measure of pa by the following map:
(v, (Z))22)) (v), (Zz) (0). (Z))
(19)
In particular, we have the following equalities:
I((Z))22 tb, [+]) Pia ((Z))22)) ((4))2)))
[+),
(20)
which results in the conditional independence
(7))221 In & [v].
(21)
Since all spaces involved are standard Borel, regular conditional distributions exist; hence the above conditional laws are well defined [91].
(Notation Remark) We remove the superscript II in expectations and simply write E when we take expectations w.r.t. II. Also, we reuse the symbol IT for the joint measure and for any of marginals or conditionals: the intended meaning will be clear from context.
28

---

B.3 Proof of Proposition
For any given and x, define the expected CHPOs under observational equivalence class & E + as
(x Φ) = Elpp( (x; x)) dd]
(22)
We can use Theorem 2 to establish a consistency result connecting the CEPO-PPDs and the functions Ir defined in (22):
Lemma 3. Under Assumptions 2 and 3, there exist sets Xo S X and vo V with (x) = 1 and
(Wa) I, such that for all E T, E Xv, and yhu E Vo if 11, Zz ..... P i..dd then
lim E Pa (.)x [M] (x [Vu]) P -a.s
(23)
11-000
where y denotes the identity map on R.
Proof. From Assumption 3, subset No S A' exists with Px(G)) 1, such that for all t T and x E Lg. we have E" [1,,(x ei]l < oc. Fix a value of Xn E Xg and to E T. A similar integrability
statement can be made for gto (Xn ; ():
Ellgt, (No
6)]] Eil (xo 16) ø]]
from (22)
I Ee (xo +) []
(Jensen's 's incquality)
Elleto (xo +)] < co.
(total expectation)
(24)
Now, we use Theorem 2 to obtain the desired results for the function 9t, (xp : Φ)) by plugging in (2, Bz) directly from notation and ( Bx) from Definition 4. Moreover, replace v and Vot by the marginals of II on the random variables & and ((Zi)z2,,)) respectively. Finally, it is easy to that all the required assumptions hold:
(i) Measurability, Follows from the measurability in Assumption 2.
(ii) No Redundancy. Follows from the definition of the quotient space in Definition 4. (iii) Integrability, Follows from (24).
As result of Theorem 2, there exists set & C + with [T)))) such that for any do to and Z1, Ze. pdo ii... we have
lim Elgro Φ) Duball gro (Xo Φo) pao-a.s.
(25)
We simplify the expectation in the L.H.S. of (25) as follows:
Blgto (xo Φ) Dibbal = E[E(Mto (xo v)) Φ) 'D
from (22)
E/El/4 (Xo v)) D Φ)
Di
from (21)
Eleto (xo 15) Den
(tower property)
E pn (- xxo Di [p],
(26)
where (+) follows from the fact that CEPO-PPD "1o is the pushforward of the posterior [[ %ba) =
( (be) under the map + Meo (Xo : v²).
We then define v as the preimage of t under the quotient mapping. is easy to verify that T (Wa) = IT = IT( = 1. For any do E Vo, set Φo = [Roo]. Combining (25) and (26), and repeating the entire argument for all to E and Xg Xg concludes the proof.
Lemma 3 establishes a consisteney result between the CEPO-PPDs and functions ge we defined on the quotient space. With the consistency proven in the observational quotient space, all that remains to connect the R.H.S. of (23) to the original CEPOs. This is where identifiability comes into play. In what follows, fix to E T and Xo E Ag:
Equivalently, ge (x Φ) Expr(x; 4) [+]] 6).
29

---

CEPO-Identifabiiiiyy = Consistency. Under CEPO-identifiabini (Definition 1), there exists
v S with 11) 1. where for all v si"d E V that [ = [i"") we have Mrao (Xo : 4/) Meg (Xo )) Define V == v n Vo and note that π (VV* 1. Consequently, for any e/* E WV,, we
also have x E V1, and
Sto (Xo
[+] - [ Pro (xo ; *²)) Eluro (xo v)
(27)
Combining (27) with Lemma 3 and repeating the argument for all to E T and Xo E Ao proves the first side of Proposition 1.
Consistency + CEPO-Identifiabiliyy When consistency holds, from (17), set J C y exists with T( (VV") 1, where for all v0* V if Z1, Z2, obs. then
lim E 10000 uov (1/x0 (w.. )
14z (Xo ; xb²) p** a.s.
(28)
Moreover, according to Lemma 3, there exists a set Vo S v with (VP) = 1, such that for all 4o E Vo, if Z1, Zz..... obs then
lim E "o (1x. Dab.) [m]] 9ta (Xu: [u/]]
obs -a.s.
(29)
Using these two identities, we can define 1 == T/" n Vo where x <V = 1, and the following holds for every t/1 1/
Pto Pto (xo
1/1) Mio (xo [+)])-
(30)
Hence, the prior x is indeed CEPO-identifiable, as we can use the functional in place of f in Definition 1. Repeating this process for all to T and X Xy concludes the proof of Proposition 1.
C Validity of the Causal Data-Prior Loss
Here, we show that the causal data-prior loss is equivalent to the expected forward KL divergence between the CEPO-PPDs and the parameterized distribution qo. For the theoretical justification, we assume a fixed observation size n and define Dobs = D with a dropped superscript for simplicity.
Assumption 4 (Existence of Densities). We assume each CEPO-PPD T x, Drise) admits a density w.r.t. Lebesgue measure and use the same symbol for its density. Moreover, we assume qa( xx f, D is probability measure with full support on I, which admits a density w.r.t. Lebesgue measure. Similar CEPO-PPDs, we use the same symbol for the measure and its density.
Definition 5.
Let Pobs be the marginal distribution of pa on (Zi)s21. Then, the expected forward-KL divergence between n and go is defined as
CKL [[x)) obe [DKL (TT (- x, Dotx)
g0(-
x, Dobs)))
(31)
where Dobs U {x} obs refers to first drawing si and then sampling Dobs (Zi,....
Zn)
i..dd from pv abs and an independent query point x P
Proposition Under Assumption 4, the causal data-prior loss from Definition 3 and the expected forward-KL KL divergence in Definition 5 have the same optina. In other words, for all E TT,
arg min CKL(()) arg min Lz()).
(32)
B
Proof. Fix a L E T. We note that
IT' (11 x, Dotes) log (o x, 1, Donn))
(0) = ED U(x) PT Eynare (-be
(33)
From (16), we know that TH (-
x, Dons) is the pushforward of the posterior T(- Dobs) by the function sb + pe(x; v). Hence, for any measurable function h IR + R. we get
E E (1xx [h(1]]] = Evpat (eer))) [x(u((x; +))]
(34)
30

---

Setting h(1) = log
x (plx. go (.t
in (34) and combining with (33) yields
log T (fu(x; 2)) X, Dohs)
C (0) Uf aan (r
(35)
(x 1/) X, L, Dobs)
T (x4(;; 1)) Dobs)
(xx) Paber poot(1
log
(36)
go( (x; 4))
x, t, Dobs
Next, use the Bayes' rule derive
Po obs (Dobs)
Tlb Dobs) = (vb)
P
obs (Dobs)
(37)
evidence
posterioe
prior
likelihood
Combining (36) and (37), we get
log TTh (ur(x v) x, Dobs)
(,)(()) = plnva, Pob Poin
(38)
ge(ui( 16) x, Dobs).
E. Giv Dxb ~ Pok.
log 20 (M)(x; v/) x, L, Dobs)] constant term in
(39)
(0) + constant term in 0,
(40)
which concludes the proof.
(Remark 1) In general, forward-KL KL divergence loss cannot estimated without estimating the true CEPO-PPD. However, with this identity established, we can justify the use of the equivalent causal data-prior loss which easily estimable.
(Remark 2) The theoretical equivalence is proved only for a fixed treatment & E T and a fixed, finite sample size n 21,2,... .). In practice, the training loss is minimized while randomizing both t and the sample size n. If the optimizer attains a near-optima of this randomized objective, the approximation go(- x, t, Dobs) = (- x, Dobs) can effectively extend all the treatment values and to almost every sample size we care about in practice.
D Experimental Details
D.1 Prior Generation & Simulating DGPs
As illustrated in Figure 4, prior generation consists of retrieving synthesizing base table, subsampling covariates and CEPOs (An and μ4, synthesizing treatments T, potential outcomes Y,, and finally, observed outcomes Y. We break down each of the components:
Data Sources for the Base Tables. We draw the base tables from two sources: (i) real-world tables from OpenML, and (ii) fully synthetic data.
(i) We use the OpenML collections used in Grinsztajn al. [34], AMLB [33], and TabZilla [77], all listed Ma et al. [69]. To widen coverage, we also add tables from CTR23 [28] and CC18 [12]. All OpenML IDs in this link. Data leakage is ruled out none of the tables that share covariates or outcomes with our test sets (Lalonde, IHDP, ACIC, Criteo, Megafon, Hillstrom, Lenta, X5) are included in training. Moreover, the propensities sampled purely synthetically, following the approach described below.
(ii)
For additional diversity, we generate synthetic tables using the random neural networks used to train TabPFN vl, with the same hyperparameters described in Hollmann et al. [42]. Inputs, from standard Gaussian distribution, are fed into the network, and subset of the outputs and hidden neurons are selected to construct the tabular data. Some columns discretized at random to produce categorical and ordinal variables to reflect the structure real-worid tabular domains. While TabPFN v2 [43] is newer and stronger model, its training data is not publicly available, so we restrict ourselves to the vl generator to ensure transparent evaluation and leakage control.
wwwggle.comfille///e //tp::////dooooooooooo/o////////
htw.://s5556--33xrerooee5o5 13sxFrcWeP52
31

---

CEPOs with Heterogeneity Control. Once the base table is given, we randomly select two columns and name them How.0 and 1lw,11 However, in practice we observe that directly using such columns for CEPOs can result in large variances (heterogeneity) for CATES. We therefore apply light-weight post-processing inspired by RealCause [81].
The post-processing requires a heterogeneity hyperparameter which we sample uniformly from [0, 1] during prior generation. Then, for N units (rows) extracted from the base table, let Tra Mraw, - Praw. be the CATE for unit n E [M], and Araw n E Traw the sample ATE. We draw i..d. {a(n) In=1 Unif(0, 1] and construct the final n-augmented CEPOs as
P1 (n)
(n) + (1 - a (")) Praw, + (1-7))(1 (n) + Araw);
(n)
(41)
(n) (1 - (1-a(a) + a ( 1raw.0 + (1 - (1-7)a (" (1(a0)) - draw)-
(n)
P0
(42)
simple algebraic check shows
((n) Mi p4
(n)
(m)
riw + (1-7))raws Var(T x] = [ar[Traw
xx]
(43)
Hence. while preserving the average treatment effect, 7 0 yields a dataset with a zero variance CATE (fully homogeneous), whereas 7 = 1 recovers the original heterogeneity.
Outcomes. After constructing the CEPO columns [x(x) and μt (x), need to turn them into potential outcomes by adding zero-mean noises. To avoid tying the data to a specific parametric noise model, we introduce additional nuisance columns, (x(x) and m (x), sampled from the base table.
Let Cg be random scalars, independent from x, with E[(e] = 0. We define the potential outcomes as
Y = (x()) + (x(x) E,, t TET.
(44)
This construction preserves the conditional means, that is E(; (x The input-dependent scale factors (x) allow for heteroscedastic noises and capture richer family of outcome distribu- tions than additive parametric noise models. For our training, we sample e from a Gaussian with variance uniformly drawn from (0, Var(w:)). This choice of noise values ensures a similar noise scale to the scale of CEPOs, resulting in training data with a more informative signal-to-noise ratio.
Propensities with Positivity Control. Given covariate vector x, the strong ignorability assumption requires the propensity values 0 < P(T = 1 X = x) < 1. Hence, due to the invertibility of the sigmoid function, it is sufficient to generate treatment logits, through any function f X - R. and then apply a sigmoid function to get values within (0, 1). To simulate different degrees of confounding, we choose f by randomly selecting one of the following mechanisms:
(i) Randomized treatments (RCT). Treatments are independent of covariates, i.e., f is con- stant. We sample Logistic(O,, 1) and set fxx c to get uniform propensities.
(ii) Linear logits. Draw the random vector w from a standard Gaussian and set f(x) = w x. (iii) Non-linear logits. Feed x into a randomly initialized MLP, similar architecture to that of Hollmamn et al. [42], to get fx).
Empirically, we observe that the above procedure yields an artificially high level of positivity. which not reflective of real-world scenarios. We therefore apply a light-weight post-processing transform. inspired by RealCause [81], to better control the positivity level. Coneretely, we sample parameter E E [0, 1] and exacerbate extreme propensity scores to mimic poor positivity:
P(T x = & {Sigmoid((x))) + (1 (1- I)f()) 0).
(45)
Here, < 1 leaves the original positivity intact. However, for smaller values, the support of the treated and control groups become increasingly disjoint, leading to low-positivity scenarios.
Treatment Assignment. Finally, each unit's treatment is drawn as T and the observed outcome Y is also derived by selecting the assigned potential outcome Y Yy-
Collectively, all of the steps above simulate different DGPs, with various levels of positivity and heterogeneity, extracted from real and synthetic sources of tabular data. This procedure creates a broad prior for CausalPFN, which is necessary for the model to work well in practice.
32

---

D.2 Model Details
Architecture & Training. We represent each context row (1, x, y) and query row (t, x) as single tokens by summing up (1) treatment embedding for t, (2) covariate embedding for x (padded to length F = 100), and (3) an outcome embedding for y (only for context rows). We use linear layers for embeddings and omit the positional encodings to preserve the permutation invariance of the context set, similar to other PFN-style transformers.
All tokens- and query- are passed into a 20-layer transformer, with hidden size of 384, QK- "K-normalization (RMS)', and a parallel SwiGLU-acttvvate [105] feed-forward block.
The transformer's query outputs are then projected to (024-dimensional logit vector, then softmaxed fixed temperature of 0r 1.0 form a discrete CEPO posterior over the interval [-10, 10]. We then scale the interval to match the scale of the outcomes and clip the out-of-range values. At inference time, we return the posterior mean as the point estimate and sample 10,000 times to estimate credible intervals at any desired significance level a.
The full model has approximately 20M parameters and is trained in two stages: (i) predictive phase that mimics standard predictive PFN training from Ma et al. [69], and (ii) a causal phase that optimizes the CEPO loss. We use AdamW [55] with warmup and cosine annealing for the predictive phase, and switch to the schedule-fre frree optimizer [24] in the causal phase. The model is trained with a maximum context length of 16K in the first phase and 2,048 in the second. We use four A100 GPUs trained for at most one week for the initial phase, and two days on an H100 for the second phase.
Finally, to enhance parallel training, we batch both the queries and the tables. That is, rather than sampling only one DGP and one query token, each gradient update samples B, DGPs, draws Bg queries per DGP, and concatenates everything into single tensor. The tensor is then passed through the transformer to get B, CEPO-PPDs. The final loss is averaged over all the batches. See Algorithm for detailed demonstration of CausalPFN's training algorithm.
Algorithm Parallel training of CausalPEN,
Require
Prior DGPs and CEPO values pe obe pel-; t)), model qo, DGP batch size Br, query batch size Bg. fixed feature length F, and histogram loss HL (10) [44].
I: while not converged do
2: Sample (i]],..., [B] ~
3: Sample [i ~ pule V1 visiS
4: Randomly sample query treatments ((-d) for S sis S B, 1si5 $ B
5: Sample query covariates (t.)) abell [1] for 1 Sig < B, 1 s S B
6: Set (e6) < M2(5.3) (x(5 +(1)
7: Pad (1.)) with zeros such that (i.)) E R
2+ B, Σi, HL (2(5)) (1q0 x(00) ((()) Patna [8)
9: Update using the gradients VoL
10: end while
Handling Large Tables at Inference Time. CausalPFN's default maximum context length is set to 4,096 at inference, but real-world tables may contain millions of rows. Training PFN-style transformers on such long contexts can be challenging due to hardware or architectural constraints. While some tabular foundation models such TabICL [92] modify the architecture itself, Thomas et al. [109] show that, retrieving a small relevant subset of rows for each query at inference time allows model with short context length to better generalize longer contexts.
We adopt this retrieval philosophy in CausalPFN to enable causal effect estimation on large tables. First, we fit lightweight gradient boosting regressor on the context data to produce weak CATE estimates for each covariate, This regressor estimates CATE by regressing outcomes on the treatment and covariates and then taking the difference in predicted outcomes between T = I and T = 0. This step applied only when the table is too large to fit within the model's 's maximum context window. We then (i) sort both the context rows and the queries based on their weak CATE estimates, which
Differren from Henry al. [38], perform normalization after the query and key projection.
33

---

effectively stratifies the data; (ii) partition the ordered queries into consecutive mini batthbattee and (iii) for each query batch, use a fast bisection search select a contiguous window of context rows whose weak CATE estimate range most closely matches that of the batch. As result, each batch is exposed only to a neighborhood of rows with similar causal effects, allowing all CEPO predictions to be computed with short forward passes.
D.3 Sensitivity to Dataset Size
During the causal phase of training, consider sample sizes up 2,048 and covariates up 100. However, during inference, CausalPFN can take up to 50,000 samples.
To assess the effect of context size and dimensionality CausalPFN's performance, run additional experiments on synthetic polynomial datasets. The test set size is fixed at 100 in all experiments. For each (rows, covariates) configuration, report mean + standard error over 50 datasets drawn from the polynomial prior with different random seeds.
Effect of Sample Size. We consider the same DGPs while increasing the number of samples and fixing the number covariates to 10. Table 4 reports PEHE for CATE across baselines. CausalPFN exhibits faster PEHE decay with increasing rows, with a slight plateau at very large contexts.
Table Effect of sample size on PEHE (mean SE). Covariates 10; averages over 50 datasets.
Method
Number of Rows
10
20
50
100
200
500
5,000
10,000
CausalPEN
1.34=0.02
1.27=0.02
1.10:0.02
0.89=0002
0.74+0.03
0.46=0.01
0.29=0.01
0.31+0.01
DA-Leamer
1.33++002
1.30=0002
1.16+0.01
1.00++0001
0.91=0.03
0.85=0.01
0.84=0.02
0.82+00.22
S-Learner
1.44+00.1
1.40+0.02
1.35++0.02
1.21+0002
1.18+0.05
1.07+0.03
1.00+0.04
1.03++0.04
T-Learnr
1.33:10022
1.30.10.2
1.1511001
0.97.10.01
0.88110.02
0.8110001
0.81:1002
0.81110.02
X-Leamer
1.35=0.02
1.32=0002
1.20+:0.02
1.04+0001
0.94+0003
0.87=0.01
0.87=0.02
0.84+00.22
Effect of Covariate Size. Next, we fix the number of samples 1,000 and vary the number of covariates. Table 5 compares CausalPFN's performance to other methods in terms of PEHE. Although CausalPFN consistently outperforms the baselines, the performance gap narrows as the number of covariates grows-likely due to training exposure being limited to up 100 dimensions, which could be mitigated by training with higher-dimensional inputs.
Table 5: Effect of covariate size on PEHE (mean + SE). Samples = 1,000; averages over 50 datasets.
Method
Number of Covariates
5
10
20
50
100
500
1,000
CausalPFN
0.0810.00
0.17-0.01
0.40:10.11
0.6710.01
0.8710.02
1.0110.02
1.28-0.02
1.32-10.02
DA-Leamer
0.21=0.01
0.59=0001
0.85+:0.11
1.04:1.01
1.14:0.01
1.22=0.01
1.30=0.02
1.32++0.02
S-Learner
0.54=0.04
0.83=0.02
1.09++0.22
1.17+0.02
1.21+0.02
1.23=0001
1.30=0.02
1.33-t0.02
T-Leamer
0,27+0.01
0.60+0.01
0.83++.011
0.98++0.01
L.9++0.01
1.18+0.01
1.28+0.02
1.32+0002
X-Leamer
0.2910.01
0.64
0.88
1.06
1.15:0001
1.23:10.01
1.30
1.33:10.02
D.4 Discussion on Inference Speed
Many applied settings prioritize throughput and latency marginal gains in asymptotic accuracy. Real-time bidding must estimate incremental ad effects and decide bids within striet millisecond budgets [119]. Likewise, e-commerce personalization depends on rapid uplift estimation within short user sessions, where serving latency directly affects conversion [101].
Although CausalPFN requires substantial offline training, it is designed for zero-shot deployment on new tables with no test-time fitting or adaptation. At inference, interventional queries reduce to a small and fixed number of forward passes, and the computation parallelizes well across large batches (e.g., using mixed precision, caching).
Accordingly, Figure does not claim that CausalPFN is intrinsically faster than every baseline; rather, reflects practitioner-facing wall-clock time from data arrival to effects returned. Baselines that require per-dataset refitting tuning incur this cost at deployment, whereas CausalPFN does not.
34

---

D.5 Baseline Hyperparameters and Results without Hyperparameter Tuning
No Hyperparameter Tuning, Table 6 summarizes the performance of all methods without hyperpa- rameter tuning. CausalPFN attains the best (second-best) average rank on CATE (ATE).
Ec Hyperparameters. For the results without hyperparameter tuning in Table 6, we the models with the recommended hyperparameters in the Jupyter notebooks from EconML [[]]. For the tuned results in Table 1, we performed hyperparameter tuning using the FLAML (AutoML) library [113] on both the propensity and outcome models with (i) Time budget of 900 seconds, (ii) K-fold cross-validation with K = 3. (iii) Early stopping, and (iv) base esti- mators ["Igbm", "xgboost", "xgb_ "xgb_limitdepth" "rf", "kneighbor", "extra_ "extra_tree", "Ir11", "Ir12"]. For Forest DR- and Forest DML, we additionally expanded the co- variates with cubic terms (polynomial degree 3), with an additional tuning of the final model.
CATE Nets. For the results without hyperparameter tuning in Table 6, we ran the models with the default hyperparameters and batch size of 512. For the tuned results in Table 1, perform grid search on the hyperparameters for the neural architecture: (i) Number of layers E (2,3), 3),,0 (ii) Representation dimension E {128, 256), (iii) Number of hidden output layers 23. and (iv) Width of the hidden output layers E {128, 256]. The rest of the hyperaparameters are left unchanged.
BART & GRE. The GRF implementation includes internal tune option. We enable this option in Table and disable it for the untuned experiment in Table 6. BART, on the other hand, offers no comparable eyperparameter-tunnng Its only alternative, full cross-fit, fit, is prohibitively slow and uses a rudimentary Bayesian routine. Thus, the BART scores appear unchanged in Tables and 6.
Table CATE & ATE results. PEHE (left half) alongside ATE relative error and its overall average (right half). PEHE for Lalonde cesesiD is shown in thousands. Best numbers are in blue; second best are purple. Cells
with indicate that the method is not applicable.
Method
Mean PEHE
Standard Error better)
Mean ATE Relative Error Standard Error (1 better)
HDP
ACIC 2016
Lalonde
Lalonde [X1IF)
IHDP
ACIC 2016 Lalonde
Lalendo
Avg.
(x109)
Rank
Rank
CausalPEN
0.58-0.00
0.92-0.11
8.96:0.02
14:40=0.20
2.17-0.09
0.20==0.04
0.05=0001
0:13+0001
0:22-0.02
4,26::0.18
DA-Leamer
2.9810.51
188.10.24
9.0110.02
13196-0.19
3.64
0.22:1004
0.0910.03
0.2210.01
0.08
4.1510.19
T-Learner
2.9420499
2.06:2020
9.2910.02
13.1-0018
4.01
0.22:0004
0.11:0.03
0:4010.01
0.07:0.01
4.6220.18
DragouiNrt
2.13=00224
2.23=00.0
10.83:0.15
16.40:0.27
5.62=0.17
0.21=2004
0.09:0.02
0.56+0003
0.4420.02
6.04-0.177
IPW
0.23-=004
0.24+00.5
0.22:0.01
0.07-0.01
4.33+-.20
TarNet
189910.15
0.26:10.20
12.0011.004
18.71
6.87.10.11
0.21:10.04
0.06100:2
0.9010.01
0.72::0.01
7.54:0.14
X-Learner
3.70100622
1.7120.31
12.2810.03
21.72=0.16
8.13:0.16
0.1920:03
0.0710.02
0.8310.01
0.92:0.01
7.9210.17
RA-Net
2.08:00199
2.42=00222
12.8620012
20.13=0044
8.18:20.16
0.20=00.44
0.07:0003
0.9620002
0.71::0.04
7.95:20117
BAKT
2.50+00.99
0.68-0.11
12.81+0.05
21.36-0.16
8.20-0017
0.44--0.09
0.04-0.01
0.99+0:01
0.86-00011
8.72+0.18
GRF
4.26110.69
0.36
12.18:10.06
21.84=0.16
8.21:10177
0.18:00.3
0.07:0.02
0.81:0.02
0.85-0002
7.78:10.17
S-Leamer
3.911:0.68
2.23-1.28
12.8810.02
12.68=0.13
9.2910.18
0.28
0.1210.05
1.0010.00
1.0300.00
9.9910.18
Forst DR Learner
3.90=0.66
1.68-0.35
26.081:4.96
22 22.55=0.25
9.51=0.18
0.19:0.04
0.06:0004
1.39:0.28
0.87:0.03
8.35
Forest DMI
4,40==0.72
1,47=0.32
15. 12:0115
23.12-0,15
10,51=0.18
0.09-0.02
0.05-00022
1.12+0022
1.02-0.01
9,37++023
All inference, including baselines, performed 80 GB H100 GPU, 32 CPUs, and 256 GB RAM.
D.6 Marketing Experiments
Datasets.
Apart from Hill() and (ill which were explained in the main text. We also run experiments on the following datasets:
1. Criteo.
25M ad-exposure records from Criteo's 's online incrementality tests: a randomly selected held-owt audience is shielded from seeing an advert, while the treated audience is shown the ad; the target is post-impression conversion flag. We use a readily provided 2.5M stratified subset of this dataset from sklift.
2. Retail-Hero (X5). Transaction logs from the X5 Retail Group hackathon. Customers are randomly offered personalized coupons (treatment); the outcome records whether the customer subsequently purchased the promoted items.
3. Lenta. SMS-based promotion experiment run by the grocery chain Lenta. The treatment group receives marketing text, and the outcome is visit after the campaign window.
35

---

4. Megafon (Mega). Synthetic yet Nommiin-aithhfe data released for the MegaFon Uplift competition. Users are randomly offered a telecom upsell offer (treatment), and the outcome indicates whether they accepted the offer.
Qini Evaluation. To build Qini curves we follow scikiituupisg recommended five-fold stratified split based on the outcome and the treatment [74]. In each fold, we hold out 20% of the data as test rows and train the baseline models on the remaining 80%. For CausalPFN we use that same 80% as context tokens and treat the held-out 20% as queries. We then rank the rows based on their CATE estimates to compute the Qini curves and the corresponding Qini scores.
Context Length Challenges. In all the marketing experiments, have increased the model's s maximum context length from the default 4,096 to 50,000 tokens. This context length is sufficient for the subsampled datasets in Table 2. However, extending beyond 50K for the full-table runs is not feasible in GPU memory. We thus use the retrieval approach explained in Appendix D.2 to achieve CATE estimates for this setting. Table 7 shows CausalPEN's performance (with the retrieval approach) compared to the baselines on the fuull-able datasets. We conjecture that the relative under-performance compared Table is due this retrieval heuristic.
Table 7: Normalized Qini scores (t better). Scores are normalized per dataset such that the top-performing model achieves 1.0 (highlighted in bold). All datasets full stratified subsamples: Hil and Hill (64K rows), Critco (2.5M rows), X5 (200K rows), Lenta (687K rows), and Mega (600K rows).
Method
Hi
Hi)
Critco
X5
Lenta
Mega
Avg.
S Learner
1.000
1.000
1.000
1.000
1.000
0.913
0.985
X Learner
0.975
0.980
0.994
0.965
0,868
0.997
0,963
DA Leamnr
0.985
0.964
0.955
0.969
0.903
1.000
0.963
T Learner 0.991
0,972
0.902
0.953
0.833
0.987
0.940
CausalPFN 0.992
0.968
0.939
0.746
0.947
0.954
0.924
D.7 Calibration, Coverage, and Credible Intervals
The Synthetic DGPs. For the calibration results in Figure 7, use two families of synthetic DGPs, polynomials and sinusoidals. As a general recipe, each DGP defines a treatment logit function f(x) IR and assigns treatments by sampling from the Bernoulli (aialliiiiimii((((((fl fx))). Moreover, each DGP specifies two CEPO functions Noo /11 : + R. It then samples the potential outcomes by y = (x) + 4 for t (0, 1}, where the noise terms 6 ~ (ormal(O, 1). Laplace(0, 1), or Uniform( (1, 1, with equal probability. We now describe each DGP family in more detail:
(a) Polynomial. We first draw the number of features d ~ Uni 20) and sample covari- ate vectors x ~ 2n We then fix maximum degree deg E {1, 3, 4), augment covariates with powers Xext (I1, .C1.....22. Fd, 2f.......... zdd dee), sample weights WHo Wai Wr Unif[-5,55 5, 5]dxd +1, and define
.
f(x) = wTxee: (x( = w/ n. Xea for EE 10.
(46)
Degrees 1, 2, and 4 give the Linear, Quadratic, Cubie, and Quartic sub-families, each degree adds new terms and is therefore a super-set of all lower degrees. We train on one degree family and test on the others to probe generalization.
(b) Sinusoidal. We draw the number of features d ~ Unif(5,... 10) and sample covariate vectors x ~ Unif[-3, 5]d, We then sample weight vectors WHo: Way, wr ~ Unif[- 10, 6j4, and a frequency w R+ We define the treatment logit function and the CEPOs as
[(x = sin(w (wTx)) + wTx, (x))) = sin(w (w,,. x]) + w/ x for L (0, (47)
For training DGPs, we create three sub-families: Linear (w = 0), (w E 00 1) and L.2 (w E (1, 2)) For test-time DPGs, we use the following: Linear (w 0). LI (w E [0.5, 1), 1.2 (w E (1.5,2) 2)), and L.3 (w E 32. This allows us to measure extrapolation to unseen frequencies. For example, L2-trained model has seen DGPs from LI and L2, but not L3.
36

---

Regression
CATE
Perfect Calibration
Linear
0.5
0.5
0.5
0.5
0
0.5
0
0.5
0.5
0.5
Sinusoidal (LI)
Coverage
0.5
0.5
0.5
0.5
0
0.5
0.5
0.5
Sinusoidal (L2)
0.5
0.5
0.5
0.5
0
0
0.5
0.5
Linear
Sinusoidal (L1)
Sinusoidal (L.2)
Sinusoidal (L3)
Confidence Level
Figure CATE and regression calibration curves for synthetic sinusoidal datasets, before calibration. Models are trained on Linear/Sinusoidal (L1 (.In/Sinsli (I.2) datasets and tested on Linear/Sinusoidal (I.isSiilloidal (1.2)/Siuusoidl (I.3) benchmarks.
Synthetic Experiments Sinusoidal. Figure 8 shows both the regression curve Cot (orange) and the CATE curve cou,- The model is overly confident in OOD scenarios (e.g.. L2 tested on an L1 trained model) and either well-calibrated or conservative otherwise. The figure also shows that the regression curve is always below the blue CATE curve. Once calibration is done on the regression curve, shown in Figure 9, the ICE,, becomes smaller, resulting in well-calibrated conservative model, even on OOD scenarios.
Synthetic Experiments Polynomial. Similar to the sinusoidal setting, the uncalibrated curves in Figure 10 show that the model becomes overly confident when tested on OOD data (c.g., testing a model trained on Quadratic data Cubic DGP). However, applying the regression calibration results in near-perfect CATE calibration, as shown in Figure 11.
Calibration of the Large-scale CausalPFN. We evaluate the calibration curves of the large-scale pre-trained CausalPFN on both synthetic and standard benchmarks in Figures 12 to 14. The model generally appears conservative. This may be attributed to the Gaussian smoothing used in the histogram loss; yet, this smoothing necessary to achieve stability training. Regardless, across all datasets, post-hoc regression calibration improves reliability: the calibrated (pink) curves adhere far more closely to the diagonal than their uncalibrated (blue) counterparts. In Figures 12 and 13 the improvement is almost perfect, while in Figure 14 it corrects the base model's 's strong conservatism on IHDP and ACIC 2016 and achieves near-ideal alignment on the Lalonde datasets.
37

---

Regression
CATE
Perfect Calibration
Linear
0.5
0.5
0.5
0.5
0
0
0
0.5
0.5
0.5
0.5
Sinusoidal (LI)
Coverage
0.5
0.5
0.5
0
0.5
0
0.5
0.5
0.5
Sinusoidal (L2)
0.5
0.5
0.5
0.5
0
0.5
0.5 Sinusoidal (L1)
0.5
0.5
Linear
Sinusoidal (L2)
Sinusoidal (L3)
Confidence Level
Figure 9: CATE and regression calibration curves for synthetic sinusoidal datasets, after calibration. Models are trained on Linear/Sinusoidal (L1 /Sinusoil (L2) datasets and tested Lincar/Simusoidal (L /Sinusoiial (L2)/Sinusoidal (L3) benchmarks.
Regression
CATE
Perfect Calihration
Quadratic
0.5
0.5
0.5
Coverage
0.5
0.5
0.5
Cubic
0.5
0.5 Cubic
0.5
Quadratic
Quartic
Confidence Level
Figure 10: CATE and regression calibration curves for synthetic polynomial datasets, before calibra- tion. Models are trained on Quadratic/Cubic datasets and tested on Quadratic/CubiicQuariii ones.
38

---

Regression
CATE
Perfect Calibration
Quadratic
0.5
0.5
0.5
Coverage
0.5
0.5
Cubic
0.5
0.5
0.5
Quadratic
Cubic
Quartic
Confidence Level
Figure 11: CATE and regression calibration eurves for synthetic polynomial datasets, after calibra- tion. Models are trained on Quadratic/Cubic datasets and tested on Quadratic/Cubiic/uartii ones.
CATE
CATE (Calibrated)
Perfect Calibration
Coverage
0.5
0.5
0.5
0.5
0
0.5
0
0.5
0.5
0.5
Linear
Sinusoidal (L1)
Sinusoidal (L2)
Sinusoidal (L3)
Confidence Level
Figure 12: CausalPEN's CATE calibration on sinusoidal datasets, before and after calibration.
CATE
CATE (Calibrated)
Perfect Calibration
Coverage
0.5
0.5
0.5
0.5
0.5 Cubic
0.5
Quadratic
Quartic
Confidence Level
Figure 13: CausalPFN's CATE calibration on polynomial datasets, before and after calibration.
CATE
CATE (Calibrated)
Perfect Calibration
Coverage
0.5
0.5
0.5
0.5
0
0.5
0.5
0.5
0.5
IHDP
ACIC2016
LaLondeCPS
LaLDndePSID
Confidence Level
Figure 14: CausalPFN's CATE calibration on standard benchmarks, before and after calibration.
39

---

E Concurrent Work on PFNs for Causal Inference
Do-PEN [96] is a concurrent approach that extends TabPFN to interventional queries by learning the interventional posterior predictive distribution, i.e., distribution over Y, given (X, Doba). . In contrast, CausalPFN targets the expectation of the interventional distribution (e.g., E[ X-x). thus removing outcome (aleatoric) noise from the prediction target. This especially relevant for uncertainty quantification, where we aim to isolate epistemic uncertainty about the causal effect.
More importantly, as described, Do-PFN does not explicitly enforce identifiability: the training prior can include observationally equivalent DGPs (distinct processes with the same P', but different effects). As formalized in Proposition if the training prior admits such cases, then any learner that conditions only on observational data cannot, in general, have its posterior predictive concentrate the true effect, even with unlimited samples and model capacity. CausalPFN avoids this by constructing a prior that satisfies the ignorability (identifiability) condition, ensuring that CEPOs functionals of Poha (one effect per observational law). Empirically, CausalPFN outperforms Do-PFN on standard benchmarks in both PEHE and ATE relative error (Table 8).
Table 8: Head-to-head comparison benchmarks (mean + SE; | is better). For PEHE, Lalonde CPS/PSID values are reported x 10
IHDP
ACIC 2016
Lalonde CPS
Lalonde PSID
CausalPHN
0.58 0.07
0.92 + 0.11
8.96 + 0.02
14.40 + 0.20
PEHE (1)
Do-PEN
6,07 0.89
4.11 0.52
12.01 0.03
20.91 0.14
CausalPFN
0.20 + 0.04
0.05 :: 0.01
0.13 0.01
0.22 0.02
ATE Relative Error (L)
Do-PFN
0.57 + 0.10
0.67 0.04
0.87 0.01
0.92 0.01
40


Reviews:
[{"review_id": "5dZpzZ6Aa2", "reviewer": "Reviewer_MX9h", "summary": "The paper introduces a new causal effect estimation method. The Authors leverage prior-fitted networks to amortize the estimation effort and increase both accuracy and efficiency. The method is theoretically justified to work under a strong ignorability assumption. Empirical results across multiple benchmarks demonstrate that the proposed approach performs competitively compared to a diverse set of baselines.", "questions": "1. How large are the datasets used in terms of sample size and number of covariates?  How many covariates are considered both during training and evaluation? How does the method’s performance in terms of accuracy scale with the number of covariates compared to other methods?\n2. Could you provide a version of Figure 1 that isolates inference time only, excluding training and hyperparameter search for the baseline methods? This would help in fairly assessing the computational efficiency of CausalPFN", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**:\n1. The presented approach is novel, interesting, and well motivated by recent advances in generative modelling and the challenges in causal inference.\n2. The paper is well written, and the core ideas are presented clearly and convincingly.\n3. The experimental evaluation is extensive and sound.\n\n**Weaknesses**:\n1. Details regarding method scalability are not discussed. (see question 1)", "was_revised": true, "final_justification": "The rebuttal assured me of the paper's quality. The additional experiments have resolved my concerns. I decided to keep my score.", "rebuttal": "We thank the reviewer for their effort and considering our paper as novel, well-motivated, clearly explained, and our experiments as extensive. In what follows, we have provided detailed responses to each individual comment:\n\n> Details regarding method scalability are not discussed. How large are the datasets used in terms of sample size and number of covariates? How many covariates are considered both during training and evaluation? \n\nIn the appendix (footnote 2 on page 28), we provided an anonymized link to the list of all the real-world datasets (OpenML tables), which includes details such as sample sizes and the number of features. For the causal phase of the training (synthetic datasets), the following table summarizes the characteristics of the tables used, which we will include in our revision.\n\n**Table: Description of Training Data**\n| Description    | Value  |\n|----|----|\n| Number of Datasets   |  $\\approx 11{,}140{,}000$   |\n| Range of Sample Sizes in the Context (Training)   | $(\\frac13 \\times 2048, \\frac23 \\times 2048)$  |\n| Range of Sample Sizes in the Query (Test)   | $(\\frac13 \\times 2048, \\frac23 \\times 2048)$  |\n| Range of the Number of Covariates   | $[1,99]$   |\n\nFor evaluation, we used standard benchmarks, and we detail the number of samples and covariates below:\n\n**Table: Description of Benchmarks**\n| Dataset   | Training Sample Size| Test Sample Size |Number of Covariates  |\n|---|----|----|--|\n| IHDP  |  $672$  | $75$ | $25$ |\n| ACIC 2016  | $4{,}321$  | $481$ | $58$ |\n| Realcause Lalonde CPS  | $14{,}559$ | $1{,}618$ | $8$ |\n| Realcause Lalonde PSID  | $2{,}407$  | $268$ | $8$ |\n\nFor the marketing datasets, as stated in the paper, we perform 5-fold honest splitting. We evaluate both the original sizes (Table 5) and a stratified 50k sub-sampling (Table 2).\n\n**Table: Description of Benchmarks**\n| Dataset | Sample Size| Number of Covariates   |\n|----|----|---|\n|Hill (1)  | $42{,}963$ | $10$ |\n|Hill (2) | $42{,}613$ | $10$ |\n|Criteo | $1{,}397{,}960$ | $12$ |\n|Lenta | $687{,}029$ | $193$ |\n|X5 |$200{,}039$|$6$|\n|Megafon|$600{,}000$|$50$|\n\nTo handle more than 99 covariates, we apply PCA to the covariates and retain only the first 99 principal components to obtain a condensed representation of the covariates that can fit into the input of the model.\n\n> How does the method’s performance in terms of accuracy scale with the number of covariates compared to other methods?\n\nBased on your suggestion, we conducted an additional ablation experiment to assess the effect of the number of covariates on the performance of CausalPFN compared to other methods. For this ablation experiment, we focus on the synthetic polynomial datasets described in lines (1262–1275) of the appendix. We fix the test size at 100 samples and number of training samples as 1000, and increase the number of covariates. For each covariate size, we report the average ± standard error of PEHE of CATE estimates over 50 tables generated from the polynomial prior with a fixed random seed to control for randomness:\n\n**Table: Effect of Covariate Size**\n| Method\\\\# of Covariates| $1$  | $5$ | $10$ | $20$ | $50$  | $100$ | $200$ | $500$  | $1000$  |\n|----|--|---|----|---|----|-----|---|---|--|\n| **CausalPFN**  | $0.08±0.00$| $0.17±0.01$| $0.40±0.01$| $0.67±0.01$| $0.87±0.02$| $1.01±0.02$| $1.17±0.02$| $1.28±0.02$| $1.32±0.02$|\n| DA Learner| $0.21±0.01$| $0.59±0.01$| $0.85±0.01$| $1.04±0.01$| $1.14±0.01$| $1.22±0.01$| $1.25±0.01$| $1.30±0.02$| $1.32±0.02$|\n| S Learner| $0.54±0.04$| $0.83±0.02$| $1.09±0.02$| $1.17±0.02$| $1.21±0.02$| $1.23±0.01$| $1.26±0.02$| $1.30±0.02$| $1.33±0.02$|\n| T Learner| $0.27±0.01$| $0.60±0.01$| $0.83±0.01$| $0.98±0.01$| $1.09±0.01$| $1.18±0.01$| $1.23±0.02$| $1.28±0.02$| $1.32±0.02$|\n| X Learner| $0.29±0.01$| $0.64±0.01$| $0.88±0.01$| $1.06±0.01$| $1.15±0.01$| $1.23±0.01$| $1.26±0.02$| $1.30±0.02$| $1.33±0.02$|\n\nThe table suggests that CausalPFN consistently maintains a performance gap over different numbers of covariates. However, the gap becomes smaller as the number of covariates increases. This is likely due to the limited number of covariates during training (up to 100). We believe that increasing the number of covariates during training can address this issue. Still, we observe that CausalPFN can generalize well to a larger number of covariates.\n\n\n> Could you provide a version of Figure 1 that isolates inference time only, excluding training and hyperparameter search for the baseline methods? This would help in fairly assessing the computational efficiency of CausalPFN.\n\n\nWe included the training and hyperparameter tuning time for other baselines in Figure 1 as this is what would normally happen in practice: when faced with a causal task, domain experts typically search over a set of possible design choices, including different hyperparameters. On the other hand, CausalPFN is only pre-trained once, and for each new dataset, it only requires running a few forward passes of the transformer model. We want to emphasize that Figure 1 is not intended to claim that the inference speed of CausalPFN is inherently faster than that of the baselines; rather, it illustrates what a practitioner would typically experience in terms of end-to-end runtime when choosing an approach. \n\nHaving said that, we agree with the reviewer that providing inference-only time can provide more insights into the efficiency of CausalPFN. However, including the inference-only time requires re-training all the baselines, which can take more than a week of computation and is infeasible for the rebuttal period. Instead, we provide results that we already had available that contain the training + inference time for the baselines, excluding the hyperparameter optimization time. \n\n**Table: Median Time for CATE Estimation (Seconds per 1000 Samples)**\n\n| Method                  | ACIC 2016 | IHDP     | RealCause CPS | RealCause PSID | Overall   |\n|------------------------|-----------|----------|----------------|----------------|-----------|\n| IPW           | `0.059753` | `0.125096` | `0.040139`     | `0.081708`     | `0.078507` |\n| S Learner    | `0.075550` | `0.454180` | `0.021157`     | `0.115120`     | `0.118159` |\n| T Learner    | `0.116674` | `0.883767` | `0.043567`     | `0.245745`     | `0.247548` |\n| DA Learner   | `0.283248` | `2.076818` | `0.089609`     | `0.535804`     | `0.548800` |\n| **CausalPFN**             | `0.630637` | `0.673687` | `0.818941`     | `0.485863`     | `0.555087` |\n| GRF          | `1.051030` | `0.596345` | `0.708011`     | `0.510705`     | `0.579403` |\n| X Learner    | `0.328857` | `2.585276` | `0.105298`     | `0.691134`     | `0.679522` |\n| TarNet       | `1.293894` | `5.866852` | `0.314323`     | `1.102849`     | `1.053019` |\n| DragonNet    | `1.506751` | `4.659946` | `1.168565`     | `2.607162`     | `1.916093` |\n| RA Net       | `4.501656` | `14.892828`| `1.129872`     | `4.681562`     | `3.883391` |\n| Forest DR Learner     | `424.962053`| `25.474570`| `2.920928`    | `3.989793`     | `3.892557` |\n| Forest DML   | `554.478033`| `32.682874`| `3.836490`    | `5.007255`     | `5.222570` |\n| BART                  | `4.509301` | `5.668311` | `9.506232`     | `9.264590`     | `8.637354` |"}, {"review_id": "z8PvZiY4uz", "reviewer": "Reviewer_drth", "summary": "This paper introduces Causal Prior-Fitted Networks, an extension of PFNs to the problem of causal effect estimation. The authors reframe the PFN framework for causal tasks and sketch a proof of consistency of the estimates. The paper makes use of an efficient data generation approach relying on real world datasets and adding appropriate variables for causal tasks. The experiments show the versatility of the framework and overall outstanding performance compared to relevant baselines.", "questions": "- Does the PFN also capture data-dependent uncertainty? Ie. do you observe the model becoming more confident with more data in the observational dataset?\n- What are the limits in terms of number of covariates? Data samples? ..?\n- How did you ensure that your training datasets did not contain data that you tested on?\n- Could this be extended into handling time series tasks?\n\n\n- L58: “Fitted on the ignorability assumption” - How do you handle real-world situations where the data might not come from a DGP that fulfils this criterion?\n- L158 “ATEs… averaging across units in D_obs” - How would this work?\n- L 179: typo -> illustrated\n- L192 / data generation: Is positivity really guaranteed? You could still generate datasets where on treatment is never sampled. Is this checked? What about edge cases where the random column chosen for Y0/1 is the copy of a column in X?\n- How would this work for continuous treatments? I guess the method itself should work and it’s “only” a data generation issue?\n- L207 How is the histogram handled for prediction? What are the bin locations? Is the data normalised to ensure equivalent bins across datasets?\n- L227ff: Could you define the metrics properly? Why are relative errors directly averageable and PEHEs not? Why not use relative errors for CATE too? \n- L243: You could still get a CATE on RCTs depending on treatment assignments? I guess here you’re talking about ITEs almost? Should be specific here.\n- Eq 10: What’s \\lambda (not \\lambda(q))?\n- Table2: How are the scores normalised?\n- Table 3: Why does fine-tuning decrease its performance?\n- A comparison to other causal zero-shot methods like [1] would have been great.\n\n[1] Zhang, Jiaqi, et al. \"Towards causal foundation model: on duality between causal inference and attention.\" arXiv preprint arXiv:2310.00809 (2023).", "limitations": "mostly. see questions", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "+ The paper is well structured and written, making it easy to follow.\n+ The paper offers a convincing and supposedly easy to use method for real-world causal effect estimation that has been evaluated on a wide range of synthetic, semi-synthethic and real tasks.\n+ The data generation procedure is simple yet effective ensuring that the datasets are not bound to certain functional relationships between covariates and outcomes.\n+ The evaluation touches upon interesting properties such as the calibration of the model.\n\n- Some areas of the paper are lacking in clarity. E.g. the metrics should be better defined.\n- Comparisons to other amortised causal inference methods are lacking and would further support this work.", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thoughtful review, insightful questions, and their positive assessment of the paper, as well as their view that CausalPFN is both convincing and easy to use.\n\n### More Confident Model With More Data\n\nCausalPFN becomes more confident as it observes more data. As an example, we use the synthetic polynomial datasets described in L1262–L1275 of the appendix and provide the length of the 95% credible interval of CATE estimates from CausalPFN, along with the confidence interval derived from a DR-Learner:\n\n**Table: The Effect of Sample Size on the Length of the 95% Credible Interval**\n\n| Method\\Sample Size | 10 | 20 | 50 | 100 | 200 | 500 | 1000 | 5000 | 10000 |\n|---|---|-|-|---|---|---|--|--|--|\n|**CausalPFN**|12.67±0.46|6.40±0.33|4.74±0.11|3.98±0.08|2.98±0.08|2.01±0.04|1.67±0.03|0.98±0.04|0.97±0.03|\n|DR-Learner|5.25±0.68|2.49±0.15|1.20±0.03|0.83±0.01|0.73±0.02|0.69±0.01|0.69±0.01|0.58±0.01|0.52±0.01|\n\nAlthough more conservative than the doubly-robust learner, the length of CausalPFN’s credible interval goes down as we observe more samples. Moreover, to ensure well-calibration, we also calculate the integrated coverage error (ICE):\n\n**CausalPFN’s ICE for Different Sample Sizes**\n\n|10  | 20| 50  | 100  | 200 | 500 | 1000 | 5000 | 10000 |\n|---|---|---|---|--|--|---|---|---|\n|0.13±0.01|0.01±0.01|0.01±0.01|0.04±0.00|0.02±0.01|0.05±0.00|0.03±0.00|-0.01±0.01|-0.04±0.01|\n\n### The Limit of Covariate and Sample Size\n\nDuring the causal phase of the training, we consider sample sizes up to 2048 and covariates up to 99. However, during inference, CausalPFN can take $50{,}000$ (Table 2) or more (Table 5).\n\nTo assess the effect of sample/covariate size on CausalPFN’s performance, we provide additional experiments. Similar to above, we focus on the synthetic polynomial datasets. We fix the test size at 100 samples in all experiments, and for each setting of sample size and covariate size, we report the average±standard error over 50 tables generated from the polynomial prior with a fixed random seed.\n\n**1) Effect of Sample Size**: Here, we consider the same data-generating process while increasing the number of samples. We fix the number of covariates to 10. The table below shows the PEHE values for CausalPFN’s CATE compared to other baselines. It is clear that CausalPFN has a better ability to utilize large sample sizes, as the PEHE drops faster.\n\n| Method\\Sample Size | 10  | 20| 50 | 100 | 200 | 500 | 1000 | 5000  | 10000 |\n|--|--|---|--|---|--|---|---|----|--|\n| **CausalPFN**  | 1.34±0.02| 1.27±0.02| 1.10±0.02| 0.89±0.02| 0.74±0.03| 0.46±0.01| 0.40±0.01| 0.29±0.01| 0.31±0.01|\n| DA Learner| 1.33±0.02| 1.30±0.02| 1.16±0.01| 1.00±0.01| 0.91±0.03| 0.85±0.01| 0.85±0.02| 0.84±0.02| 0.82±0.02|\n| S Learner | 1.44±0.01| 1.40±0.02| 1.35±0.02| 1.21±0.02| 1.18±0.05| 1.07±0.03| 1.09±0.03| 1.00±0.04| 1.03±0.04|\n| T Learner | 1.33±0.02| 1.30±0.02| 1.15±0.01| 0.97±0.01| 0.88±0.02| 0.81±0.01| 0.83±0.01| 0.81±0.02| 0.81±0.02|\n| X Learner | 1.35±0.02| 1.32±0.02| 1.20±0.02| 1.04±0.01| 0.94±0.03| 0.87±0.01| 0.88±0.02| 0.87±0.02| 0.84±0.02|\n\n**2) Effect of Covariate Size**: Here, we fix the number of samples to 1000 and increase the number of covariates. Similar to above, the table below compares the performance of CausalPFN to other methods in terms of PEHE. Although CausalPFN consistently maintains a performance gap, the gap becomes smaller as the number of covariates increases. This is likely due to the limited number of covariates during training (up to 100) and can potentially be addressed by increasing the number of covariates during training.\n\n| Method\\\\# of Covariates | 1 | 5 | 10 | 20 | 50 | 100 | 200 | 500  | 1000 |\n|-|--|--|--|--|--|--|--|--|--|\n| **CausalPFN** | 0.08±0.00| 0.17±0.01| 0.40±0.01| 0.67±0.01| 0.87±0.02| 1.01±0.02| 1.17±0.02| 1.28±0.02| 1.32±0.02|\n| DA Learner| 0.21±0.01| 0.59±0.01| 0.85±0.01| 1.04±0.01| 1.14±0.01| 1.22±0.01| 1.25±0.01| 1.30±0.02| 1.32±0.02|\n| S Learner | 0.54±0.04| 0.83±0.02| 1.09±0.02| 1.17±0.02| 1.21±0.02| 1.23±0.01| 1.26±0.02| 1.30±0.02| 1.33±0.02|\n| T Learner| 0.27±0.01| 0.60±0.01| 0.83±0.01| 0.98±0.01| 1.09±0.01| 1.18±0.01| 1.23±0.02| 1.28±0.02| 1.32±0.02|\n| X Learner | 0.29±0.01| 0.64±0.01| 0.88±0.01| 1.06±0.01| 1.15±0.01| 1.23±0.01| 1.26±0.02| 1.30±0.02| 1.33±0.02|\n\n### Ensuring Training Datasets Did Not Contain Test Data\n\n1. Most of the pre-training data used in our model are purely synthetic, following a similar approach to TabPFN [42]. We will publish the training code, along with the pre-training data, for the final version.\n\n2. For the real-world part of the pre-training data, we only use real-world covariates and generate the treatments through synthetic functions and add synthetic noise to the CEPO columns to generate potential outcomes.  We included the list of real-world datasets (OpenML datasets) in footnote 2 on page 28 of the appendix, which shows that none of the benchmarks are part of those datasets. \n\n### Extension to Time Series Tasks\n\nPotentially, there have already been a few attempts to use in-context learners on time-series data [B, C].\n\n[B] Hoo, Shi Bin, et al. \"From Tables to Time: How TabPFN-v2 Outperforms Specialized Time Series Forecasting Models.\" arXiv:2501.02945\n\n[C] Taga, Ege Onur, et al. \"TimePFN: Effective multivariate time series forecasting with synthetic data.\" AAAI 2025.\n\n### L58: The Ignorability Assumption\n\nCausalPFN is only applicable for the ignorability assumptions where identifiability holds. In most cases with no ignorability (i.e., with unobserved confounding), the causal effects can be generally unbounded. This is why many standard causal effect estimators are based on this assumption, and rely on a domain expert to assess whether ignorability holds.\n\n### L158: ATE Calculation by Averaging Across Units in $\\mathcal{D}\\_\\text{obs}$\n\nWe run the model to estimate CATE for each observed unit and calculate the average of all to estimate the ATE.\n\n### L192: Is Positivity Guaranteed? \n\nPositivity is guaranteed in the population as the treatment logits go through a Sigmoid function to generate propensities $P(T=1|X) \\in (0, 1)$. Still, there is a small chance that the data will only contain one treatment due to finite samples. We explicitly discard those edge cases during training.\n\n### Continuous Treatments\n\nIn principle, the CEPO-PPDs are still well-defined and the causal data-prior loss should still yield valid CEPO-PPDs. The challenge here, as correctly mentioned by the reviewer, is to implement a diverse and smooth data generation pipeline.\n\n\n### L207: Histogram Loss\nAs provided in Section B.2 L1175-L1179 of the appendix, the data is normalized first, then 1024 equidistant bins are allocated from -10 to +10; every point outside of this [-10, +10] is treated as an outlier and clamped. \n\nOnce the prediction is done in [-10, +10], we use the stored scale and shift parameter using for normalization to undo it and bring it back to the original scale.\n\n### Metric Definitions\n\nFor CATE evaluation, for a test dataset of size $N$ and ground-truth CATEs $\\tau$ alongside their estimations $\\hat{\\tau}$, the evaluation metric, PEHE, is defined as:\n\n$$\\text{PEHE}(\\hat{\\tau}) = \\sqrt{\\frac{1}{N} \\sum_{n=1}^{N} (\\hat{\\tau}(\\textbf{x}^{(n)}) - {\\tau}(\\textbf{x}^{(n)}))^2}$$\n\nFor ATE evaluation, given a ground-truth ATE $\\lambda$ and its estimation $\\hat{\\lambda}$, the relative error is defined as: $\\frac{|\\lambda - \\hat{\\lambda}|}{|\\lambda|}$.\n\nWe will include these equations in the revised version.\n\n> Why are relative errors directly averageable and PEHEs not? \n\nPEHE depends on the scale of CATE values. Hence, averaging across different datasets would be biased towards datasets with larger output values, while the scale of ATE is normalized in calculating the relative error. For more consistency, we will update Table 1 with average rank instead of average relative errors for ATE estimation, similar to CATE.\n\n> Why not use relative errors for CATE? \n\nPEHE is a standard metric used for CATE estimation in the causal inference literature. Moreover, the CATE error is defined as an average metric over $N$ samples, while the ATE error is defined only for one value per dataset. \n\n### L243: ITE vs CATE\n\nWe consider CATE, but the conditioning variables consist of all the observed covariates. In that sense, it is close to ITEs (though we still estimate an expected value). We will clarify this in the paper.\n\n### Eq 10: What is $ \\lambda$?\n\n$\\lambda$ coincides with ATE in equation (2) (same notation). We will recall its definition for more clarity.\n\n### Table2: Score Normalization\n\nWe normalize each column by dividing all values by the maximum score in that column. This normalization enables us to compute an average performance metric across datasets\n\n### Table 3: Fine-tuning vs. Performance\n\nIn Table 3, the first row corresponds to CausalPFN, while the second and third rows report results for a different underlying architecture (TabPFNv2): one without fine-tuning (row 3) and one with fine-tuning on our causal prior (row 2). In all cases, fine-tuning improves performance when comparing the second and third rows.\n\n### Comparison to \"Towards causal foundation model”\n\nWe have already briefly discussed this method (see [110]). Unfortunately, we could not find any implementation for this method. Moreover, as mentioned in Section 5.3 in [110], the zero-shot version of their method, CInA (ZS), is only applicable for benchmarks that naturally come with multiple datasets and there is no clear instruction on how to use the zero-shot version for standard casual effect estimation tasks akin to the ones we have used."}, {"review_id": "l1T52IHFjU", "reviewer": "Reviewer_8PoL", "summary": "The paper proposes and clearly evaluates a scalable amortized approach for causal effect estimation. The high level idea of using amortized inference has been very successfully applied in causal discovery and this work is one of a couple of concurrent works which explore this idea for cause effect estimation. \n\nThe paper is very nicely written, experiments are extensive. The combination of amortization and PFNs makes sense and should be explored. This will be a tool which hopefully can make it into the toolbox of practitioners.", "questions": "Why do you only gather identifiable DAGs? Would you not want the distribution over all equivalent DAGs no matter if they are identifiable or not? This would probably give you less clean estimates but would that not better? I.e. right now the argument seems that you only care about a DAG if it is identifiable while in the real world that might not be the case. \n\nWhy do you only provide inference code and not training code?", "limitations": "The theoretical framework for causal inference from observational data is unfortunately relatively weak but I do not see much that the authors can do about it. \n\nOverall if training code and the PyPy package is fully released I think it is a pretty good framework to build on top e.g. including interventions, other treatment forms etc.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "[1,2,3] are very closely related papers. Some even doing causal inference via PFNs similar to the current work. While all of these papers have been very recent (most of them actually only available after the NeurIPS deadline), it might still be good to discuss these approaches at least or potentially even compare e.g. to [3]. I am already giving the current paper a very high score so I am aware that prior to the submission the authors could not even have known the papers. \n\n\n[1]Bynum, L. E., Puli, A. M., Herrero-Quevedo, D., Nguyen, N., Fernandez-Granda, C., Cho, K., & Ranganath, R. (2025). Black box causal inference: Effect estimation via meta prediction. arXiv preprint arXiv:2503.05985.\n\n[2]Robertson, J., Reuter, A., Guo, S., Hollmann, N., Hutter, F., & Schölkopf, B. (2025). Do-PFN: In-context learning for causal effect estimation. arXiv preprint arXiv:2506.06039.\n\n[3] Ma, Y., Frauen, D., Javurek, E., & Feuerriegel, S. (2025). Foundation Models for Causal Inference via Prior-Data Fitted Networks. arXiv preprint arXiv:2506.10914.", "was_revised": true, "final_justification": "I have nothing to add to my review. It is a good paper and should be accepted.", "rebuttal": "We thank the reviewer for their positive feedback and encouraging comments. We are glad they found the paper well-written, the experiments extensive, and the combination of amortization and PFNs promising. We also appreciate their view that this work could become a valuable tool for practitioners.\n\n> [1,2,3] are very closely related papers. Some even doing causal inference via PFNs similar to the current work. While all of these papers have been very recent (most of them actually only available after the NeurIPS deadline), it might still be good to discuss these approaches at least or potentially even compare e.g. to [3].\n\nThank you for pointing out these related works. Except for [1], which is already discussed in the paper as reference [14], the other papers first became public on arxiv approximately **one month after the conference submission deadline**. Among the mentioned papers, [3] does not provide any code for using their model and mainly benchmarks on synthetic datasets that we do not have access to. Similarly, we could not find any code for the method in [1]. Specifically, as mentioned in Section 3.6 (page 9) of [1], the method requires having access to the underlying family of data-generating processes for each benchmark (e.g., for IHDP, they use 13,000 training datasets sampled from the underlying outcome surface model). In contrast, CausalPFN provides a zero-shot estimator. Below, we compare to Do-PFN [2], which is the most similar method to ours.\n\nWe use the IHDP, ACIC, and Lalonde datasets for both ATE estimation (relative error) and CATE estimation (PEHE). Note that Do-PFN is also a zero-shot method so we do not run any hyperparameter tuning or pre-processing on the datasets. \n\n**1) Mean PEHE ± Standard Error (↓ better)**\n\n| Method | IHDP | ACIC 2016 | Lalonde CPS (×10³) | Lalonde PSID (×10³) |\n|----|---|---|---|--|\n| **CausalPFN** | **0.58±0.07** | **0.92±0.11** | **8.96±0.02** | **14.40±0.20** |\n|Do-PFN| 6.07±0.89 |  4.11±0.52 | 12.01±.03 | 20.91±0.14|\n\n**2) Mean ATE Relative Error ± Standard Error (↓ better)**\n\n| Method | IHDP | ACIC 2016 | Lalonde CPS | Lalonde PSID |\n|--|--|--|----|----|\n| **CausalPFN** | 0.20±0.04| 0.05±0.01 | **0.13±0.01** | **0.22±0.02** | \n|Do-PFN|0.57±0.10| 0.67±0.04| 0.87±0.01|0.92±0.01|\n\n\nAs it is clear from both tables, CausalPFN achieves superior performance in all the benchmarks. We hypothesize that this is due to the non-identifiability of the prior data that they use, which we will elaborate in the next part. \n\n\n> Why do you only gather identifiable DAGs? Would you not want the distribution over all equivalent DAGs no matter if they are identifiable or not? This would probably give you less clean estimates but would that not better? I.e. right now the argument seems that you only care about a DAG if it is identifiable while in the real world that might not be the case. \n\nWe have the following reasons to only include identifiable data-generating processes (DGP) in our prior generation:\n\n1. Non-identifiability of the DGP means that for the _same observational distribution_, we can get an _unbounded set_ of causal effects. Therefore, if we allow for non-identifiable settings, in the worst case, we can have DGPs that all agree on the observational data (the only available information during inference) while having a range of $(-\\infty, \\infty)$ for the causal effects, which lacks any useful insights. Specifically, this worst-case scenario is more likely when generating large-scale prior data. This is perhaps one of the reasons we see worse performance for Do-PFN compared to our CausalPFN, as demonstrated in the previous part, since Do-PFN uses non-identifiable prior data, potentially having unobserved confounding.\n\n\n2. More concretely, we have provided an updated version of Proposition 1 here, that states that **identifiability is necessary for convergence** of the posterior-predictive distribution of causal effects (the quantity that CausalPFN aims to learn) in the limit:\n\n\n**Proposition 1** (Informal). Under mild regularity assumptions, for almost all $\\psi^\\star \\sim \\pi$ and any set of i.i.d. samples\n$\\mathcal{D}\\_\\text{obs}\\sim\\mathrm{P}^{\\psi^\\star}\\_\\text{obs}$, we have that as $|\\mathcal{D}\\_\\text{obs}|\\to\\infty$,\n\n$$\\begin{align}\n  \\mathbb{E}^{\\pi^{\\mu\\_t}} \\bigl[\\mu \\mid \\textbf{X},\\mathcal{D}\\_\\text{obs}\\\\bigr]\n  \\overset{a.s.}{\\longrightarrow}\n  \\mu_t(\\textbf{X}; \\psi^\\star), \\quad \\forall t \\in \\mathcal{T},\n  \\quad \\textbf{X}\\sim\\mathrm{P}^{\\psi^\\star},\n\\end{align}$$\n\n**if and only if** the support of the prior $\\pi$ only contains $\\psi$ with **identifiable** CEPOs almost everywhere.\n\nWhat this theory directly implies is that the identifiability (or in our case ignorability) requirement of the DGPs is not only sufficient for consistent estimation of the causal effects, but it is also a _necessity_. We have a formal proof for this proposition, which we will include in the final version of the paper. We would also be happy to provide the proof during the discussion period if requested.\n\n> Why do you only provide inference code and not training code?\n\nWe will release the training code, along with the synthetic prior data used for the training, with the published version of the paper."}, {"review_id": "8V9uLVeIEd", "reviewer": "Reviewer_URzh", "summary": "PFN is a method that trains a Transformer to input a training dataset (X_train, y_train) and input data X_test for inference, and then outputs the inference result y_test. The authors applied this method to causal effect inference and proposed a method for creating a set of training datasets that satisfy strong ignorability assumption. Specifically, two columns of the explanatory variables in the data are designated as latent outcomes (Y0, Y1), and a random propensity score is used to generate T from the remaining variables, with the corresponding latent outcomes Y_T designated as factual outcome Y.\nThe learned model achieves higher accuracy than several existing methods and is also faster (benefiting from the PFN-type model where the learner is pre-trained).", "questions": "[Q1] Since quite high accuracy is reported for IHDP CATE and Lalonde CPS ATE, is there any concern that the pre-training data might contain these datasets? How is it possible to confirm that such dataset leaks have not occurred?\n\n[Q2] Is it possible to train a model that directly estimates CATE instead of CEPO? That is, a transformer that takes (X_train, T_train, Y_train, X_test) as input and outputs τ_test? In causal inference, it is important to focus on the causal effect τ itself, which is often relatively smaller than the variation of Y on X (since we focus only on interventionable factors as $t$). The proposed method selects Y1 and Y0 randomly from X, which does not reproduce such a situation.", "limitations": "A limited amount of training data is inherent in the PFN approach, which is properly discussed.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\n\nIt is significant to demonstrate that the PFN approach is also effective for causal inference.\n\n## Weaknesses\n\n[W1] Baselines are somewhat outdated, and thus, the benefit in performance is unclear. Although they used the CATENets library, they did not compare SOTA DNN-based methods in it, such as SNet and FlexTENet.\n\n[W2] Causal inference usually involves statistical performance issues, and computational performance is not particularly important in practical terms. The advantage at this point may not be appealing to readers.\nTherefore, speed alone is not a compelling reason for adoption.\nAuthors might want to add some explanations of use-cases where speed is important, or of the value added by high-speed inference.", "was_revised": true, "final_justification": "The main concern regarding the outdated baselines is properly addressed, adding FlexTENet and SNet.", "rebuttal": "We thank the reviewer for their thoughtful review and constructive feedback. Below, we have provided detailed responses to each of the concerns raised. If our clarifications address these points satisfactorily, we would be very grateful if you consider raising the score.\n\n\n> Baselines are somewhat outdated, and thus, the benefit in performance is unclear. Although they used the CATENets library, they did not compare SOTA DNN-based methods in it, such as SNet and FlexTENet.\n\nWe have used two widely-used libraries for heterogeneous treatment effect estimation: EconML (4.2k stars on Github), and CATENets (another widely used library), in addition to more classical R-based baselines like GRF and BART. Moreover, we tuned the hyperparameters carefully for most of the baselines. Based on the reviewer’s suggestion, we also provide additional baselines from the CATENets library, namely SNet, FlexTENet, as well as XNet.\n\nHere, we compare CausalPFN to these additional baselines on the IHDP, ACIC, and Lalonde datasets for both ATE estimation (relative error) and CATE estimation (PEHE). We perform hyperparameter tuning similar to that used for the other baselines in the CATENets library, and still show competitive performance often exceeding that of these new baselines.\n\n**1) Mean PEHE ± Standard Error (↓ better)**\n\n| Method | IHDP | ACIC 2016 | Lalonde CPS (×10³) | Lalonde PSID (×10³) |\n|---|--|---|----|---|\n| **CausalPFN** | **0.58±0.07** | **0.92±0.11** | **8.96±0.02** | **14.40±0.20** |\n|SNet|2.12±0.13 | 1.46±0.17 |12.27±0.09 | 15.42±0.45|\n|FlexTENet|2.08±0.11|2.30±0.26|10.33±0.10| 17.18±0.73|\n|XNet|3.02±0.34| 2.34±0.26| 12.11±0.18    | 20.22±1.00  |\n\n**2) Mean ATE Relative Error ± Standard Error (↓ better)**\n\n| Method | IHDP | ACIC 2016 | Lalonde CPS | Lalonde PSID |\n|---|---|----|---|--|\n| **CausalPFN** | 0.20±0.04| 0.05±0.01 | **0.13±0.01** | **0.22±0.02** | \n|FlexTENet|0.21±0.04    |**0.04±0.01**    |0.53±0.02    |0.40±0.06|\n|SNet|0.21±0.04    |**0.04±0.01**    |0.79±0.02| 0.25±0.03|\n|XNet|**0.20±0.03**|0.06±0.02    |0.63±0.03|0.49±0.08|\n\n\n> Causal inference usually involves statistical performance issues, and computational performance is not particularly important in practical terms. The advantage at this point may not be appealing to readers. Therefore, speed alone is not a compelling reason for adoption. Authors might want to add some explanations of use-cases where speed is important, or of the value added by high-speed inference.\n\nFirst, we note that speed is not the main advantage of CausalPFN. As shown in Figure 1, as well as Tables 1 and 2, CausalPFN provides superior performance (for CATE estimation) on average, even without taking the computational performance into account.\n\nSecond, while it is true that many traditional causal inference applications prioritize statistical performance, a large number of real-world industry use-cases require fast/adaptive causal estimators. For instance, real-time bidding systems must estimate the incremental effect of ads and make bidding decisions with minimum latency [A]. Another example is e-commerce personalization systems that rely on fast uplift models to tailor recommendations in the course of a short user session, where latency directly impacts conversion rates [B].\n\nTherefore, even though speed is not a primary advantage of our method, we find it to be beneficial in many practical scenarios. We will add further clarifications and add some discussion to further clarify this in our revision.\n\n[A] Yuan et al. \"Real-time bidding for online advertising: measurement and analysis.\" Proceedings of the seventh international workshop on data mining for online advertising. 2013.\n\n[B] Sawant et al. \"Contextual multi-armed bandits for causal marketing.\" arXiv:1810.01859.\n\n\n> [Q1] Since quite high accuracy is reported for IHDP CATE and Lalonde CPS ATE, is there any concern that the pre-training data might contain these datasets? How is it possible to confirm that such dataset leaks have not occurred?\n\nThis is quite an important concern for all large-scale training paradigms, including CausalPFN. We made sure that none of the benchmarking datasets was part of the pre-training data based on the following:\n\n1. Most of the pre-training data used in our model are purely synthetic, following a similar approach of using randomly generated MLPs as in TabPFN v1 [42]. Specifically, no information regarding the benchmarks is used during this synthetic generation. \n\n2. For the real-world part of the pre-training data, we only use real-world covariates and generate the treatments through synthetic functions and add synthetic noise to the CEPO columns to generate potential outcomes. In that sense, there is only a chance for covariate leakage. However, we have also published the list of real-world datasets (OpenML datasets) in footnote 2 on page 28 of the appendix, which shows that none of the benchmarks are part of those datasets. \n\n> [Q2] Is it possible to train a model that directly estimates CATE instead of CEPO? That is, a transformer that takes (X_train, T_train, Y_train, X_test) as input and outputs τ_test? In causal inference, it is important to focus on the causal effect τ itself, which is often relatively smaller than the variation of Y on X (since we focus only on interventionable factors as ). The proposed method selects Y1 and Y0 randomly from X, which does not reproduce such a situation.\n\n\nYes, our framework allows us to train CausalPFN to directly estimate CATE values instead of separate estimation of CEPO values $\\mathbb{E}[Y_1 | X]$ and $\\mathbb{E}[Y_0|X]$. We chose to estimate CEPOs instead of CATE values mainly for the following reason: when calibrating the estimator, we often do not have access to ground-truth CATE values but factual observations, which can correspond to $Y_0$ or $Y_1$ based on the treatment value. Therefore, with the current choice, we can use the observational data to calibrate the model, as we detail in lines 271-307. Ultimately, estimating CEPOs can easily lead to many well-known causal estimands, such as CATE, ATE, local average treatment effects, and can potentially generalize to multiple-treatment settings."}]

Author rebuttal:

Meta-review: This paper introduces CausalPFN, a novel framework that amortizes causal effect estimation via prior-fitted networks and in-context learning. The method achieves strong zero-shot performance across standard benchmarks, is theoretically well grounded, and demonstrates practical utility by offering an out-of-the-box estimator without further tuning. Reviewers consistently found the paper novel, technically solid, and impactful, and all recommended acceptance.

The main points for improvement are relatively minor and can be addressed in the camera-ready version:
* Expand the discussion of very recent related works (e.g., concurrent PFN-based causal inference approaches) to better situate the contribution.
* Incorporate additional clarifications on scalability that were provided during rebuttal.
* Emphasize practical scenarios where computational speed is particularly beneficial.
* Ensure full release of both inference and training code, along with synthetic prior data, to guarantee reproducibility.
Decision: Accept (spotlight)