Title: CausalPFN: Amortized Causal Effect Estimation via In-Context Learning
Venue: NeurIPS 2025
Accepted: yes
Keywords: Causal Inference

Prior-Data Fitted Networks

Bayesian Inference

Ignorability

In-Context Learning

Meta-Learning

Automated Causal Inference
Abstract: Causal effect estimation from observational data is fundamental across various applications. However, selecting an appropriate estimator from dozens of specialized methods demands substantial manual effort and domain expertise. We present CausalPFN, a single transformer that *amortizes* this workflow: trained once on a large library of simulated data-generating processes that satisfy ignorability, it infers causal effects for new observational datasets out of the box. CausalPFN combines ideas from Bayesian causal inference with the large-scale training protocol of prior-fitted networks (PFNs), learning to map raw observations directly to causal effects without any task-specific adjustment. Our approach achieves superior average performance on heterogeneous and average treatment effect estimation benchmarks (IHDP, Lalonde, ACIC). Moreover, it shows competitive performance for real-world policy making on uplift modeling tasks. CausalPFN provides calibrated uncertainty estimates to support reliable decision-making based on Bayesian principles. This ready-to-use model requires no further training or tuning and takes a step toward automated causal inference (https://github.com/vdblm/CausalPFN/).


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nPFN is a method that trains a Transformer to input a training dataset (X_train, y_train) and input data X_test for inference, and then outputs the inference result y_test. The authors applied this method to causal effect inference and proposed a method for creating a set of training datasets that satisfy strong ignorability assumption. Specifically, two columns of the explanatory variables in the data are designated as latent outcomes (Y0, Y1), and a random propensity score is used to generate T from the remaining variables, with the corresponding latent outcomes Y_T designated as factual outcome Y.\nThe learned model achieves higher accuracy than several existing methods and is also faster (benefiting from the PFN-type model where the learner is pre-trained).\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\nIt is significant to demonstrate that the PFN approach is also effective for causal inference.\n\n## Weaknesses\n\n[W1] Baselines are somewhat outdated, and thus, the benefit in performance is unclear. Although they used the CATENets library, they did not compare SOTA DNN-based methods in it, such as SNet and FlexTENet.\n\n[W2] Causal inference usually involves statistical performance issues, and computational performance is not particularly important in practical terms. The advantage at this point may not be appealing to readers.\nTherefore, speed alone is not a compelling reason for adoption.\nAuthors might want to add some explanations of use-cases where speed is important, or of the value added by high-speed inference.\n\nQUESTIONS:\n[Q1] Since quite high accuracy is reported for IHDP CATE and Lalonde CPS ATE, is there any concern that the pre-training data might contain these datasets? How is it possible to confirm that such dataset leaks have not occurred?\n\n[Q2] Is it possible to train a model that directly estimates CATE instead of CEPO? That is, a transformer that takes (X_train, T_train, Y_train, X_test) as input and outputs τ_test? In causal inference, it is important to focus on the causal effect τ itself, which is often relatively smaller than the variation of Y on X (since we focus only on interventionable factors as $t$). The proposed method selects Y1 and Y0 randomly from X, which does not reproduce such a situation.\n\nLIMITATIONS:\nA limited amount of training data is inherent in the PFN approach, which is properly discussed.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes and clearly evaluates a scalable amortized approach for causal effect estimation. The high level idea of using amortized inference has been very successfully applied in causal discovery and this work is one of a couple of concurrent works which explore this idea for cause effect estimation. \n\nThe paper is very nicely written, experiments are extensive. The combination of amortization and PFNs makes sense and should be explored. This will be a tool which hopefully can make it into the toolbox of practitioners.\n\nSTRENGTHS AND WEAKNESSES:\n[1,2,3] are very closely related papers. Some even doing causal inference via PFNs similar to the current work. While all of these papers have been very recent (most of them actually only available after the NeurIPS deadline), it might still be good to discuss these approaches at least or potentially even compare e.g. to [3]. I am already giving the current paper a very high score so I am aware that prior to the submission the authors could not even have known the papers. \n\n\n[1]Bynum, L. E., Puli, A. M., Herrero-Quevedo, D., Nguyen, N., Fernandez-Granda, C., Cho, K., & Ranganath, R. (2025). Black box causal inference: Effect estimation via meta prediction. arXiv preprint arXiv:2503.05985.\n\n[2]Robertson, J., Reuter, A., Guo, S., Hollmann, N., Hutter, F., & Schölkopf, B. (2025). Do-PFN: In-context learning for causal effect estimation. arXiv preprint arXiv:2506.06039.\n\n[3] Ma, Y., Frauen, D., Javurek, E., & Feuerriegel, S. (2025). Foundation Models for Causal Inference via Prior-Data Fitted Networks. arXiv preprint arXiv:2506.10914.\n\nQUESTIONS:\nWhy do you only gather identifiable DAGs? Would you not want the distribution over all equivalent DAGs no matter if they are identifiable or not? This would probably give you less clean estimates but would that not better? I.e. right now the argument seems that you only care about a DAG if it is identifiable while in the real world that might not be the case. \n\nWhy do you only provide inference code and not training code?\n\nLIMITATIONS:\nThe theoretical framework for causal inference from observational data is unfortunately relatively weak but I do not see much that the authors can do about it. \n\nOverall if training code and the PyPy package is fully released I think it is a pretty good framework to build on top e.g. including interventions, other treatment forms etc.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Causal Prior-Fitted Networks, an extension of PFNs to the problem of causal effect estimation. The authors reframe the PFN framework for causal tasks and sketch a proof of consistency of the estimates. The paper makes use of an efficient data generation approach relying on real world datasets and adding appropriate variables for causal tasks. The experiments show the versatility of the framework and overall outstanding performance compared to relevant baselines.\n\nSTRENGTHS AND WEAKNESSES:\n+ The paper is well structured and written, making it easy to follow.\n+ The paper offers a convincing and supposedly easy to use method for real-world causal effect estimation that has been evaluated on a wide range of synthetic, semi-synthethic and real tasks.\n+ The data generation procedure is simple yet effective ensuring that the datasets are not bound to certain functional relationships between covariates and outcomes.\n+ The evaluation touches upon interesting properties such as the calibration of the model.\n\n- Some areas of the paper are lacking in clarity. E.g. the metrics should be better defined.\n- Comparisons to other amortised causal inference methods are lacking and would further support this work.\n\nQUESTIONS:\n- Does the PFN also capture data-dependent uncertainty? Ie. do you observe the model becoming more confident with more data in the observational dataset?\n- What are the limits in terms of number of covariates? Data samples? ..?\n- How did you ensure that your training datasets did not contain data that you tested on?\n- Could this be extended into handling time series tasks?\n\n\n- L58: “Fitted on the ignorability assumption” - How do you handle real-world situations where the data might not come from a DGP that fulfils this criterion?\n- L158 “ATEs… averaging across units in D_obs” - How would this work?\n- L 179: typo -> illustrated\n- L192 / data generation: Is positivity really guaranteed? You could still generate datasets where on treatment is never sampled. Is this checked? What about edge cases where the random column chosen for Y0/1 is the copy of a column in X?\n- How would this work for continuous treatments? I guess the method itself should work and it’s “only” a data generation issue?\n- L207 How is the histogram handled for prediction? What are the bin locations? Is the data normalised to ensure equivalent bins across datasets?\n- L227ff: Could you define the metrics properly? Why are relative errors directly averageable and PEHEs not? Why not use relative errors for CATE too? \n- L243: You could still get a CATE on RCTs depending on treatment assignments? I guess here you’re talking about ITEs almost? Should be specific here.\n- Eq 10: What’s \\lambda (not \\lambda(q))?\n- Table2: How are the scores normalised?\n- Table 3: Why does fine-tuning decrease its performance?\n- A comparison to other causal zero-shot methods like [1] would have been great.\n\n[1] Zhang, Jiaqi, et al. \"Towards causal foundation model: on duality between causal inference and attention.\" arXiv preprint arXiv:2310.00809 (2023).\n\nLIMITATIONS:\nmostly. see questions", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

comments: SUMMARY:
The paper introduces a new causal effect estimation method. The Authors leverage prior-fitted networks to amortize the estimation effort and increase both accuracy and efficiency. The method is theoretically justified to work under a strong ignorability assumption. Empirical results across multiple benchmarks demonstrate that the proposed approach performs competitively compared to a diverse set of baselines.

STRENGTHS AND WEAKNESSES:
**Strengths**:
1. The presented approach is novel, interesting, and well motivated by recent advances in generative modelling and the challenges in causal inference.
2. The paper is well written, and the core ideas are presented clearly and convincingly.
3. The experimental evaluation is extensive and sound.

**Weaknesses**:
1. Details regarding method scalability are not discussed. (see question 1)

QUESTIONS:
1. How large are the datasets used in terms of sample size and number of covariates?  How many covariates are considered both during training and evaluation? How does the method’s performance in terms of accuracy scale with the number of covariates compared to other methods?
2. Could you provide a version of Figure 1 that isolates inference time only, excluding training and hyperparameter search for the baseline methods? This would help in fairly assessing the computational efficiency of CausalPFN

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-03