Title: Effortless, Simulation-Efficient Bayesian Inference using Tabular Foundation Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: simulation-based inference

foundation models

Bayesian inference

density estimation
Abstract: Simulation-based inference (SBI) offers a flexible and general approach to performing Bayesian inference: In SBI, a neural network is trained on synthetic data simulated from a model and used to rapidly infer posterior distributions for observed data. 
A key goal for SBI is to achieve accurate inference with as few simulations as possible, especially for expensive simulators. 
In this work, we address this challenge by repurposing recent probabilistic foundation models for tabular data: We show how tabular foundation models---specifically TabPFN---can be used as pre-trained autoregressive conditional density estimators for SBI. 
We propose Neural Posterior Estimation with Prior-data Fitted Networks (NPE-PFN) and show that it is competitive with current SBI approaches in terms of accuracy for both benchmark tasks and two complex scientific inverse problems. Crucially, it often substantially outperforms them in terms of simulation efficiency, sometimes requiring orders of magnitude fewer simulations. NPE-PFN eliminates the need for selecting and training an inference network and tuning its hyperparameters. We also show that it exhibits superior robustness to model misspecification and can be scaled to simulation budgets that exceed the context size limit of TabPFN.
NPE-PFN provides a new direction for SBI, where training-free, general-purpose inference models offer efficient, easy-to-use, and flexible solutions for a wide range of stochastic inverse problems.


Peer reviews:
comments: SUMMARY:
This work tackles an idea that has been gaining more and more traction in the AI4Science literature : how to leverage foundational models to accelerate science discovery? The authors use the recently proposed TabPFN architecture, initially conceived for univariate regression/classification problems, in a simulation-based inference setting. The main idea is to recast TabPFN as a univariate density estimator and to plug into an autoregressive factorisation of a possibly multivariate posterior distribution.

STRENGTHS AND WEAKNESSES:
The paper reads very well and is based on a rather popular and easily understandable engine, i.e. TabPFN. The way I see, the main methodological contributions of the authors were:
- Recasting TabPFN to SBI by leveraging the autoregressive factorisation of pdfs
- Despite the rather tedious and potentially slow sampling that entails from the pdf factorisation, the authors proposed a nice "ratio trick" to enable faster sampling from the trained approximate posterior at inference time

QUESTIONS:
1) I would have appreciated having a clearer explanation (maybe with a figure? diagram?) of how the sampling of the final posterior approximator is done. The approach is a bit convoluted envolving a classifier, TSNPE, TabPFN, rejection sampling, etc. It took me a bit of time to understand how each brick connects and I'm not sure that every reader would have the energy/time to do the same.

2) In the autoregressive factorization, how does the ordering of the choice of dimensions play a role? Is there an optimal way in practice? Or is the TabPFN model completely equivariant to the ordering of the input dimensions?

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 4
IMPACT: 3
ORIGINALITY: 4
DATE: 2025-06-19

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes using Tabular Foundation Models, specifically TabFPN for SBI. The new approach is evaluated on a few standard benchmarks and two more challenging models.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The idea is very cool!\n- The reported results are largely convincing, especially when it comes to simulation-efficiency.\n\nWeaknesses:\n\n- The used standard (low dimensional) benchmarks are a bit outdated and do not resemble many challenges we face in practice. I know they continue to be reused (because they are easy to fit well I assume) but I don’t see how results are of much use to judge the real-world applicability of the benchmarked methods. In addition, the evaluation of only 10 test data sets is a limitation especially since we apply amortized methods, where we should easily validate results on 100s of datasets.\n- Some of the results are presented in a way that it boarderlines on selective reporting/discussing. For example, the sampling times of the new method NPE-PF seems to be around 10-100 seconds per dataset (based on what I see in Figure 2). In contrast, NPE with normalizing flows requires only 0.1 seconds (or less?) per dataset. This is shown (in a very small manner) in Figure 2, the text says NPE-PF is slower than NPE. While true, the multiple orders of magnitude in speed difference are never explicitly mentioned. I suspect they could easily be missed by the unattentive reader. If we combined the slower training time of NPE with the faster inference time, when would NPE become overall faster again than NPE-PF (given similar calibration)?\n- The fact that a summary (embedding) network isn’t naturally fitting into the framework is another limtation of the approach. For almost all non-trivial data, we would want a summary network to handle the data structure properly. Yet, to use it in combination with NPE-PF, we have to pre-train it separately, reducing the argument of NPE-PF being “training-free”. How much additional training and simulation-budget do we need in this case? Is it substantially smaller (or faster) than training NPE+summary network combined? Also, I fail to find information in B.2 how good the “sufficient statistics” approximation described in EQ B-3 is. What are potential issues with this approach?\n\nQUESTIONS:\n- What is the inference time of the methods in the two real world case studies 3.3 and 3.4?\n- How fast is density evaluations of NPE-PF relative to NPE with normalizing flows?\n- Since we are using TabPFN in an autoregressive manner, is the number of features (only) d_x or d_x + d_theta(<j)? Since we seem to use both data and previous parameters as features? Perhaps you say it somewhere but it wasn’t entirely clear to me when reading the paper.\n- How good are the sbi pacakge defaults for the flows? Could it be that NPE is just less good the NPE-PF because the defaults are far from ideal? How much more efficient can things be if we tune the architectures to the given examples? I know this requires effort and expertise, but if we really case about simulation-budget, than this optimizing is probably the smallest evil. I am just worried that the better NPE-PF results are partially due to bad NPE defaults.\n- When indexing the pre-conditioning/training data, you sometimes use M and N. What is the difference between M and N?\n- In 2.2 You say, “predict the next dimension of data”. Do you mean “predict the next dimension of parameters”?\n- In the calibration results Figure D-3, could you include the confidence envelope assuming well calibration? And perhaps consider difference plots so one has an easier time judging (mis)-calibration which is currently very hard to see in the existing plot.\n- In MMS case study 3.2, I am not sure I understand what you considered the “reference”. Is it the analytical posterior of the misspecified model? Or is it some other target/reference? The text appears unclear in that regard.\n- In the limitations, you speak of “continuous parameters”. Since the benchmarks you use all have continuous parameters, I don’t understand what you mean by that term as a limitation?\n\nLIMITATIONS:\nA lot of limitations are clearly discussed although the magnitude of these limitations may be understated to some degree. See \"weaknesses\".", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper explores the use of in-context-learning (ICL) in SBI settings. It employs TabPFN as backbone model, avoiding the use of inference networks, for instance, in NPE (neural posterior estimation). The proposal employs a sequential inference step to handle models with multiple parameters and adopts nearest-neighbor-based filtering to overcome the limited context size of TabPFN. Experiments on synthetic and real data demonstrate that the proposal is more sample-efficient (in terms of number of simulations) and robust to model misspecification than baseline frameworks (e.g., NPE, NLE).\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n- Overall, the empirical results are promising and significant.\n- This is the first paper to explore ICL approaches in the context of SBI. \n- The paper is well-written and relatively easy to follow.\n\n\n**Weaknesses** \n- The paper does not provide in-depth insights into when and why the ICL approach works across different tasks or simulators, limiting the contribution to the community. Is TabPFN expected to beat NPE in all problems? What are the limits of TabPFN? Is there any relation between the datasets on which TabPFN was trained and the SBI tasks? The paper briefly mentions \"modeling complex parameter spaces—such as spatially varying or continuous parameters—remains challenging\". Where does this conclusion come from?\n\n- The paper aims for an \"effortless\" SBI approach. However, the paper does not mention how much time the proposal saves compared to the baselines. I believe the paper lacks a comparative study from a computational (time) perspective. In practice, how many hours would a practitioner save by using the proposed approach?\n\n- The paper would also benefit from an \"ablation-like\" study. For instance, how does the performance change as a function of the number of neighbors N? How many samples are filtered (on average)? This could provide more insights into how TabPFN works and its limits. \n\n- From a technical perspective, the paper brings little to the table. In particular, the application of the ICL approach is straightforward, and the solution used to overcome the limited context size  (i.e., nearest-neighbor-based filtering) is not novel.\n\nQUESTIONS:\n1. Could the authors elaborate on the sensitivity of the performance to some modeling choices, e.g., number of neighbors? \n2. What is the computational impact of the proposal? How much does it improve over standard NPE, for instance?\n3. On which datasets is TabPFN pre-trained? Could these datasets say something about settings (problems) in which the proposal work?\n4. What makes the TabPFN model the best foundation model for SBI? For instance, how does the method in [1] perform? \n\nPlease, see \"Weaknesses\" for additional questions.\n\n[1] TabICL: A tabular foundation model for in-context learning on large data?, ICML 2025.\n\nLIMITATIONS:\nThe paper includes a paragraph \"Limitations\". However, I believe the paper should better explore the limits of using TabPFN (or ICL in general) for SBI (see Weaknesses).", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces NPE-PF, a simulation-based inference method leveraging\nTabPFN as a pre-trained density estimator. The use of a tabular foundation\nmodel in the SBI pipeline enables training-free density evaluation, and builds\non useful prior knowledge from pre-training that can lessen the simulation\nburden required for producing accurate posterior approximations.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n- The paper is well-organized, includes a comprehensive literature review, and\n  features clear figures and diagrams.\n- Incorporating larger (foundation) models in SBI pipelines presents an\n  interesting opportunity for NPE workflows, and has not been sufficiently\n  explored in the literature to my knowledge. Further, while the idea of\n  \"swapping in\" a foundation model sounds simple on its face, the methodology\n  is carefully considered and attends to many important practical\n  considerations in its construction (e.g., employing efficient ratio-based\n  evaluation, adopting TSNPE, studying robustness to misspecification, etc).\n- The reported performance evaluation is convincing, comparing the proposed\n  method to relevant baselines on several common SBI tasks with clear\n  advantages across many of the explored settings.\n\n**Weaknesses**\n\n- Provided the differences in sampling costs between NPE-PF and flow-based NPE\n  methods, a more comprehensive analysis of computational costs (especially in\n  high-dimensional settings) would be helpful. In particular, it would be\n  practically useful to clarify the boundary beyond which one might consider a\n  computationally faster method (in total), even with NPE-PF's possible sample\n  efficiency gains and benefits of no training.\n- The evaluation of the method on real-world environments is somewhat limited.\n  While the Hodgkin-Huxley and pyloric network models are challenging tasks\n  outside the SBI benchmark suite [1], evaluation on inference tasks from a\n  wider range of scientific disciplines would better position NPE-PF's\n  general utility.\n- A principled analysis of NPE-PF's diminishing advantages over standard NPE\n  methods under increased simulation burdens would be useful to include. At\n  what upfront sample size is TabPFN only minimally comparatively informative\n  (or even a burden)? How does this change with the dimensionality of $\\theta$\n  and/or $x_o$? How does a well-selected embedding network shorten the\n  runway of the performance advantage? Characterizing these practical\n  considerations would be useful in determining NPE-PF's feasibility for use in\n  larger-scale SBI tasks. \n\n[1] Lueckmann, J. M., Boelts, J., Greenberg, D., Goncalves, P., & Macke, J.\n(2021). Benchmarking simulation-based inference. AISTATS.\n\nQUESTIONS:\n1. In regards to the point about inheriting TabPFN's properties: did you\n   encounter any particularly surprising failure modes or poorly represented\n   dynamics in your experiments with TabPFN? While TabPFN is a proven and\n   generally capable model, it would be interesting to characterize when/if it\n   systematically struggles under certain dynamics.\n2. In Figure D-7, permutations in the dimension have little effect on the performance. This was calculated over a set of the SBI benchmarks, with a maximum of 10-dimensional parameters. Was this tested on the 31-D pyloric network example? How large of an effect might this have on even higher-dimensional inference tasks?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-03"}