Can Transformers Learn Full Bayesian Inference in Context?
Authors: Arik Reuter
Tim G. J. Rudner
Vincent Fortuin
David Rügamer
Venue: ICML 2025 poster (2025)
Keywords: In-Context Learning
Prior-Data Fitted Networks
Bayesian Inference
Primary area: probabilistic_methods
TL;DR: We show that transfomers can effectively solve full Bayesian inference in context.

Abstract: Transformers have emerged as the dominant architecture in the field of deep learning, with a broad range of applications and remarkable in-context learning (ICL) capabilities. While not yet fully understood, ICL has already proved to be an intriguing phenomenon, allowing transformers to learn in context—without requiring further training. In this paper, we further advance the understanding of ICL by demonstrating that transformers can perform full Bayesian inference for commonly used statistical models in context. More specifically, we introduce a general framework that builds on ideas from prior fitted networks and continuous normalizing flows and enables us to infer complex posterior distributions for models such as generalized linear models and latent factor models. Extensive experiments on real-world datasets demonstrate that our ICL approach yields posterior samples that are similar in quality to state-of-the-art MCMC or variational inference methods that do not operate in context. The source code for this paper is available at https://github.com/ArikReuter/ICL_for_Full_Bayesian_Inference

Can Transformers Learn Full Bayesian Inference In Context?
Arik Reuter! Tim G. J. R
Vincent Fortuin 345 David Rügamer 15
Abstract
tion answering or text summarization, using fixed model without requiring any gradient-based fine-tuning, simply by referencing the context. Thereby, ICL enables the genera- tion of real- real-time solutions through a localized understanding of data without explicit re-training (Dong et al., 2022; Garg et al., 2022).
Transformers have emerged as the dominant archi- tecture in the field of deep learning, with broad range of applications and remarkable in-context learning (ICL) capabilities. While not yet fully understood, ICL has already proved to be an in- triguing phenomenon, allowing transformers to learn in Sontext-Withhoe requiring further train- ing. In this paper, we further advance the under- standing of ICL by demonstrating that transform- can perform full Bayesian inference for com- monly used statistical models in context. More specifically, we introduce a general framework that builds on ideas from prior fitted networks and continuous normalizing flows and enables us infer complex posterior distributions for models such as generalized linear models and latent factor models. Extensive experiments on real-world datasets demonstrate that our ICL ap- proach yields posterior samples that are similar in quality to state-of-the-art MCMC or variational inference methods that do not operate in con- text. The source code for this paper is available at https:: /ttt:///gttttbb// com/Arikkeuter/ ICL for Full Saweeiian_nnfeeeeeeeeeeeeeeeeeee
A fundamental benefit of ICL with LLMs its versatility, Almost every NIPP task involving small data can be solved in context using LLMs, while the performance often surpasses existing baselines (Touvron et al., 2023; OpenAI, 2023; Anil et al., 2023). Additionally, achieving this performance can be very straightforward, requiring only suitably formulated prompts in natural language. Excellent results across a broad variety of tasks, combined with fast inference times and ease of usability, have made in-context learning a machine learning tool employed by millions of people (Eloundou et al., 2023).
Furthermore, ICL has recently shown remarkable promise for regression and classification tasks involving tabular data, with tabular prior-data fitted networks (TabPFNs) dominat- ing benchmarks alongside minimal prediction time (Holl- mann et al., 2022: 2025; Hoo et al., 2024; Robertson et al., 2024). While the internet serves the massive data needed to train in-context learners on text, TabPFNs demonstrate that training on purely synthetic data facilitates the development of in-context learners for tabular data.
suitable source for
1. Introduction
While PFNs perform Bayesian inference, they target uni- variate, typically discrete, posterior predictive distribution. In numerous applications, however, high-dimensional and continuous posteriors pa of (latent) variables z given data æ play a key role. This includes areas such as health- care (Kyrimi et al., 2021; Abdullah et al., 2022; Etzioni & Kadane, 1995), physics (Gebhard et al., 2025; Brehmer Cranmer, 2022: Dax et al., 2024), and neuroscience (Lueck- mann et al., 2017; Sohn & Narain, 2021). We use the notion of full Bayesian inference for methods yielding potentially complex and high-dimensional posterior distributions- contrast to, for instance, methods that yield only the pos-
In-context learning (ICL) has become fundamental prin- ciple in natural language processing (NLP) with large lan- guage models (LLMs) as ubiquitous in-context learners. The core principle of ICL is that system adapts to given task based information provided in its context. This ables the system to address complex problems, such as ques-
'Deparmment of Statistics, LMU Munich, Munich, Ger- many 'Center for Data Science, New York University, New York, USA 'Department of Computer Science, Technical Uni- versity of Munich, Munich, Germany Hellmholt AI. Mu- nich. Germany. 'Munich Center for Machine Learning (MCML), Munich, Germany. Correspondence to: Arik Reuter <arik.reuter@ mmmpus.
'We do not assume any specific form of z. That is, there be a single zj associated with each data point zy in æz, but the case where a single "global" governs the behavior of each æ, in æ equally included in this notation.
Proceedings of the 4nnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Can Transformers Learn Full Bayesian Inference In Context?
"Summary: The
fairytale about..."
tK tk
31
22
KK
"Summarize this text: Once upon
Dataset
time there was girl named..."
(a) ICL for text summarization using LLMs.
(b) ICL for full Bayesian inference.
(a) An LLM generates summary 81, $2.... of a text (1, l2,..., Ix through autoregressive sampling while referring to the context using masked self-attention. (b) A dataset æ is processed with a transformer encoder. Subsequently. cross attention allows generating samples from the posterior conditioned on æ in context using a diffusion transformer (decoder). The samples generated by solving neural differential equation defining a continuous normalizing flow.
Figure I:
terior predictive point estimates of the posterior as, for example Hollmann et al. (2022). However, performing full Bayesian inference can be challenging, even for relatively simple models such as generalized linear models (GLMs; Nelder Wedderburn, 1972). Two common issues when performing full Bayesian inference include (a) slow infer- ence time, particularly when using sampling-based methods (Sommer et al., 2025; 2024), and (b) model misspecification. Although potentially restrictive modeling assumptions are often necessary to make Bayesian inference efficient or even feasible, they can lead to suboptimal predictive performance (Wang& & Blei, 2019; Walker, 2013).
To summarize, main contributions follows:
1. We develop, train, and examine a model that yields samples from the posterior distribution p=|æ given data x as context without any (explicit) parameter updates or parametric assumptions about the posterior.
2.
To achieve this, we propose to use synthetic samples from the joint distribution pæ.* in order to train large transformer model that performs ICL regarding the pos- terior pa and provide a general framework to ana- lyze the cireumstances that enable learning p= purely through samples from pæ.z
3. We then analyze the efficacy of our approach for GLMs and latent factor models, namely Gaussian mixture mod- els (GMMs) and factor analysis (FA). For these appli- cations, we show that including the "prior" used for TabPFNs results in reliably inferring posterior distribu- tions on real-world data.
In this paper, we address the following question: Can leverage in-context learning to effectively perform full Bayesian inference? In doing so, aim obtain in- context learner that can perform the mapping æ pz for a specific probabilistic model, and, analogous to LLMs, (a) allows for the rapid generation of samples from a posterior of interest during deployment and (b) can flexibly adapt to a broad range of inputs, thereby overcoming issues arising from model misspecification. More specifically, our ap- proach combines TabPFN encoder (Hollmann al., 2022) and diffusion transformer-decode (Peebles & Xie, 2023) that is trained via flow matching (Lipman et al., 2022).
4. In a variety of experiments, we demonstrate that this ap- proach yields posterior samples that are very similar to those from Hamiltonian Monte Carlo sampler. Further- more, we find that the quality of the samples from ICI. approach is preferable, when compared to various popular VI techniques that do not operate in context.
We present the results of our in-context learning approach on extensive real-world and synthetic datasets in Section 4 and discuss the challenges and the transformative potential of in-context learning for full Bayesian inference in Section 5.
5. Finally, we conduct ablation studies of our approach, examining, for instance, alternative diffusion objectives and Gaussian approximations in place of flow matching, the model's performance on -isttri-ittiitoionn data, and the impact of problem dimensionality.
2

---

Can Transformers Learn Full Bayesian Inference In Context?
2. Related Work
contrast to our more general assumption, each datapoint my in is assumed to have corresponding latent variable Zj. While the parameter 0 determines global aspects of the variational distribution, the function h& is shared for all æ and thus amortized across data a. Variational autoencoders (Kingma, 2013; Rezende et al., 2014) and neural processes (Garnelo et al., 201 2a;;;;; Rudner et al., 2018) are important model classes based on amortized inference.
Beyond the perspective of prior-data fitted networks, the contribution of this work can be summarized from the view- points of recent work on in-context learning, amortized Bayesian inference, and, simulation-basee based inference.
In-Context Learning. ICL is a special case of meta- learning (Hospedales al., 2021) characterized by using a large pre-trained model in order to learn from a context dataset without explicitly updating task-specific parameters. Several recent lines of work investigate the in-context learn- ing capabilities of transformers (Garg et al., 2022; Ahuja et al., 2023; Wang et al., 2024; Chan et al., 2022).
In comparison, our ICL approach amortizes its parameters on the level of datasets, such that a single functional re- lationship is learned for a set D C (X' x 2) of datasets. From this point of view, D = { }; i-1 comprising N datasets I; E X and the corresponding latent variables 2, E 2 can be seen as a "meta-dataset" for which we per- form amortized inference. This is similar in nature to the setup by Le et al. (2017), who use recurrent neural net- works to "compile" inference based on execution traces of probabilistic programs by training on simulated data.
Garg et al. (2022) show that model similar to GPT-2 can implicitly implement various interesting function classes in context. More specifically, the model learns to reproduce the predictions of different statistical models such as (sparse) linear functions, decision trees, and even two-layer neural networks. This approach can be extended to multiple fam- ilies of functions and even mixtures of tasks (Ahuja et al., 2023). Kirsch et al. (2022) investigate ICL as a general prin- ciple for meta-learning However, the results by Garg al. (2022) and Ahuja et al. (2023) are restricted to relatively small problem scales and scalar-valued predictions instead of multivariate posterior distributions. Additionally, the ex- periments are conducted exclusively on simulated data, In contrast, our results show that (large) transformer models can effectively learn multivariate posterior distributions over latent variables in context on real-world datasets. Further- more, the focus on latent-variable models naturally steers our investigation toward unsupervised in-context learning, where the primary objective is to uncover the underlying structure of the data rather than to make predictions based on input-target paris presented in the context.
Unlike in amortized variational inference, we do not use the notion of an evidence lower bound (Blei et al., 2017) or even the Kullback- Leibler divergence to learn the posterior, but utilize ideas that also appear in the context of simulation- based inference.
Simulation-Basse Inference. Analogously to latent vari- able models, some scientific simulations, for instance in neu- roscience or astrophysics (Fan & Markram, 2019; Schmit & Pritchard, 2018), allow to draw samples from the joint dis- tribution p#,* of data and latent variable of interest. Amor- tized posterior inference in this context is referred to as simulation- inference (SBI; Cranmer et al., 2020). Sev- eral recent approaches focus on using neural networks to directly infer aspects of the likelihood p(z|z) z)) the poste- rior p=læ or the joint distribution p=.=. More specifically, techniques based on discrete normalizing flows (Dax et al., 2021) or flow-matching (Wildberger et al., 2024) are used to approximate the posterior p=|a while (Gloeckler et al., 2024) propose to use a transformer-based diffusion model in order to approximate the joint distribution p=.*. In recent work, Vetter et al. (2025) directly a pre-trained TabPFN to auto-regressively sample pal leveraging ICL.
Concurrently, Mittal al. (2025a) conduct comparative analysis of amortized in-context Bayesian posterior estima- tion methods, ablating over different optimization objectives and architectural choices, and also reporting results on out- of-distribution performance and flw-matching methods. They focus on evaluating the posterior mean and down- stream predictive performance, whereas we evaluate full posterior distributions. In an analogous setup, Mittal al. (2025b) assess the effectiveness of learning point estimates versus learning entire distributions in context for the goal of predictive performance.
From simulation-based inference viewpoint, we demon- strate that sample-based posterior estimation can be used for full Bayesian inference in complex scenarios arising in commonly used latent variable models, and demonstrate the effectiveness of this approach on real-world datasets.
Amortized Inference. Amortized inference is central paradigm in the field of variational inference (Kingma, 2013; Zhai et al., 2018; Kim et al., 2018; Margossian & Blei, 2023). A commonly used idea here is to model the posterior distribution p=|æ of latent variables x given a dataset x via a factorized density p(2|ap Re f (2, (mr,)) In
3. In-Context Learning for Full Bayesian Inference
Bayesian inference is a tool of central importance for count- less applications. However, exact posterior inference can
3

---

Can Transformers Learn Full Bayesian Inference In Context?
become computationally expensive when using sampling- based methods (Hastings, 1970; Hoffman et al., 2014; Be- tancourt, 2017) and even impossible when relying on fully factorized VI methods, which can incur substantial approxi- mation errors (Bishop et al., 2002; Blei, 2012; Margossian & Blei, 2023). Amortized variational inference can alleviate those issues but typically requires the development of spe- cialized and complex modeling frameworks (Kingma, 2013: Srivastava & Sutton, 2017; Garnelo et al., 201 8b; Lin et al., 2021). Another issue with variational inference arises from having to choose variational distribution. While insuffi- cient flexibility in this respect can lead to overly simplistic posteriors, a too flexible variational distribution might over- fit the given data (Cremer et al., 2018).
directly learn the mapping Jo X MM x + pale, where M(2) is the space of all probability measures. There- fore, we want a model fax = Q for the posterior to be as close as possible to the true posterior p= fo (z). We measure "closeness" w.r.t. some divergence d' M(2))xM(2) + [0, oo). When considering the expected divergence over data samples a P this gives rise to the following objective: RR := Pe~p(a) [d (fo(z)) f(()))), which can also be directly expressed as
[d (oz : p=lm))
Rg = Eewpl()
(I)
Note that we use the notion of a divergence d loosely to refer to any measure of similarity of two distributions. Although RS itself is usually intractable, specific choices of d and the use of the joint distribution pe.* make Equation (1) accessible via
We propose a simple and effective solution based on ideas from ICL, which can be seen conducting amortized infer- ence on dataset level. Training a model on potentially unlimited amount of synthetic datasets yields an in-context learner that can not only approximate a vast, almost arbitrar- ily large, class of distributions but is also highly efficient when used for sampling. Furthermore, this does not incur the same issues with overly or insufficiently flexible distri- bution assumptions that are present in VI. More specifically, empirical results show that a major strength of TabPFN, for instance, is its ability to adapt flexibly to the complexity of the problem at hand, thus removing the need for extensive hyperparameter tuning (Hollmann et al., 2022).
Ro:- Ex.xpp((.ss) [Ca( z, 2,]
(2)
where the loss function La depends on d and the structure of 2 (discussed in detail later). Performing empirical risk minimization for RS with samples from the joint dis- tribution pæ.z then corresponds to learning to approximate pziz The model for the posterior pz|= is thereby only implicitly defined by the joint distribution pæ,=, While this requires the ability to sample from p=.=, drawing samples from the joint distribution is often a weak requirement in terms of model specification that immediately follows from specifying the generative process of a model. Furthermore, simple sufficient condition that follows directly from the law of total expectation implies the equivalence of Ro and RØ:
In the following, we describe a sufficient general condi- tion, as well as a specific framework that allows to train probabilistic in-context learners on simulated data.
The idea underlying the proposed approach is founded on two observations relating to full Bayesian inference and the working principle of PFNs: First, many Bayesian models have a generative formulation that allows the simulation of arbitrarily large amounts of training samples from the joint distribution p=.z. We assume that samples from p=.= com- prise a dataset x 1 containing K samples x, X and a corresponding (latent) variable E 2.2 This joint distribution p=,= corresponds to the "prior" in PFNs and allows the training of a large neural network that implicitly learns to perform Bayesian inference. Second, Bayesian inference is especially useful for smaller datasets x that can be processed in single forward pass. This makes an entire dataset a viable context for Bayesian ICL.
Proposition 1. Let d( pzle)- = / (0% i dp=lm
for some measurable functional 7: M(Z)) + R. Then RS RR, with Ca z, 0) = (Q)
For instance, choosing d to be the forward Kullback- divergence dKL. (Q p=læ) 1Dx [r(1 [0 (-Izx) im- plies that L (x, z, 0) = - log go (z)x) + const. (Müller et al., 2021). In this case, minimizing R2 thus directly cor- responds performing maximum likelihood inference on samples from p=.=.
More specifically, the central goal is to develop method allowing to infer the posterior distribution palæ of latent variables z E Z, given observations x E X using ICL. From a pneervised-lcarning perspective, we thus aim to
3.1. Defining the Form of the Posterior
To learn the posterior distribution palæ in context, we use the framework of flow matching (Lipman et al., 2022). More specifically, we utilize continuous normalizing flows (CNFs) to specify and ultimately sample from p=|æ CNFs, cur- rently excelling in the field of image synthesis (Esser et al., 2024), do not only allow to flexibly learn almost arbitrary distributions, but are also found to be more sample-efficient
We do not assume any specific form of z. That is, there can be single zj associated with each data point æ; in æ. but the case where a single "global" z governs the behavior of each æy in is equally included this notation.
4

---

Can Transformers Learn Full Bayesian Inference In Context?
in training than for instance diffusion objectives (Lipman et al., 2022; Wildherger al., 2024). Furthermore, unlike discrete normalizing flows (Papamakarios et al., 2021a), CNF objectives do not limit the architecture of the used neu- ral network, allowing to incorporate complex conditioning on the data æ in addition to flexibly modeling the posterior, which is a crucial aspect of our ICL framework. Refer to Appendix D for more information on CNFs.
Assuming Gaussian conditional probability paths with an optimal- mean- and variance-function (Lipman et al., 2022), one obtains the following discrepancy measure deFM between Qo := ( Pss and pale:
(0% , p=lm)
dCFM
ti (7(2 (1=(0))))000)) - -(x(1) wz w (5)
E
3.1.1. I. NORMALIZING FLOWS
where the expectation is taken w.r.f. to three random vari- ables: a uniform time-step t ~ =[(O,, 1]), samples from the base distribution =0)) ~ Ps. and samples from the ground- conditional distribution 2(1) pz(a, We de-
The key idea of modeling distribution pxle with normal- izing flows (see, e.g., Papamakarios et al., 2021b), which are the basis of CNFs, is to assume that p=|æ is the result of "pushing forward" a simple base distribution PB into pala using a conditional flow (-(ax))
fine (=(11)= = (1 wf)= (0) (2)
We refer to (Wildberger et al., 2024) for mathematical results on the relationship of dCFM and the (forward) Kullback- Leibler divergence. The hyperparameter w = - min- where min is the variance at time = 1 in the Gaussian conditional probability paths, appears to have negligible in- fluence when set value sufficiently close one (Lipman et al., 2022).²
pala
( PB
(3)
Therefore, one assumes that samples from p=la are gener- ated by first drawing x0)) ~ PB. and then applying (-(()))),
such that (0) æ) p=læ The base distribution Ps
is commonly set to be a standard normal distribution, i.e.,
Ps = (O I). The conditional flow tho (-|zz) is the object to be learned, such that our model of pa is defined as Q = [<>(-|=)]
In order to make optimizing
Emep(a)) |drr (0g p
(6)
3.1.2. CONTINUOUS NORMALIZING FLOWS
tractable, and thus train our in-context learner, make use of the sufficient condition in Proposition 1. Thus, the divergence deFM admits the re-formulation as an objective R, using samples from the joint distribution p=.=, We can therefore optimize R2 using N independent and identically distributed (i.i.d)) samples f, ~ U([O, 1]) from the time- distribution, 2 Pg from the base distribution, and (z .x;) ~ pa,* from the joint distribution. With this, we obtain the following objective function used for the training of the ICL models:
In flow matching (Lipman et al., 2022), which we will use to obtain an in-context learner for full Bayesian inference, the normalizing flow the (-lar) is implicitly defined via (conditional) vector field of. of ordinary differential equation (ODE):
d (2.1(2/z) = - (1),, (0,0(2|=) = (4)
dt
where 1 S1S1. 1. The first condition 4 vo, (2(ar) of.... (v0,((2(=)) ))) means that ef describes the change in (z(x( at time 1, and the second condition (=)r)) implies that initially the flow is just the identity. The fam- ily of vector fields of_. is parameterized by a neural network whose parameters will be learned. In order to ultimately compute the flow uf... that yields 2 [ (-1x));PPr a numerical ODE solver can be used to forward-solve the ODE, which ultimately corresponds to evaluating a/,, a at a data point =0) ~ Ps. This construction implies very generic assumptions regarding the structure of Q² " which include the existence of a density of the target distribu- tion wrt. the Lebesgue measure, and the assumption that pajæ can be represented by mixture distribution over the marginal probability paths at time point t = 1 (Lipman et al., 2022). Please note that the mathematical understand- ing of Flow Matching, its properties and assumptions, are still actively researched (Wildherger et al., 2024).
NV Rs
Σ
2 ah (1,, (2 (2(")))))) (0) 20(0 wx
(0)
(7)
2
3.2. Sampling from the Joint Distribution
In order to learn model that can perform posterior in- ference according to Section 3.1, we require sample (z, z) ~ p=.=_ Given p(æ, x) p(æ ((x))))((2) this is al- ways possible as long as one can draw samples from p* and then from p=|= Hence, this is a relatively weak require- ment allowing for a broad variety of priors and observation models. More specifically, for ICL, we generate a training dataset DD which comprises i.i.d. samples {(x;, 2, )))² re- sulting from sampling 2; ~ p= and then æ; pais, We
IIn our experiments, follow (Wildherger al., 2024) and set w == - 14 for all experiments.
5

---

Can Transformers Learn Full Bayesian Inference In Context?
use this simple yet fundamental and very general template to generate samples from the joint pæ,* for GL.Ms, factor analysis (FA), and Gaussian mixture models (GMMs) in our later applications. Please refer to Appendix A for more details on the data generating processes.
(l =t)= + 1)
MLP with Conditioning
3.3. The Architecture
Scale
In order to implement the idea of learning full Bayesian inference in context, we extend ideas of diffusion trans- formers (Peebles & Xie, 2023), where the conditioning on the time is implemented via adaptive layer norm (adaLN) blocks initialized as the identity function. As we potentially require complex conditioning on the data ac, additional transformer encoder is added. The input to the decoder is vector in the form (1 - =t)=(0) + tz which is treated as a sequence with length one and processed by transformer decoder without self-attention but the adaLN blocks. There- fore, the decoder has an equivalent interpretation as a multi- layer perceptron with conneconnec,i cross- cooss-attention, and adaptive layer normalization. For the final processing in the decoder, only conditional feedforward layers with adaptive layer normalization are used. This corresponds exactly to the architecture of the decoder before, albeit without cross attention. We call this part an "MLP with Conditioning". Samples for the time / E [0, 1] are mapped onto condi- tioning vector using several fully connected layers, which yields a richer representation of that is well-suited as an input to the adaLN blocks. Figure 2 depicts of the resulting architecture.
Feed Forward
Scale and Shift
Norm
Ningera
Scale
Cross Attention
Scale and Shift
Encoder
Norm
MLP
(1 wt) z (0)
+ 1x
Architecture to perform ICL for full Bayesian inference. A relatively large transformer encoder, similar to that in TabPFN (Hollmann et al., 2022) processes a dataset x and yields a representation used in the decoder. The decoder outputs vector field defining flow for given input vector conditioned on the encoder output and the time. We condition on the time in each cross-attention and each feed hord-forward block. Please note that the size of the parts of the architecture does not correspond to the number of allocated parameters in this figure.
Figure 2:
3.4. Implementing Flow Matching
During the training phase, tuple (2(1), .²) is drawn from the distribution p=.æ. Additionally, a time step ( ~ (0, 1] and a sample =0)) is drawn from the base distribution Ps.
4. Experiments
which is a standard Gaussian for all our applications. Sub-
To show that the proposed methodology is not just ab- stract concept, we derive exemplary use cases that demon- strate how well ICL is able to keep up with MCMC and VI approaches in practice.
sequently, the ground-trut truth conditional flow e (x(())))) [z) = (0) + (1) is computed, pushing forward Ps into p=l up to time-point f. The transformer encoder processes and the decoder takes the representation of the encoder into account in order to output (1, (z))) This output should match the vector field that describes how the ground- truth flow (²)= |æ:) continues at time I. The discrepancy to the truuh vector field is measured with the MSE-loss in Equation (7).
For this, we will use two prominent statistical modeling classes, namely generalized línear models (GLMs) and la- tent factor models. For the latent factor models, we consider factor analysis (FA) and Gaussian mixture models (GMMs).
Modeling Scenarios. We use seven different scenarios for the GLMs, where we vary the prior distribution on the parameters, the conditional distribution of the response, and whether intercept is included. For FA, vary the form of the priors and dimensionalities of variables leading to four different scenarios. For the GMMs, we investigate different dimensionalities as well as prior configurations also in four different scenarios. We refer to Appendix A for details on the model structure and scenarios.
In the sampling phase, we are given æ and the goal is to sample from p=|= To do so, first vector (0) Ps is drawn. The data is passed through the encoder. The decoder defines a function that maps time-point and a vector v onto a vector field: (1, - (v) taking x into account. This function is given to an ODE-solver in order to forward-solve the corresponding ODE with boundary conditions < 5 1.
6

---

Can Transformers Learn Full Bayesian Inference In Context?
Table I: Summarized results for GLMs. Average perfor- mance of VI methods and our ICL approach on 50 synthetic and 17 real-world datasets across different GL.M scenarios. Comparison to the analytical solution when available and HMC otherwise. Lower is better for all metrics. The best average result is marked in bold.
Table 2: Results for GLMs. Real-world Evaluation on 17 datasets: Linear regression with a gamma prior on the coef- ficients 33 and an inverse gamma prior on the variance o² of the responses (scenario 5). Comparison to HMC samples. All results within two standard errors of the best average result are marked in bold.
Synthetic
Real-Worls
Model
C2ST
MMD
W/2
Model
C2ST
MMD
W2
C2ST
MMD W2
LA
1.000 (1: 0.000)
1.982 (# 0.126)
0.623 (# 0.084)
VI: Diagonal
0.810 0.036)
0.441 (+ 0.252)
0.384 0.089)
LA
1.000
2.770
2.049
1.000
2.091
0.849
VI: Full
0,711 (+: 0.038)
0.148 (+ 0.093)
0.279 (#: 0.056)
VI: Diagonal
0.869
1.586
1.742
0.819
0,583
0,529
VI: Structured
0.705 (+: 0.032)
0.140 (+ 0.081)
0.269 (+ 0.045)
VI: Full
0.714
1.016
1.601
0.668
0.116
0.374
VI: IAF
0.777 (2: 0.106)
0.684 (# 0.939)
0.625 (#: 0.525)
VI: Structured
0.711
0.929
1,580
0.664
0.109
0.370
ICL (ours)
0.610 (#: 0.045)
0.046 (# 0.020)
0.242 (#: 0.038)
VI: IAF
0.784
1.648
2.349
0.732
0.516
0.680
ICL (ours)
0.657
0.183
0.556
0.648
0.090
0.387
rior distribution. The first metric is a classifier 2-sample test (C2ST; Lueckmann et al., 2021; Lopez-Paz & Oquab, 2016). where the ROC-AUC score of a random forest classi- fier, trained to distinguish between samples from the gold standard and the method in question, is utilized. For random forest, we use default hyperparameters, as defined in Scikit- learn (Pedregosa al., 2011) and 10-fold cross- valsddaali, We use random forest with the given hyperparameters as a highly performative classifier in order to detect small devia- tions in distributions, even though this incurs the risk that the C2ST quickly saturates at a value of one, especially in nigh-dimensional cases. The second metric is the maximum mean discrepancy (MMD) between the two distributions (gold-standard and each tested method) with exponential kemel (Gretton et al., 2012). The third metric is the empir- ical Wasserstein-2 distance (W2; Givens & Shortt, 1984) of the two distributions, as implemented in the POT library (Flamary et al., 2021).
For our experiments, we train a separate model from scratch for each GLM, GMM, and FA scenario using synthetic samples from the joint distribution pæ.z; i.e. we train seven separate models to cover the GLM scenarios. six separate models for the FA scenarios and four separate models for the GMM scenarios. Please refer to Appendix B for more details.
Datasets. We evaluate the methods on 50 synthetic datasets and 17 real-world datasets from benchmark suite for tabular regression problems proposed by Grinsztajn et al. (2022). We refer to Appendix C for more details on the preprocessing of the datasets.
Methods. Apart from a comparison with a gold standard, we compare our ICL approach to a Laplace approximation (LA; Daxberger et al., 2021) and different established VI methods based on automatic differentiation VI (Kucukel- bir et al., 2017). For the variational distribution, we use normal distribution with 1) diagonal and 2) full covari- ance matrix, as well as 3) structured normal distribution with linear dependencies between the latent variables, and 4) an approach based on inverse autoregressive flows (IAF; Kingma et al., 2016). Appendix F includes a discussion regarding the hyperparameters of all considered methods.
4.1. Generalized Linear Models
Across seven different variants of GLMs, we find that ICL. yields samples that have overall the highest agreement with the gold standard (see Table 1). Specifically on the synthetic datasets, the C2ST, MMD and W2 metrics indicate that the posterior distribution can be approximated more accurately with ICL than via variational inference.
Evaluation Process. For every synthetic and real-world dataset, 1000 posterior samples from each method are com- pared against samples from the analytical solution, if avail- able, or from a Hamiltonian Monte Carlo (HMC) sampler with a NUTS kernel (Hoffman et al., 2014) as the gold standard. If posteriors are unimodal, we run a single chain. In the multimodal case, we use three times the number of modes as the number of Markov chains.
Particularly in cases where the posterior has shape deviat- ing from a normal distribution, ICL and HMC agree more closely than VI. For instance, in the case where a gamma prior, i.c. a skewed distribution, is used on the coefficients of a regression model, we find that ICL substantially outper- forms VI both on synthetic and real-world data (see Table 2). On the real-world data, ICL still matches the performance of VI methods and has the best (or not significantly worse than the best) performance in terms of C2ST in four out of seven cases (see Table 2). Please refer to Appendix for the detailed experimental results summarized in Table 1.
Evaluation Metrics. Three metrics are employed to com- pare samples from different approximations of the poste-
7

---

Can Transformers Learn Full Bayesian Inference In Context?
Table 3: Summarized results for FA. Average performance of VI methods and our ICL approach on 50 synthetic and 17 weal-world datasets across 6 different FA scenarios. Com- parison to HMC samples. Lower better for all metrics. The best average result is marked in bold.
Table 4: Summarized Results for GMMs. Average perfor- mance of VI methods and our ICL approach on 50 synthetic and 17 real-world datasets across 4 different GMM scenar- ios. Comparison to HMC samples in all cases. The best average result is marked in bold.
Synthetic
Real-World
Synthetic
Real-World
Model
Model
C2ST
MMD
W2
C2ST
MMD
W2
C2ST
MMD
W2
C2ST
MMD
W2
LA
1.000
4.115
2.543
1.000
4.127
0.597
LA
1,000
3,916
8.324
1,000
3.385
12.740
VI: Diagonal
0.999
3,321
1.998
0.960
1.220)
0.288
VI: Diagonal
0.994
2.676
7.938
0.992
2.182
11.633
VI: Full
0.993
3.222
1.955
0,950
1.173
0.281
VI: Full
0.995 2.556
7.947
0.987
2.143
11.696
VI: Structured
0.995
3.404
2.079
0.955
1.189 0.283
VI: Structured
0.994
2.595
7.929
0.988
2.129
11.521
VI: IAF
0.987
3.226
1.973
0.902
0.969
0.251
VI: IAF
0.985
2.308
7.489
0.957
1.845
11.541
ICL (ours)
0.568
0.057
0.409
0.751
0.673
0.583
ICL (ours)
0.825
0.706
4.348
0.881
1.051
10.691
4.2. Factor analysis
shows high agreement between the posterior distributions of both HMC and ICL while VI is incapable of perfectly approximating a bimodal distribution and exhibits typical mode-seeking behavior (Figure 3). Note that also the VI approach based on inverse autoregressive flows, which in theory allows flexible modeling of wide range of posterior shapes, fails to learn the bi-modality accurately from the limited number of 50 data points in this GMM scenario. This demonstrates the strength of our ICL approach in ficxibly learning distributions agnostic of the provided sample size. Please refer to Appendix I for the detailed experimental results summarized in Table 4.
On the factor analysis tasks, ICL has notably lower dissim- ilarity scores compared to the gold standard than all other considered methods in the synthetic evaluation (Table 3). Notably, an average C2ST score of 0.568 is remarkably close to the theoretical lower bound of 0.5. Regarding the real world datasets, C2ST and MMD indicate that our ICL approach yields samples most similar to the reference, while the average W2 score is substantially higher. We hypoth- esize that this discrepancy in the metrics might be caused by numerical issues when computing the empirical W2 dis- tance. Furthermore, the relatively high number of latent variables in comparison to the limited number of data-points can yield overly flexible assumptions on the variational pos- terior causing the VI methods to overfit. See Appendix I for the detailed experimental results summarized in Table 3.
4.4. A Ablations and Further Experimental Results
In this subsection, we present various ablations concerning our ICL approach to full Bayesian inference. Due limited space, most of the results are deferred to the appendix.
4.3. Gaussian Mixture Models
Full Bayesian inference for GMMs is more challenging than for GLMs or FA. First, the generative process of GMMs involves discrete assignments to clusters, which poses a challenge not only for NUTS, but especially for VI methods. Second, the dimensionality of the posterior samples can be relatively large since for diagonal normal distributions, each component of the mixture has mean and a variance param- eter per dimension. Finally, the considered GMMs are not identifiable leading to multi-moda posterior distributions, which are impossible to perfectly approximate with com- monly used VI methods based on Gaussian approximations.
Alternatives to Flow Matching. Appendix L contains results from an ablation study using diffusion objectives instead of flow matching, while Appendix K investigates the use of multivariate Gaussian parametrize 1% The empirical results from these ablations strongly indicate that flow matching is essential for achieving a close approxi- mation of the gold-standard posterior in the scenarios we consider.
Dimensionality. In addition, we investigate the effect of the dimensionality K of the latent variable z E RR for all seven different GLM scenarios. The key takeaway from our results is that for K 20 and K 50, the ICL ap- proach performs comparably to the other methods in terms of sample similarity to HMC, but does not outperform them. Please refer to Appendix O for more details. We hypothe- size that a key reason for the failure to detect meaningful differences between the methods in high dimensions is due to the curse of dimensionality affecting our metrics.
Due to this inherent difficulty of the GMM scenarios, we find the overall performances of all models be worse than in the GLM and FA cases. In particular, the C2ST metric is almost saturated for the VI approaches and has value of around 83 percent for ICL (Table 4). The MMD and W/2 metrics also indicate that ICL yields samples with higher agreement with the reference than the other approaches on synthetic data. A plot of the marginals of the posterior
8

---

Can Transformers Learn Full Bayesian Inference In Context?
5. Discussion
Dimension 1
Dimension 2
Dimersion 3
Dimension 6
0.4
This paper explores in-context learning for the purpose of full Bayesian inference in latent variable models. We pro- pose use conditional flow matching as a generic and flexible framework to approximate posterior distributions and an architecture that utilizes a transformer encoder for po- tentially complex conditioning on the data. We find that our ICL approach yields a closer approximation of the posterior than several state-of-the-arr variational inference methods across different datasets and model setups. This does not only hold for synthetic, but also real-world tabular datasets.
6.4
0.2
0.2
Vt: Diegonal
0.4
0.2
0.75
Mt: Full
0.25
Structured 0.5
0.2
Limitations. While our experiments indicate the effective- ness of ICL as a Bayesian inference method, it requires an extensive up-front training routine on modern GPU hard- ware. Despite ICL being consistently faster at inference time than the considered HMC methods, the overall compu- tational burden to train our approach is much higher.
Vr IAF 1.0
0.5
Figure 3: Density plots for the marginals of the posterior for GMM scenario 1. Comparison to HMC samples on a synthetic dataset whose density is depicted as a dotted line. Only the marginals of the first two components of the mean and the variance are shown. The density of the posterior obtained via HMC is depicted ICL method aligns with the gold-standard HMC, the VI methods have lower level of agreement and exhibit mode- seeking behaviour.
Furthermore, the goal of this work to show that ICL can effectively learn full Bayesian inference. Our experiments therefore focus on relatively simple posterior distributions where we can compare against established methods, such as HMC. Additionally, increased dimensionality of the prob- lems considered poses a challenge to both the ICL method and the metrics we employ. Further, as with many other ICL approaches, large datasets as a context can become computationally very expensive.
dotted line. While the
Out-off distibiitinttonn Performance. We further investigate the robustness of our method under mild distribution shifts in Appendix N. Our results indicate that the performance of our ICI. method remains relatively stable for small dis- tribution shifts, but increasingly degrades for a larger gap between the training and testing distribution.
Outlook and Future Work. Despite its vast up-front com- putational cost, ICL has not only proven fundamentally transformative in the field of NLP (Brown et al., 2020; Tou- vron et al., 2023), but has recently started to transform the field of tabular machine learning (Hollmann et al., 2022). Exploring the frontiers of ICL in terms of full Bayesian inference, starting from the feasibility results of this work, might therefore lead to similarly fertile territories.
Predictive Performance. Additionally, we evaluate our ICL method and all variational approaches with respect to the predictive performance of the considered GLM setups in Appendix J. Results confirm the strong performance of variational inference methods in terms of point prediction, especially for high dimensionalities, while the ICL method is generally competitive.
Although ICL performs well even when trained on data that may differ from real-world distributions, its flexibility is limited by the structure of the training data. If the synthetic data is highly unrealistic, ICL may fail- much like any model with a misspecified hypothesis space that imposes an unsuitable inductive bias.
Architecture. Appendix M discusses results regarding the effect of using an Ml.e-based architecture. Our experi- mental findings confirm that the transformer- based encoder performs significantly better than an equally sized MLP encoder.
While flexible state-of-the-art sampling-hasee methods, such HMC, serve as an efficient and highly effective reference in terms of inference for standard and statistical methods discussed in this paper, the proposed ICI. approach is fundamentally more general in nature. In particular, any probabilistic model for which generative process is con- ceivable can be fitted using our ICL approach- the potential for fitting models beyond the horizon of standard Bayesian methods is therefore manifold.
The Classifier in the C2ST Metric. Finally, we validate the choice of random forest classifier for the C2ST metric (Appendix Q). We find that employing a nonlinear neural network and utilizing a random forest yields an overall anal- ogous picture in terms of the performance of all methods.
9

---

Can Transformers Learn Full Bayesian Inference In Context?
Impact Statement
Brosse, N., Durmus, A., and Moulines, E. The promises and pitfalls of stochastic gradient langevin dynamics. Ad- vances in Neural Information Processing Systems, 31, 2018.
This paper presents work whose goal to advance the field of machine learning. There are many potential societal consequences of our work, none of which feel must be specifically highlighted here.
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in newral information processing systems, 33: 1877-1901, 2020.
Acknowledgements
We thank Beste Aydemir and Svea Reuter for their valuable support and insightful comments. VF was supported by the Branco Weiss Fellowship. DR's research is funded by the Deutsche Fonschungsermeinsccatt (DFG, German Research Foundation)- 548823575. We also gratefully acknowledge funding provided to AR by the Munich Center for Machine Learning (MCML).
Chan, S. C., Dasgupta, I., Kim, J., Kumaran, D., Lampinen, A. K., and Hill, F. Transformers generalize differently from information stored context vs in weights. arXiv preprint arXiv:2210.05675. 2022.
Chen, R. T. Q. torchdiffeq, 2018. URL https://
github wtt.ccoo/iiieeeee wwe.giihhnn/orrcc.fffeee
References
Chen, T., Fox, E., and Guestrin, C. Stochastic gradient hamiltonian monte carlo. In International conference on machine learning, pp. 1683-1691. PMLR, 2014.
Abdullah, A. A., Hassan, M. M., and Mustafa, Y. T. A review on bayesian deep learning in healthcare: Appli- cations and challenges. IEEE Access, 10:36538-36562, 2022.
Cranmer, K., Brehmer, J., and Louppe, G. The frontier of simulation-baset inference. Proceedings of the National Academy of Sciences, 117(48)330055-M0
Ahuja, K., Panwar, M., and Goyal, N. In-context learning through the bayesian prism. arXiv preprint arXiv:2306. 04891, 2023.
2020.
Cremer, C., Li, X., and Duvenaud, D. Inference subop- timality in variational autoencoders. International conference on machine learning, pp. 1078-1086 PMLR, 2018.
Anil, R., Borgeaud, S., Alayrac, J.-B., Yu, J., Soricut, R., Schalkwyk, J., Dai, A. M., Hauth, A., Millican, K., al. Gemini: a family of highly capable multimodal models. arXiv preprint 2rXiv:2312.//880 2023.
Dao, Q., Phung, H., Nguyen, B., and Tran, A. Flow match- ing latent space. arXiv preprint arXiv:2307.08693 2023.
Betancourt, M. A conceptual introduction hamiltonian monte carlo. arXiv preprint 1 2017.
Dax, M., Green, R., Gair, J., Macke, J. H., Buonanno, A., and Schölkopf, B. Real-time gravitational wave science with neural posterior estimation. Physical review letters, 127(24):24110) 1003, 2021.
Bingham, E., Chen, J. P., Jankowiak, M., Obermeyer, F., Pradhan, N., Karaletsos, T., Singh, R., Szerlip, P., Hors- fall, P., and Goodman, N. D. Pyro: Deep universal prob- abilistic programming. Journal of machine learning re- search, 20(28):1-6, 2019.
Dax, M., Green, S. R., Gair, J., Gupte, N., Pürrer, M., Ray- mond, V., Wildberger, J.. Macke, J. H., Buonanno, A., and Schölkopf, B. Real-time gravitational-wan inference for binary neutron stars using machine learning. arXiv preprint arXiv: 2407:0:900. 2024.
Bishop, C., Spiegelhalter, D., and Winn, J. Vibes: A varia- tional inference engine for bayesian networks. Advances in neural information processing systems, 15, 2002.
Blei, D. M. Probabilistic topic models. Communications of the ACM, 55(4):77-84,200
Daxberger, E., Kristiadi, A., Immer, A., Eschenhagen, R., Bauer, M., and Hennig, P. Laplace redux-effortless bayesian deep learning. Advances in Neural Informa- tion Processing Systems, 34:20089-20103, 2021.
Blei, D. M., Kucukelbir, A., and McAuliffe, J. D. Varia- tional inference: A review for statisticians. Journal of the American statistical Association, 112(518):859-877, 2017.
Dong, Q., Li, L.., Dai, D., Zheng, C., Wu, 7., Chang, B., Sun, X., Xu, J., and Sui, Z. survey on in-context learning. arXiv preprint arXiv:2301.003 100034, 2022.
Brehmer, J. and Cranmer, K. Simulation baased inference methods for particle physics. In Artificial Intelligence for High Energy Physics, pp. 579-611. World Scientific, 2022.
Dormand, J. R. and Prince, P. J. A family of embedded runge-kutta formulae. Journal of computational and ap- plied mathematics, 6(1):19-22 1980.
10

---

Can Transformers Learn Full Bayesian Inference In Context?
Eloundou, T., Manning, S., Mishkin, P., and Rock, D. Gpts are gpts: An early look at the labor market im- pact potential of large language models. arXiv preprint 1rXiv:2303.10130 2023.
Gretton, A., Borgwardt, K. M., Rasch, M. J., Schölkopf, B., and Smola, A. A kernel two-sample test. The Journal of Machine Learning Research, 13(1):723-773, 2012.
Grinsztajn, I.., Oyallon, E.,, and Varoquaux, G. Why do tree- based models still outperform deep learning on typical tabular data? Advances in neural information processing systems, 35:507-520, 2022.
Esser, P., Kulal, S., Blattmann, A., Entezari, R., Müller, J., Saini, H., Levi, Y., Lorenz, D., Sauer, A., Boesel, F., et al. Scaling rectified flow transformers for high-resolution image synthesis. Forty-first International Conference on Machine Learning, 2024.
Grünwald. P. and Ommen, T. Inconsistency of bayesian inference for misspecified linear models, and proposal for repairing it. Bayesian Analysis, 12(4):1099-1103, 2017.
Etzioni, R. D. and Kadane, J. B. Bayesian statistical meth- ods in public health and medicine. Annual review of public health, 16(1):23-41, 1995.
Hastings, W. Monte carlo sampling methods using markov chains and their applications. Biometrika, 57(1):97-109 1970.
Fahrmeir, L., Kneib, T., Lang, S., Marx, B., Fahrmeir, L., Kneib, T., Lang, S., and Marx, B. Regression models. Springer, 2013.
Hoffman, M. D., Gelman, A., et al. The no-u-turn sampler: adaptively setting path lengths in hamiltonian monte carlo. J. Mach. Learn. Res., 15(1):1593-166 2014.
Fan, X. and Markram, H. brief history of simulation neuroscience. Frontiers in neuroinformatics, 13:32, 2019.
Hollmann, N., Müller, S., Eggensperger, K., and Hut- ter, F. Tabpfn: transformer that solves small tabu- lar classification problems in a second. urXiv preprint arXiv:2207.01 848, 2022.
Flamary, R., Courty, N., Gramfort, A., Alaya, M. Z., Bois- bunon, A., Chambon, S., Chapel, L.,, Corenflos, A., Fa- tras, K., Fournier, N., et al. Pot: Python optimal trans- port. Journal of Machine Learning Research, 22(78):1-8, 2021.
Hollmann, N., Müller, S., Purucker, L., Krishnakumar, A., Körfer, M., Hoo, S. B., Schirrmeister, R. T., and Hutter, F. Accurate predictions on small data with tabular foundation model. Nature, 637(8045):319
Garg, S., Tsipras, D., Liang, P. S., and Valiant, G. What can transformers learn in-context? a case study of sim- ple function classes. Advances in Neural Information Processing Systems, 35:30583-30598, 2022.
2025.
Hoo, S. B., Müller, S., Salinas, D., and Hutter, F. The tabular foundation model tabpfn outperforms specialized time series forecasting models based simple features. In NeurIPS Workshop on Time Series in the Age of Large Models, 2024.
Garnelo, M., Rosenbaum, D., Maddison, C., Ramalho, T., Saxton, D., Shanahan, M., Teh, Y. W., Rezende, D., and Eslami, S. Conditional neural processes. Interna- tional conference on machine learning, pp. 1704-1713. PMLR, 2018a
Hospedales, T., Antoniou, A., Micaelli, P., and Storkey, A. Meta-learning in neural networks: A survey. IEEE transactions on pattern analysis and machine intelligence, 44(9):5149-5166 2021.
Garnelo, M., Schwarz, J., Rosenbaum, D., Viola, F., Rezende, D. J., Eslami, S., and Teh, Y. W. Neural pro- cesses. arXiv preprint arXiv: 1807.0180000112.. 2018b.
loffe, S. Batch normalization: Accelerating deep net- work training by reducing internal covariate shift. arXiv preprint arXiv: 0 50..000....6,. 2015.
Gebhard, T. D., Wildberger, J., Dax, M., Kofler, A., Anger- hausen, D., Quanz, S. P., and Schölkopf, B. Flow match- ing for atmospheric retrieval of exoplanets: Where reli- ability meets adaptive noise levels. Astronomy & Astro- physics, 693:A42, 2025.
Izmailoy, P., Vikram, S., Hoffman, M. D., and Wilson, A. G. G. What are bayesian neural network posteriors really like? In International conference on machine learning, pp. 4629-4640. PMLR, 2021.
Givens, C. R. and Shortt, R. M. A class of wasserstein met- rics for probability distributions. Michigan Mathematical Journal, 31(2):231-240 1984.
Kim, Y., Wiseman, S., Miller, A., Sontag, D., and Rush, A. Semi- amortizer variational autoencoders. In Interna- tional Conference on Machine Learning, pp. 2678-2687. PMLR, 2018.
Gloeckler, M., Deistler, M., Weilbach, C., Wood, F., and Macke, J. H. All-in-one simulation- based inference. arXiv preprint arXiv:2404. 09636, 2024.
Kingma, D. P. Auto-encoding variational bayes. arXiv preprint orXiv: 2013.

---

Can Transformers Learn Full Bayesian Inference In Context?
Kingma, D. P. Adam: A method for stochastic optimization. arXiv preprint arXiv: 1412.6980, 2014.
Lueckmann, J.-M., Goncalves, Bassetto, G., Ócal, K., Nonnenmacher, M., and Macke, J. H. Flexible statistical inference for mechanistic models of neural dynamics. Advances in neural information processing systems, 30, 2017.
Kingma, D. P., Salimans, T., Jozefowicz, R., Chen, X., Sutskever, I., and Welling, M. Improved variational infer- ence with inverse autoregressive flow. Advances in neural information processing systems, 29, 2016.
Lueckmann, J.-M., Boelts, J., Greenberg, D., Goncalves, P., and Macke, Benchmarking simulation-basee based inference. In International conference artificial intelligence and statistics, pp. 343-351. PMLR. 2021.
Kirsch, L., Harrison, J., Sohl-Dickstein, J., and Metz, L. General-purpose in-context learning by meta-learning transformers. arXiv preprint arXiv:2212.04458. 2022.
Mangoubi, O. and Vishmoi, N. K. Nonconvex sampling with the metropolis-adiusted langevin algorithm. Confer- ence learning theory, pp- 2259-2299 PMLR, 2019.
Kucukelbir, A., Tran, D., Ranganath, R., Gelman, A., and Blei, D. M. Automatic differentiation variational infer- ence. Journal of machine learning research, 18(14):1-45, 2017.
Margossian, C. C. and Blei, D. M. Amortized varia- tional inference: When and why? arXiv preprint arXiv:2307. 1018, 2023.
Kyrimi, E., McLachlan, S., Dube, K., Neves, M. R., Fahmi, A., and Fenton, N. A comprehensive scoping review of bayesian networks in healthcare: Past, present and future. Artificial Intelligence in Medicine, 117:102108, 2021.
Mittal, S., Bengio, Y., Malkin, N., and Lajoie, G. In-context parametric inference: Point or distribution estimators?
arXiv preprint arXiv:2502. 1617,, 2025a.
Lawley, D. N. and Maxwell, A. E. Factor analysis as statistical method. Journal of the Royal Statistical Society, Series D (The Statistician), 12(3):209-223 1962.
Mittal, S., Bracher, N. L., Lajoie, G., Jaini, P., and Brubaker, M. Amortized in-context bayesian posterior estimation. arXiv preprint 2.0.2x::2002006000 2025b.
Le, T. A., Baydin, A. G., and Wood, F. Inference compila- tion and universal probabilistic programming. In Artificial Intelligence and Statistics, pp. 1338-1348. PMLR, 2017.
Müller, S., Hollmann, N., Arango, P., Grabocka, J., and Hutter, Transformers can do bayesian-inferente by meta-learning on prior-data. In Fifth Workshop on Meta- Learning at the Conference Neural Information Pro- cessing Systems, 2021.
Li, C., Chen, C., Carlson, D., and Carin, L. Preconditioned stochastic gradient langevin dynamics for deep neural net- works. Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, pp. 1788-1794, 2016.
Murphy, K. P. Probabilistic machine learning: Advanced topics. MIT press, 2023.
I Lienen, M., Kollovieh, M., and Günnemann, S. Generative modeling with bayesian sample inference. arXiv preprint arXiv:2502.07550. 2025.
Nelder, J. A. and Wedderburn, R. W. Generalized linear models. Journal of the Royal Statistical Society Series A: Statistics in Society, 135(3):370-384, 1972.
Lin, X., Wu, J., Zhou, C., Pan, S., Cao, Y., and Wang, B. aask-adaptive neural process for user cold-start recom- mendation. Proceedings the pp. 1306-1316, 2021.
OpenAL. Gpt-4 technical report. arXiv:2303.08774, 2023.
arXiv preprint
the Web Conference 2021,
Papamakarios, G., Nalisnick, E.,, Rezende, D. J., Mohamed, S., and Lakshminarayanan, B. Normalizing flows for probabilistic modeling and inference, Journal of Machine Learning Research. 22(57): 2021a.
Lipman, Y., Chen, R. T., cen-Hamu, H., Nickel, M., and Le, M. Flow matching for generative modeling. arXiv preprint wwXiv:2210.0274: 2022.
Papamakarios, G., Nalisnick, E.. Rezende, D. J., Mohamed, S., and Lakshminarayanan, B. Normalizing flows for probabilistic modeling and inference. Journal of Machine Learning Research, 12(5)):1-64 2021b.
Lopes, H. F. and West, M. Bayesian model assessment in factor analysis. Statistica Sinica, pp. 41-67, 2004.
Lopez-Paz, D. and Oquab, M. Revisiting classifier two- 2016.
sample tests. arXiv preprint
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., et al. Scikit-learn: Machine learning in python. the Journal of machine Learning research, 12:2825-28.3 2011.
Loshchilov, and Hutter, F. Sgdr: Stochastic gra- dient descent with warm restarts. arXiv: 1608. 03983, 2016.
arXiv preprint
12

---

Can Transformers Learn Full Bayesian Inference In Context?
Peebles, W. and Xie, S. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF Interna- tional Conference Computer Vision, pp. 4195-4205, 2023.
Song, Y. and Ermon, S. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.
Song, Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Er- mon, S., and Poole, B. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.134555 2020.
Phan, D., Pradhan, N., and Jankowiak, M. Composable effects for flexible and accelerated probabilistic program- ming in numpyro. arXiv preprint arXiv:1912.1155 2019.
Srivastava, A. and Sutton, C. Autoencoding variational infer- ence for topic models, arXiv preprint arXiv: 1703.0000,, 2017.
Rezende, D. J., Mohamed, S., and Wierstra, D. Stochastic backpropagation and approximate inference in deep gen- erative models. In International conference on machine learning, pp. 1286. PMLR, 2014.
Touvron, H., Martin, L.,, Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama Open foundation and fine- tuned chat models. arXiv preprint arXiv:2307.09288, 2023.
Robertson, J., Hollmann, N., Awad, N., and Hutter, Fairpfn: Transformers can do counterfactual fairness. In ICML 2024 Next Generation of Al Safety Workshop, 2024.
Rudner, T. G., Fortuin, V., Teh, Y. W., and Gal, Y. On the connection between neural processes and gaussian processes with deep kernels. In Workshop on Bayesian Deep Learning, NeurIPS, pp. 14, 2018.
Vetter, J., Gloeckler, M., Gedon, D., and Macke, J. H. Effort- less, simulation-efficciet bayesian inference using tabular foundation models. arXiv preprint arXiv:2504. 17660, 2025.
Rummel, R. J. Applied analysis. Northwestern Uni- versity Press, 1988.
Walker, G. Bayesian inference with misspecified models. Journal of statistical planning and inference, 143(10): 1621-1633, 2013.
Sahoo, S., Gokaslan, A., De Sa, C. M., and Kuleshov, V. Diffusion models with learned adaptive noise. Advances in Neural Information Processing Systems, 37:105730- 105779, 2024.
Wang. X., Zhu, W., Saxon, M., Steyvers, M., and Wang, W. Y. Large language models are latent variable mod- els: Explaining and finding good demonstrations for in- context learning. Advances Neural Information Pro- cessing Systems, 36, 2024.
Salazar, S. Vart: variational regression trees. Advances in Neural Information Processing Systems, 36:45681- 45693, 2023.
Wang. Y. and Blei, D. Variational bayes under model mis- specification. Advances in Neural Information Processing Systems, 32, 2019.
Schmit, C. J. and Pritchard, J. R. Emulation of reioniza- tion simulations for bayesian inference of ustrophysics parameters using neural networks. Monthly Notices of the Royal Astronomical Society, 475(1):1213-1223, 2018.
Welling, M. and Teh, Y. W. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th international conference on machine learning (ICML-11), pp. 681-688. Citeseer, 2011.
Sohn, H. and Narain, D. Neural implementations of bayesian inference. Current Opinion in Neurobiology, 70: 121-129, 2021.
Wildberger, J., Dax, M., Buchholz, S., Green, S., Macke, J. H., and Schölkopf, B. Flow matching for scalable simulation-base inference. Advances in Neural Informa- tion Processing Systems, 36, 2024.
Sommer, E., Wimmer, L., Papamarkou, T., Bothmann, L., Bischl, B., and Rügamer, D. Connecting the dots: Is moode-connectedness the key to feasible sample-based inference in bayesian neural networks? In Proceedings of the 41st International Conference on Machine Learning. PMLR. 2024.
Yeo, I.-K. and Johnson, R. A. A new family of power transformations to improve normality or symmetry. Biometrika, 87(4):954 959, 2000.
Sommer, E.,, Robnik, J., Nozadze, G., Seljak, U., and Rügamer, D. Microcanonical Langevin Ensembles: Ad- vancing the Sampling of Bayesian Neural Networks. In The Thirteenth International Conference on Learning Representations, 2025.
Yim, J., Campbell, A., Foong, A. Y., Gastegger, M., Jiménez-Luna, J., Lewis, S., Satorras, V. G., Veeling, B. S., Barzilay, R., Jaakkola, T., et al. Fast protein back- bone generation with se (3) flow matching. arXiv preprint arXiv:2310.05297, 2023.
13

---

Can Transformers Learn Full Bayesian Inference In Context?
Yim, J., Campbell, A., Mathieu, E., Foong, A. Y., Gastegger, M., JiménnzzLLuna, J., Lewis, S., Satorras, V. G., Veeling,
B. S., Noé, F,, et al. Improved motif-scaffolding with se (3) flow matching. arXiv preprint arXiv:2401.04082, 2024.
Zhai, J., Zhang. S., Chen, J., and He, Q. Autoencoder and its various variants. In 2018 IEEE international conference on systems, man, and cybernetics (SMC), pp. IEEE, 2018.
Zhao, W., Shi, M., Yu, X., Zhou, and Lu, J. Flowturbo: Towards real-time flow-based image generation with locity refiner. arXiv preprint arXiv:2409,/8//
2024.
Zheng. K., Lu, C., Chen, J., and Zhu, J. Improved techniques for maximum likelihood estimation for diffusion odes. In International Conference Machine Learning, pp. 42363-42389, PMLR, 2023.
14

---

Can Transformers Learn Full Bayesian Inference In Context?
Appendix
A. Data-generating Processses
This section contains more details on the data generating processes of the latent variable models we fit via ICL.
A.1. Generalized Linear Models
In this section we expand the description and explanation regarding GLMs from Section 3.2. GLMs are among the most commonly used statistical models with myriads of applications (Nelder Wedderburn, 1972; Fahrmeir et al., 2013). the context of GLMs, we assume that the response y follows distribution pal" depending on the linear predictor v) = u and additional parameter o². We denote the covariates u, the regression coefficients as f, and use x² for the variance of the response. The mean of py1a depends on the linear predictor via link function g. such that g (IE[)/ uu]) u G. Ultimately, the density of distribution of the response depending on the linear predictor and the additional parameter is denoted by p(ylg (u ( To showcase the flexibility of our framework, we experiment with different priors p3 on the regression coefficients, pa² on the parameter o², and also different parametric distributions of the response. Additionally, to include covariates that resemble practically relevant tabular data in the generative process, allowing for meaningful inference real-world datasets, we utilize samples from the Tab-PFN "prior" for p".
GLMs belong to the framework of latent variable models defined by data and (latent) variables z. where the data comprises covariates and response a := (u, y). The variables of interest are the coefficients 2 == B. This yields the following generative process for set of synthetic samples D { 2, }== from pæ.#
We consider seven different GLM scenarios by varying the structure of the prior distributions and the conditional distribution of the response (Table 5). In particular, we consider a normal (0, 1) prior, a Laplace 1) and a gamma Ga(1, 1) prior that factorizes over the coefficients 3, contained in B = (B1...., 3p).. In two cases we include an intercept in the model using a normal prior N (0, 9) with a relatively large variance. We consider regression cases with a normally distributed response Nuu [3 o²), a Bernoulli distributed response Bin (1, sigmoid ()), i.e. logistic regression, and response following gamma distribution Ga(o-2 exp (u ( exp(2u ()). In the last case, set exp(u (G) be the mean and o² to be the conditional variance of the response. An inverse gamma prior IG(5, 2) is used on the variance o² for each scenario except the logistic regression. We fix the number of covariates and thus also the dimensionality of /3 at p 5 and set the number of data points per dataset to K 50.
Algorithm Generation of synthetic data for GLMs
Require: Number of datasets N, number of samples per dataset K, distributions På po² pu Ensure: A dataset ID of input-output pairs (æ;; 2,) for i 1 ,,..,,.. N.
1: Initialize D +- Ø
2: for = 1 + N do
3: Draw 3, ~ pa
4: Draw of ~ po²
5:
f =1 + K do
6:
Draw ui pd
Draw Viy ~ p/ g g-'((T,, of)
7:
8:
end for
K Set = ((uij 3(.5)) 1=1
9:
10: Set 3r
11:
Update D - DU (1:1,
12:
end for
15

---

Can Transformers Learn Full Bayesian Inference In Context?
Table 5: Distribution of variables for the considered GLM scenarios.
Scenario
Big
Bio
of
yt.j (ui.j. Bi, 14.80..07
Scenario 1
AO 1)
IG 2)
N(u, B.,0
Scenario 2
NO 1)
N/0,9) 9)
2)
N(u B
Scenario 3
Laplace(O, 1)
()
Nuu, B,,0
Scenario 4
Laplace(O,
(0,
2))
N(u 8,,0
Scenario 5
1)
2G)
N , B,,0
Scenario 6
1)
Bin(1, signoi,
Scenario 7
10,1
IG(
Ga(ø, exp (,),,B)). exp(2uu,,
A.2. Factor Analysis
The goal of factor analysis is to explain data a in terms of latent, typically lower-dimensional, factors z (Lawley & Maxwell, 1962; Rummel, 1988). In the Bayesian setting, one assumes a prior ps on the latent variable z, a prior pW on the factor loading matrix W and additional priors pV and ps on the covariance matrix and the mean vector. The conditional distribution p=(= of the data given has mean E [x] W + μ and covariance matrix Cov[z|æ: where pa and pa(æ are Gaussian, one can pa NO, I) and assume diagonal covariance matrix T without loosing expressiveness of the model (Murphy, 2023). We make the assumption that W is lower triangular with positive entries on the diagonal in order to ensure identifiability of the model (Lopes & West, 2004), Additionally, we assume that the distributions V and pw fully factorize. In order to ensure that the diagonal of W is positive, we consider absolute values the generative process. Algorithm 3 details the data generating process.
v. the case
Table summarizes the different configurations for FA. We assume a Gaussian prior on the mean components, and an inverse gamma prior on the elements of the diagonal covariance matrix V. For the factor loading matrix W, independent normal and Laplace priors are investigated. Furthermore, we use normal prior on the latent factors z; in five cases and Laplace prior in case. We vary the number of samples K per dataset æ, the dimensionality P of each data point, well as the dimensionality Zdim.
Table 6: Distribution and dimensionalitites of variables for the considered FA scenarios.
Scenario
K
P
Pijj
Viji
Wiiik
Zi.j
Zdim
Scenario
50
3
(0
IG(5, 1)
(0, 1)
N(0,
Scenario 2
50
3
0.1,0.1)
IG(5,1) 1)
Laplace(O, 10)
N(0,
3
Scenario 3
25
5
(.1,0.1)
IG(5,2)
(0(0,3)
N(
3
Scenario 4
25
15
(.0000.1
IG(5,2)
N(O,1)
5
Scenario 5
25
5
0.10
IG(5,2)
Laplace(0,
10,
3
Scenario 25
5
0.10
IG(5,2)
(0003)
Laplace(0,)
Algorithm Generation of synthetic data for FA
Require: Number of datasets N, number of samples K, and distributions Ph, p p
Ensure: A dataset D containing (æi, 2;) for = =1,..... N....,N. N.
1: Initialize < Ø
2: for í = 1 - N do
3: Draw μ1; Pl
4: Draw V Pv
5:
Draw W,- PW
6:
Draw ~ P2
7:
for j = + K do
8: Draw T5 ~ N/ W pi, V.)
9:
end for
Update DU (
11:
end for
16

---

Can Transformers Learn Full Bayesian Inference In Context?
A.3. Gaussian Mixture Models
Table 7: Distribution and dimensionalitites of variables for the considered GMM scenarios.
Scenario
K
M
Φ;
T.m.
Pi.m.l a i.m.l
Scenario
50
Dir(1)
IG 2)
[0,, 30 i.m.)
Scenario 2
25
Dir(1)
(G
NO 30 m.m.))
Scenario 3
50
Dir(0.5)
IG(5.2)
[00, 50 s.m.))
Scenario 4
50
3
Dir(1)
IG(5,2) 2)
[00,, 30 1.m.t)
In GMMs one assumes that the data of interest is generated by convex combination of M (multivariate)) normal distributions, such that p(æ]==) = Σm=1 @mPm (x=), where the probability vector Φ = (éi..... &x) comprises the mixture weights and Pm denotes the mn-h mixture component. We consider pm to take the form of a diagonal Gaussian with mean vector pmm. and covariance matrix with diagonal elements o We assume a prior ph on Φ, prior po on the variances of each component and a prior pulo" for the means that depends on the variance of the respective component. More specifically, we assume a symmetric Dirichlet prior on & such that ph = Dir(a and an independent inverse gamma distribution as prior on each component o of o² The prior on each component of js.m E RR is then given by an independent normal distribution pulei.. (O, Am. We use Wa, to denote the assignment of datapoint a component. Algorithm details the data generating process and Table summarizes the different setups regarding the prior distributions.
Algorithm 4 Generation of synthetic data for GMM.
Require: Number of datasets N, mixture dimension parameters M, L, number of samples K, and distributions pd po' pula".
Ensure: A dataset 'D containing (æ,, zi) for i = =1.i= I.....N.
1: Initialize D + Ø
2: for = 1 + N do
3: Draw Φ; pd
4: for m
M do 5: for 1 = 1 do
6:
Draw o.mmat po?
Draw Ham, pulll... i.m.
7:
8:
end for
9:
end for
10:
for j= K do
11:
Draw Wis ~ Cat (c:)
12:
Draw Iijj ~ N/ (i.we.n) (umss)
end for
14:
Set (( Mirm.l) m=l...,.M I=l....h
Update DU (I 2
end for
17

---

Can Transformers Learn Full Bayesian Inference In Context?
B. Generating Realistic Data
While we assume a data-generating process such as the one in Algorithm 2, this is not necessarily the data-generating process that produces the data in the model's application as an in-context learner. Even when the generative process p underlying a statistical model sophisticated and complex in nature, model misspecification is inevitable in almost every practical application. While mismatches between the real fata-generating processes and model assumptions can lead to various problems in traditional Bayesian modeling (Grünwald & van Ommen, 2017), the question of model misspecification plays a somewhat different and yet an especially central role for our ICL approach.
More specifically, the ICL model learns the relationship between p*|* and a datapoint a exclusively based on synthetic samples from the marginal pe implied by the statistical model with generative process pæ.*, Given real-world dataset x= pø" model misspecification in terms of pa implies that the in-context learner needs infer the posterior based on out-of-distribution data, where the problem is aggravated the more unrealistic pa is.
To be able to access a reference ground truth distribution, the data generating processes in our experiments need to match the structure of the GLM, FA and GMM approaches. While the generative processes of FA and GMMs directly prescribe how all parts of the data are generated, this can potentially cause a discrepancy between synthetically generated and rorldworld datasets. However, our empirical results (Section 4.1) demonstrate that the in-context learner can generalize to real-world data despite the discrepancy to the simulated datasets.
In the aforementioned GLM case, the distribution of the covariates pa does not affect the structure of palæ in the data generating process (cf. Algorithm 2). We can therefore use flexible prior p" such the TabPFN- "abior-"prior" (Hollmann et al., 2022) to generate covariates and thereby effectively tackle the issue of model specification.
C. Preprocessing of the Real-world Datasets
The real-world datasets considered for the evaluation of all methods are proposed in a benchmark study by Grinsztajn al. (2022). We standardize all features, scale and shift the target such that has the mean and variance implied by the prior structure of the respective generative model. Furthermore, for the GLM scenarios, we apply Yeo-Johnson transform on the target variable (Yeo & Johnson, 2000) before applying the scaling. In cases where the number of features in the real-world dataset exceeds that of our scenario, we select those features with the most distinct values in the original dataset and randomly sub-sample the appropriate number of samples from the real-world datasets for our experiments.
D. Background on Conditional Flow-matching
Flow matching, initially used in image synthesis leverages normalizing flows (Papamakarios et al., 2021b) to model arbitrary distributions. Continuous normalizing flows (Lipman et al., 2022) have emerged as a potent tool for modeling complex distributions. For example, recent advancements have shown its effectiveness in state-of-the-ard image generation, outperforming -aassdsooo-baddd methods in likelihood and sample quality on ImageNet (Lipman et al., 2022). Techniques like Flroo have accelerated class-conditional and text-to-image generation, setting new benchmarks (Zhao et al., 2024). Additionally, applying flow matching in latent spaces of pretrained autoencoders has enhanced computational efficiency and scalability for high-resolution image synthesis (Dao et al., 2023). Similarly, flow-based models have been successfully applied to protein structure prediction, improving accuracy and efficiency in modeling complex protein conformations (Yim al., 2024; 2023).
In the area of Limulation-based inference, Wildberger et al. (2024) introduce the idea of using continuous normalizing flows in order to efficiently approximate complex posterior distributions. In particular, they apply the framework to the field of gravitational inference, substantially outperforming approaches based on discrete flows. Furthermore, they demonstrate good performance on the existing SBI-Benchmark (Lueckmann et al., 2021) using a simple MLP-based architecture.
E. Relationship of our approach to density estimation methods
An alternative to flow matching for parameterizing our model Q=|m would be explicit (conditional) density estimation. While knowledge of the explicit density of a distribution can be useful for downstream tasks, we would like to reemphasize that we consider the problem of full Bayesian inference via sampling from the posterior in this paper.
18

---

Can Transformers Learn Full Bayesian Inference In Context?
Popular explicit density estimation methods include i-DODE (Zheng al., 2023), which proposes several techniques for improving maximum likelihood estimation from diffusion ordinary differential equations (ODEs), including velocity parameterization and techniques for variance reduction, leading to faster convergence. Additionally.in (Sahoo et al., 2024) log-like eegllikeihoodd estimation is improved by casting the learned diffusion process as a variational posterior that yields a tighter evidence lower bound on the actual likelihood. (Lienen et al., 2025) propose a novel generative model using iterative Gaussian posterior inference and empirically demonstrate that it yields strong results in log-likelihood estimation. Furthermore, Salazar (2023) use variational inference to learn Bayesian regression trees, which could be used for multivariate density estimation.
Note that it is, in principle, also possible recover the explicit density of Q=1e, which is parametrized in the Flow Matching Framework that is used for our approach (Wildberger et al., 2024).
F. Hyperparameters, Software and Computational Setup In this section, we detail the hyperparameters, used software and computational setups for all our experiments.
F.I ICL
To ensure maximum comparability across different experiments, we fix the hyperparameters for all ICL experiments: For the architecture of the model introduced in Section 3.3, we use the following configuration: The dimensionality of encoder representations is set to 512 and is expanded 1024 in the feed-forward blocks. We use 8 heads and 8 encoder layers with dropout rate of 0.1. For the decoder part we also use 512 as the dimensionality of the representations and 1024 as the intermediate representation in the feed-forware layers and dropout rate of 0.1. Furthermore, 3 simple fully connected layers with adal NN conditioning are used for final processing in the decoder. For the time conditioning, we use 3 simple fully connected layers to map the scalar-valued time ł onto 512 dimensional conditioning vector that is used for the adaLN blocks in the decoder. This yields a model of around 43.1 million parameters. We use no tokenization for either the encoder or the decoder and simple embedding layers to map the encoder- and decoder-input onto the feed nerd-frrwrrd dimensions.
We use an Adam optimizer (Kingma, 2014) with cosine learning rate schedule (Loshchilov & Hutter, 2016), where the maximum learning rate is 5 10-4, the final division factor is 10* and 10 percent of the epochs used for warm-up. We use weight decay parameter of 10 and batch size of 1024 and gradient clipping with maximum gradient norm of one. We use in total 75 million synthetic samples for all scenarios. Of the total number, half, i.e. 37.5 million, are used for training and 10 percent for validation and the remaining 40 percent for testing. Note that we observe convergence of the loss usually much carlier than after this training duration, but fix the number of samples for consistency across experiments. A single L44 GPU is used for the GLM scenarios and single A100 GPU for the FA and GMM cases.
To solve the ODE for the sample generation, dopri5 (Dormand & Prince, 1980) as implemented in Torchdiffeq (Chen, 2018) is used in the adjoint version. We set the relative and absolute tolerance to 10-? 7 The min parameter in the CNF-loss is set to 10-4.
E22 HMC
We use HMC with NUTS kernel (Hoffman et al., 2014) reference for all experiments where no analytical solution is available. We set the number of burn-in samples to 500 and use one chain for all uni-modal problems and three times the number of potential modes in all other cases. More specifically, we use M x 3 chains for all GMM scenarios. The Pyro implementation of NUTS is used for the GLM scenarios (Bingham et al., 2019) and the conceptually identical, albeit computationally faster implementation in Numpyro for the FA and GMM cases (Phan et al., 2019).
F.3. VI
For the variational inference methods, we utilize automatic guide generation based on the ground-truth data-generating processes (Kucukelbir et al., 2017). Pyro is used for the implementation of the probabilistic programs, which we also use to sample the synthetic training data, for the automatic guide generation, and for the implementation of the actual VI methods (Bingham al., 2019). Default hyperparameters, as well as an Adam optimizer (Kingma, 2014) with a learning rate of 10-2 is used for all methods except for AutolAF where learning rate of 10-3 is used. We perform 2000 full-batch gradient update steps for each method.
19

---

Can Transformers Learn Full Bayesian Inference In Context?
ko Zas
14
24
44-
14
Figure Learning curves for GLM sce- nario with a Normal Prior on the coef- ficients| 3 and Inverse Gamma prior on 02
Figure 5: Learning curves for GMM scenario 1 with M components, K 50 datapoints and L dimen- K 50 datapoints and 5 dimen- sions.
Figure 6: Learning curves for GMM scenario 3 with M components,
sions.
G. Runtimes
We single L4 GPU for generating samples based ICL approach and HMC in the GLM scenarios, single A100 for our ICL approach and HMC in the FA and GMM scenarios, and an Intel(R) Xeon(R) CPU @ 2.20GHz CPU with two virtual cores and 40 gigabytes of RAM for the VI methods. Across all considered GLM scenarios, pre-training takes on average 14.89 hours with a standard error of 18.01 minutes. For the FA scenarios, on average 3.95 hours with a standard error of 11.38 minutes is used for pretraining and for the GMM scenarios 10.63 with a standard error of 72.88 minutes.
When applied in order to generate samples for new dataset, the benchmarked VI methods have, expected the lowest runtime. The Laplace approximation the fastest of all methods, while our ICL appraoch has consistently a lower runtime compared to HMC. Overall, the ICL method takes around 2 minutes the GLM tasks, around 30 seconds in the FA scenarios and less than minutes for the inference regarding the GMM tasks.
This difference is especially pronounced in the FA and GMM scenarios. Please note that the runtime of the ICL method also fundamentally depends on the used precision for solving the underlying differential equation where we use a relatively high relative and absolute precision of 10-7, Decreasing this value might lead to significantly faster inference time while maintaining sample quality.
20

---

Can Transformers Learn Full Bayesian Inference In Context?
Table 8: Runtime Metrics for all GLM, FA, and GMM Scenarios
Scenario
Method
Mean Runtime (s)
Laplace Approximation
10.48 (+0.25)
VI: DiagonalNormal
12.02 (#0.26)
VI: MultivariiaeeNormal
13.70 (#0.29)
GLM
VI: Structured Normal
19.81 (+0.98)
VI:AF
15.44 (++.33)
HMC
120.24 (+11.994)
ICL (ours)
107.79 (#17.36)
Laplace Approximation
17.85 (=0.21)
VI: DiagonalNormal
20.94(+0.66)
VI: MultivarriaeNNomal
20.84
FA
VI: Structured Normal
36.17(#0.00
VIHAF
23.75 (+:0.88)
HMC
248.26 (+55.88)
ICL (ours)
31.49 (+4.97)
Laplace Approximation
27.52 (+:0.40)
VI: DiagonalNormal
29.74 (+0.57)
VI: MultivariateNNrmall
30.50 (#0.41)
GMM
VI: Structured Normal
42.44 (#0.44)
VIHAF
33.39 (+0.49)
HMC
239.67 (+32.71)
ICL (ours)
93.88 (=10.4
H. Ablation: Different Learning Rates for VI
To investigate the role of the learning rate parameter for the benchmarked VI methods, we record the performance for learning-rate values of 10-2, 10-3 and 10-4 across a prototypical GLM, a FA and a GMM scenario, where we use 10 synthetic and 10 real-world datasets. In summary, while we find the VI methods to often be quite robust to the choice of the learning rate, those results also confirm our choice of setting the learning rate to 10-2 2 for the Laplace approximation, variational inference with diagonal normal distribution, multivariate normal distribution and a structured normal distribution, and value of 10 for the VI approach with inverse autoregressive flows.
For the Gi--sccnai,,, we find in terms of the C2ST metric that VI with an ordinary multivariate normal distribution and VI with structured normal distribution and learning rate of 10-2 are the best models on the synthetic data. While MMD also indicates that this learning rate yields ideal results for those models, VI with inverse auoregressive flows has good values across the different learning rates with the minimum for 10 -3 The W2 metric indicates a similar tendency.
Regarding the learning rate for the FA scenario, one can first see that no single learning rate seems to dominate substantially given the variance of the results. However, the synthetic data for the Laplace approximation, well as VI with diagonal normal distribution, multivariate normal and structured normal distribution, the lowest average result is obtained for learning rate of 10-2, while for VI with inverse autoregressive flows the best performance is obtained when the learning rate equals 10 10-3, The real-world results are the best for VI with structured normal distribution and learning rate of 10 10-2
For the GMM scenario, we find that VI with a diagonal, structured and ordinary normal distribution obtain the best results, namely for learning rates of 10-2 and 10-3, taking the variance into account. Just considering the averages leads to the conclusion that 10-2 is the best choice here. The results on the real- world data confirm that 10-2 is the optimal choice for VI with diagonal normal and ordinary multivariate normal, while VI with inverse autoregressive flows has good results across all choices regarding the learning rate.
21

---

Can Transformers Learn Full Bayesian Inference In Context?
Table 9: Results of VI methods with different learning rates on 10 synthetic and 10 real-world datasets: Linear regression with normal prior on the coefficients (3 and an inverse gamma prior on the variance o² (scenario 1). Comparison to HMC samples. All results within two standard errors of the best average result are marked in bold.
Model
LR
Synthetic Evaluation
Roal-World Evaluation
C2ST (L)
MMD ())
W/ (L)
C2ST (U)
MMD (1)
Wi (U)
Laplace Approximation
1c-2
1.000 (#: 0.000)
2.342 (#: 0.390)
2.121 (#: 0.100)
1.000 (#: 0.000)
2.134 (# 0.107)
2.095 (= 0.062)
Laplace Approximation
le-3
1.000 (1: 0.000)
2.341 (4: 0.389)
2.121 (1: 0,100)
1.000 (1: 0.000)
2.133 (1 0.108)
2.095 (1: 0.062)
Laplace Approximation
lc-4
1.000 (#: 0.000)
2.341 (#: 0.389)
2.121 (#: 0.100)
1.000 (# 0.000)
2.133 (# 0.108)
2.095 (= 0.062)
VI: DiagonalNornal
le-2
0.892 (d: 0.074)
0,921 (d: 0.374)
1,411 (t: 0.174)
0.889 (t: 0.062)
0.819 (+: 0.343)
1,339 (:: 0.190)
VI: DiagonalNormal
1e-3
0,966 (.L 0.024)
1.588 (1 0.540)
1.672 (1 0.203)
0.981 (1 0.017)
1.685 (1 0.331)
1.739 (1 0.139)
VI: DiagonalNormal
1c-4
0.971 (#: 0.010)
1.572 (+ 0.300)
1.666 0.081)
0.849 0.030)
0.575 (= 0.127)
1.221 (= 0.098)
VI: MultivariateNormal
lc-2
0.725 (+: 0.064)
0.523 (+: 0.242)
1.114 (#: 0.261)
0.625 (#: 0.051)
0.470 (+: 0.066)
0,918 (= 0.119)
VI: MultivariateNormal
1o-3
0,964 (#: 0.008)
1.455 (+: 0.327)
1.617 (#: 0.100)
0.853 (#: 0.052)
0.634 (= 0.266)
1.238 (= 0.151)
VI: Muliiaariateoormal
1e.4
0,984 (+: 0.005)
1.848 (1: 0.324)
1.773 (t: 0.079)
0.899 (:: 0.020)
0.807 (+: 0.094)
1.345 (= 0.079)
VE: Structured Normal
lc-2
0.734 (1:: 0.063)
0.541 (1: 0.254)
1.119 0.264)
0.670 (1: 0.047)
0.467 [1 0.086)
1.060 (= 0.130)
VI. Structured Normal
1e-3
0.882 (+: 0.042)
0.719 (+ 0.315)
1.335 0.149)
0.776 0.045)
0.473 + 0.081)
1.064 (= 0.131)
VI: Structured Normal
le-4
0.890 (+ 0.027)
0.710 (+ 0.290)
1.347 0.138)
0.771 (+ 0.049)
0.468
(+ 0.078)
1.062 (= 0.128)
VI. TAF
le-2
0.840 (+ 0.036)
0.502 (+ 0.262)
1.272 (+ 0.170)
0.614 (+ 0.045)
0.455
(+ 0.048)
0.957 (- 0.105)
VE IAF
1e:3
0.797 (+: 0.065)
0.485 (+: 0.556)
1.169 (t: 0,313)
0.619 (+: 0.036)
0.469
(+ 0.064)
0,989 (= 0.124)
VE IAF
lc-4
0.803 (+: 0.068)
0.475 (+: 0.535)
1.162 (#: 0.291)
0.612 (#: 0.034)
0.457
( 0.055)
0.977 (= 0.113)
Table 10: Results of VI methods with different learning rates on 10 synthetic and 10 real-world datasets: Factor analysis with Gaussian priors on the weights and the latents and K = 25 datapoints, P = 5 features, and dimensionality of the latents 7dim =3 3 (scenario 3). Comparison to HMC samples. All results within two standard errors of the best average result are marked in bold.
Model
LR
Synthetic Evaluation
Woal-World&oo Evaluation
C2ST (L)
MMD (1)
W22 (L)
C2ST (L)
MMD (4)
We (4)
Laplace Approximation
1e-2
1.000 (1: 0.000)
3.449 (1: 0.821)
1.773 (1 0.539)
1.000 (1 0.000)
2.703 (1 0.312)
0.362 (1: 0.017)
Laplace Approximation
1e-3
1.000 (1 0.000)
4,288 (1 0.853)
2.263 (1 0.732)
1.000 (1 0.000)
2,896 [1 0.238)
0.376 (1 0.022)
Laplace Approximation
lc-4
1.000 (+ 0.000)
4.252 (+ 0.61
2.122 (# 0.430)
1.000 (= 0.000)
2.805 (= 0.181)
0.368 (= 0.017)
VI: DiagonalNornal
1e-2
0.998 (+ 0.002)
2.880 (+: 1.046)
1.457 0.559)
0.944 0.008)
1.022 (+ 0.067)
0.230 (+ 0.010)
VI: DiagonalNormal
1c-3
0.998 (#: 0.002)
2.973 (+: 0.834)
1.465 0.540)
0.941 (#: 0.006)
0.997 (= 0.056)
0.229 (= 0.010)
VI: DiagonalNormal
1c-4
1.000 (#: 0.001)
3.416 (+: 0.761)
1.602 (# 0.437)
0.943 (#: 0.009)
0.997 (= 0.057)
0.229 (= 0.010)
VI: MultivariateNormal
1e-2
0,993 (1:: 0.007)
2.969 (1: 1.089)
1.506 (1 0.659)
0.929 (1 0.007)
0.957 [+ 0.048)
0.224 (= 0.010)
VI: MultivariateNormal
1e-3
0.996 (+ 0.004)
3.140 (+ 0.910)
1.570 (+ 0.625)
0.934 (+ 0.009)
0.971 (+ 0.054)
0.225 (- 0.010)
VI: MultivariateNormal
le-4
0.997 (+ 0.007)
3.464 (+ 0.791)
1.639 (+ 0.426)
0.934 (+ 0.005)
0.962 (+ 0.049)
0,225 (+ 0.010)
VI: Structured Normal
1c-2
0.998 (= 0.002)
3.005 (+ 0.871)
1.481 (+ 0.504)
0.947 (#: 0.005)
1.003 (= 0.066)
0.230 (= 0.009)
VI: Structured Normal
1c-3
0.999 (#: 0.001)
3.244 (+ 0.665)
1.619 (+ 0.559)
0.948 (+ 0.007)
1.033 ( 0.078)
0.232 (= 0.009)
VI: Structured Normal
lc-4
0.999 (1:: 0.001)
3.119 (1: 0.612)
1.487 (1: 0.400)
0.943 (1: 0.007)
0,998 (1 0.056)
0.229 (= 0.010)
VI: IAF
1c-2
0.939 (#: 0.040)
2.836 (+: 0,293)
1.247 (#: 0.297)
0.944 (# 0.008)
1,518 (= 0.048)
1.332 (= 0.027)
VI IAF
1e-3
0.927 (1: 0.047)
2.758 (1 0.342)
1.195 (1L 0.331)
0.949 (1: 0.009)
1,560 (1 0.031)
1.392 (= 0.024)
VI IAF
1c-4
0.842 (=: 0.038)
2.862 (+: 0.296)
1.281 0.292)
0.943 0.008)
1.493 + 0.039)
1.302 (= 0.039)
22

---

Can Transformers Learn Full Bayesian Inference In Context?
Table 11: Results of VI methods with different learning rates 10 synthetic and 10 real-world datasets: Gaussian Mixture
Model with K 50 datapoints, L 1 features (univariate case), M 5 components, À = 3, and Outir 1 (scenario 1), Comparison to HMC samples. All results within two standard errors of the best average result are marked in bold.
Model
LR
Synthetic Evaluation
Real- World Evaluation
C2ST (1)
MMD ())
W22 (L)
C2ST (J)
MMD (
W2 (L)
Laplace Approximation
le-2
1,000 (1 0.000)
4.380 (1. 1.386)
4.838 1.521)
1.000 0.000)
4.588
1.229)
6.813 (1 1.697)
Laplace Approximation
1c-3
1.000
0.000)
3.893 (+ 1.433)
4.010 1,233)
1.000 0.000)
4.699
1.193)
6.986 0.981)
Laplace Approximation
lc-4
1.000
0.000)
4.463 (+ 1.117)
4.610
1.027)
1.000 0.000)
4.710
1.205)
6.995 0.869)
VI: DingonalNornal
1c-2
0.979 (#: 0.138)
1.370
(#: 1,394)
3,522 (#: 1.634)
0.985 (#: 0.030)
2.384 (# 1.318)
6.202 (= 1.747)
VI: DiomoaaNNornal
1e-3
0.990 (d: 0.096)
1.454
(d: 1,454)
3.650 (:: 1.743)
0.999 (1: 0.002)
3.026 (1: 0.977)
6.959 (1 0.890)
VI: DiogonalNormal
le-4
1.000 (1 0.001)
2.390
(1 1.177)
4.903 (1: 1,278)
0.998 (1L 0.007)
2,830 (1 1.001)
7.007 (1 0.987)
VI MulrivariateNormal
le-2
0.978
0.119)
1.351 1.410)
3.474
1.604)
0.987 0.024)
2.375
1.304)
6.189 1.761)
VI. MuliivariateNormal
1e-3
0.980
0.089)
1.476
1.480)
3.681 1.734)
0.997 0.008)
2.808
1.014)
6.964 (+ 0.944)
MultivariatoNormal
1c-4
1.000 (#: 0.001)
2.114 (#: 1.140)
4.532
1.187)
0.997 0.007)
2.799
(# 1.012)
6.963 (= 0.950)
VI: Structured Normal
1c-2
0.958 (#: 129)
1.246 (+: 1.615)
3.225 (#: 1.701)
1.000 (=: 0.001)
2.911
(= 0.753)
6.675 (= 1.403)
VI: Structured Normal
1e-3
0.979 (1 0.092)
1.593 (1: 1,561)
3.395 (1 1.440)
0.998 (1 0.007)
2,882
[1 1.070)
6.968 (= 0.941)
VI: Structured Normal
le-4
1.000 (+ 0.001)
2.270 (+ 1.133)
4.733 (+ 1.162)
0.997 0.009)
2.802
1.012)
6.953 (= 0.948)
IAF
1e-2
0.998
0.003)
1.539 (1: 0.691)
8,371
0.750)
0.987 0.022)
0.799)
8.082 1.352)
IAF
1c-3
0.997 (#: 0.004)
1.443 (+ 0.564)
8.517
0.820)
0.988 0.020)
1.304
(= 0.855)
8.425 (= 1.281)
IAF
1c-4
0.997 (+: 0.004)
1.602 (+ 0.628)
7,888 0.783)
0.987 (= 0.020)
1,380
0.848)
7.729 (= 1.322)
23

---

Can Transformers Learn Full Bayesian Inference In Context?
I. Detailed Experimental Results
In this section, we describe our experimental results detail, discussing how different scenarios for GLMs, FA and GMMs affect the performance of different approaches.
I.I. Generalized Linear Models
Dimension
Dimension
Dimension
Dimension
2.0
2.0
ICL
2
2
Gold Slandard (HMC)
1.5
VI: Diagonal Normal
Value
Value
Value
Value
VI: Multivariate Normal
1.0
1.0
Structured Normal
0.5-
0.5
VI: IAF
2
0
2
0.0
1
2
0.0
2
Density
Density
Density
Density
Figure 7: Density plots for first three the marginals of the posterior in GLM with gamma prior the coefficients 3, and an inverse gamma prior on the variance o² of the responses. The data is part of the Miami housing 2016 dataset.
Table 45 contains detailed results regarding the performance of the proposed ICL and the reference VI approaches. In summary, we find that on the synthetic data, our ICL method has the overall best performance, or a performance not significantly worse than that the best model, with respect to the C2ST metric.4 More specifically, ICL significantly outperforms all other models in out of seven cases w.r.t. the C2ST and also the MMD metric. While the W2 metric exhibits larger variance, also indicates that on the synthetic data, ICL yields the significantly best result in those cases.
On the real-world data, the differences between ICL and VI are less pronounced, and ICL attains the best average result without any other model within two standard errors in three scenarios in terms of the C2ST metric. ICL. is among those models not significantly worse than the best in four cases with respect to the C2ST metric, in six cases in terms of the MMD metric, and also in six cases in terms of W2.
In scenario 1, which is linear regression scenario with a normal prior on the coefficients 33 and an inverse gamma prior on the variance o², ICL and HMC show a similarly large agreement with the analytical solution. Furthermore, the VI approaches with ordinary multivariate normal distribution, a structured normal distribution well as the approach based on inverse autoregressive flows also show large agreement with the analytical solution, which is to be expected since scenario is has conjugate prior structure yielding multivariate t-distribution for the posterior of the coefficients (Murphy, 2023).
Scenario and scenario 4 are those where an intercept is included in the generative structure the GLM. The notably superior performance of the ICL approach in those two cases might be explained by its ability to model distributions with substantially different variances different dimensions better than VI. Similarly, the posterior in scenario is determined by the gamma prior the coefficients leading to (slightly) skewed posterior distribution, which might explain the good relative performance of ICL. See Figure 7 for a plot of the marginals of the posterior this scenario on the Miami housing 2016 dataset.
Finally, scenarios 6 and 7 demonstrate the versatility of the ICL. method terms of posterior inference for logistic regression and regression with a gamma response.
*e refer to a difference that larger than two standard deviations as "significant".
24

---

Can Transformers Learn Full Bayesian Inference In Context?
Table 12: Generalized Linear Models: Evaluation on 50 synthetic and 17 worl-world datasets for seven different scenarios. All results within two standard errors of the best average result for each scenario are marked in bold.
Synthetic Evaluation
Real-World Evaluation
Scenario
Model
C2ST ())
MMD ())
W22 (3)
C2ST ())
MMD (U)
W22 (L)
Luplace Approximation
1.000 (#: 0.000)
2.738 (#: 0.721)
0,825 (2 0.279)
1.000 (#: 0.000)
2.150 (= 0.323)
0.642 (= 0.124)
VI: Normall
0.904 (+ 0.076)
1.452 (+ 0.984)
0.669 (+ 0.301)
(17997 (+ 0.083)
0.612 [+ 0.511)
0.414 (+ 0.152)
VI: Mulrivariateormal
0.750 (+: 0.128)
0.735 (:: 0.733)
0.565 (:: 0.292)
0.607 (+: 0.070)
0.167 ( (.16))
0.301 (# 0.123)
Scenario
VI: Structured Normal
0.753 (:: 0.126)
0.736 (4: 0.737)
0.570 (4: 0,310)
0.600 (:: 0.070)
0.169 (+: 0.214)
0.306 (#: 0.131)
VI: IAF
0.777 (#: 0.122)
0.864 (#: 0,844)
0.725 0.523)
0.683 0.132)
0.440 (4 0.559)
0.503 (# 0.383)
HMC
0.745 (+ 0.130)
0.722 (+ 0.732)
0.569 0.301)
0.595 (+: 0.075)
0.173 (+ 0.213)
0.321 (+ 0.140)
ICL (ours)
0.765 (1: 0.123)
0.767 (+: 0.727)
0.585 0.301)
0.614 (#: 0.074)
0.175 ( 0.219)
0.310 0.138)
Laplace Appreximattiii
Approximation
1.000 (+ 0.000)
4.853 (+ 2.333)
5.770 (+ 5.946)
1.000 (+ 0.000)
2.572 (+ 0.206)
0.809 (+ 0.149)
VI: DiagonalNormal
0.957 (#: 0.091)
3.906 2.679)
5.628 6.092)
0.892 (#: 0.044)
0.847 ( 0.389)
0.530 0.175)
Scenario 2
VI: MuliivriittNNoraal
0.910 (+: 0.131)
3,407 (+: 2.781)
5,584 (#: 6.104)
0.820 (+: 0.031)
0.243 (+ 0.148)
0,408 0.118)
VI: Structured Normal
0.908 (+: 0.119)
3.139 (+: 2.763)
5,480 (#: 6.164)
0.824 (#: 0.023)
0.215 (= 0.110)
0,392 (= 0.109)
VI: IAF
0.968 (+ 0.063)
4.416 (+ 2.473)
7.474 (+ 6.235)
01888 (+ 0.067)
0.921 [+ 0.860)
0.942 (+ 0.733)
ICL (ours)
0.839 (+: 0.072)
0.707 (#t: 0.658)
1,111 (#: 0,300)
0.768 (+: 0.033)
0.143 ( 0.089)
0.411 (= 0.094)
I Laplace Approximation
1.000 (+ 0.000)
2.203 (+ 0.997)
1.170 (+ 0.949)
1.000 (+ 0.000)
1.841 [+ 0.185)
0.729 (+ 0.175)
VI: DiagonalNormal
0.866 (+: 0.101)
1.069 (+ 1.150)
0.846 (# 0.747)
0.797 (+: 0.083)
0.526 ( 0,361)
0.480 (= 0.207)
VI: MulrivariateNommal
0.656 (#: 0.131)
0.445 (+ 1.061)
0.660 0.737)
0.560 (+: 0.035)
0.032 (=: 0.028)
0.249 (= 0.069)
Scenario 3
VI: Structured Normal
0.653 (:: 0.125)
0.421 (+ 0.993)
0.659 (# 0.736)
0.552 (+: 0.028)
0.027 (+ 0.015)
0.239 (# 0.055)
0.751 (+ 0.148)
0.939 (+ 1.349)
0.964 (+ 0.924)
0.673 (+ 0.141)
0.399 [+ 11543)
0.563 (+ 0.433)
ICL (ours)
0.611 (+: 0.070)
0.089 (1 0.114)
0.423 0.348)
O1 0.027)
0.037 ( 0.026)
0.257 0.044)
Laplace Approximation
1,000 (+ 0.000)
3.511 (+ 2.025)
2.166 (+ 1.722)
1.000 (+: 0.000)
2.011 (+ 0.058)
0.993 (+ 0.144)
VI: DiagonalNormal
0.968 (#: 0.036)
2.798 (1 2.255)
2.065 1.745)
0,916 (2: 0.040)
0.928 ( 0.339)
0.732 0.181)
Scenario 4
VI: MulivariiteNorraal
0.855 (+: 0.123)
1.648 (+: 2.052)
1,853 1.745)
0.771 (+: 0.017)
0.087 (+ 0.030)
0,539 0.070)
VI: Structured Normal
0.847 (+: 0.116)
1.505 (+: 1,978)
1.889 (#: 1.883)
0.769 (+: 0.012)
0.083 (+: 0.018)
0.543 0.070)
V1: IAF
0.942 (+ 0.077)
3.029 (+ 2,210)
3.554 (+ 2.715)
01833 (+ 0.069)
0.636 [+ 0.756)
0.978 (+ 0.600)
ICL (ours)
0.753 (+: 0.049)
0.171 (1 0.153)
0.631 (1: 0,294)
0.762 (+: 0.015)
0.105 ( 0.046)
0.597 (= 0.104)
Laplace Approximation
1,000 (+: 0.000)
2.060 (+ 0.472)
0.797 (+: 0.577)
1.000 (+: 0.000)
1.982 (+ 0.126)
0.623 (# 0.084)
VI: DiagonalNormal
0.866 (#: 0.085)
0.954 (1:: 1.022)
0.651 (1 0,549)
0.810 (+: 0.036)
0.441 [ 0.252)
0.384 (= 0.089)
VI: MuliivariatNNoraal
0.765 (+: 0.100)
0.537 (+: 1.019)
0.633 (#: 1.067)
0,711 (+: 0.038)
0.148 (+:: 0.093)
0.279 (# 0.056)
Scenario 5
VI: Structured Normal
0.758 (:: 0.098)
0.447 (4: 0.818)
0.572 (:: 0.816)
0.705 (+: 0.032)
0.140 ( 0.081)
0.269 (= 0.045)
VI: IAF
0,814 (#: 0.105)
0.953 (1: 1.165)
0.881 (# 1.067)
0,777 (+: 0.106)
0.684 (1: 0.939)
0.625 (= 0.525)
ICL (ours)
0.621 (=: 0.063)
0.067 (1: 0.080)
0.299 0.195)
0.610 (=: 0.045)
0.046 (1 0.020)
0.242 0.038)
Luplace Approximation
1.000 (+: 0.000)
2.026 (+: 0.027)
1.612 (# 0.162)
1.000 (+: 0.000)
1.993 ( 0.032)
1.299 (= 0.106)
VI: DiagonalNormal
0.724 (= 0.060)
0.185 (1 0.082)
0.787 0.078)
0.703 (2: 0.039)
0.147 [ 0.063)
0.637 0.089)
VI: MulrivariateNormall
0.534 (+: 0.018)
0.014 (4: 0.006)
0.581 0.074)
0.538 (+: 0.019)
0.016 [+: 0.007)
0,466 0.029)
Scenurio 6
VI: Structured Normal
0.536 (+: 0.016)
0.014 (+: 0.005)
0.583 (2 0.071)
0.536 (+: 0.019)
0.017 (+: 0.009)
0.469 (= 0.033)
VI: IAF
0.542 (+: 0.026)
0.031 (+ 0.031)
0,613 (# 0.092)
0.535 (+: 0.015)
0.015 ( 0,006)
0,467 (= 0.031)
ICI (ours)
0.532 (+ 0.019)
0.016 (+ 0.008)
0.590 (+ 0.066)
0.556 (+ 0.017)
0.035 [+ 0.015)
0.504 0.038)
Luplace Approximation
1.000 (+: 0.000)
3.559 (+ 1.933)
1.347 (# 1.067)
1.000 (+: 0.000)
2.016 (+ 0.080)
0,763 (= 0.174)
VI: DiagonalNormal
0.938 (+: 0.074)
2.536 (1: 2.097)
1.142 (1 0.993)
0.936 (+: 0.024)
1.029 ( 0.255)
0.579 (= 0.181)
VI: Nuliivariateoommall
0,814 (+: 0.181)
1.999 (+ 2,283)
1.033 (+: 0.969)
0.741 (+: 0.020)
0.093 (+ 0.025)
0.391 (+ 0.074)
Scenurio 7
VI: Structured Normal
0.824 (:: 0.177)
18911 (4: 2.127)
1.041 (1: 0.934)
0.734 (+: 0.025)
0.072 ( 0.019)
0.385 (= 0.065)
VI: IAF
0.939 (+: 0.091)
2.707 (+: 1.712)
1,590 0.820)
0.864 (+: 0.093)
0.830 ( 0.697)
1.064 (= 0.616)
ICL (ours)
0.700 (+ 0.116)
0.317 (+ 0.355)
0.400 0.286)
0.773 (+ 0.048)
(+ (l457)
0.559 0.256)
25

---

Can Transformers Learn Full Bayesian Inference In Context?
1.2. Factor Analysis
Table 46 contains detailed results regarding FA for 50 synthetic and 17 real-world datasets across 6 different scenarios. We find that overall the ICL method has a very high agreement with the gold standard HMC reference with scores of more than than 56 percent in five scenarios the synthetic data. In comparison, the C2ST metric is almost saturated for all considered VI methods. For MMD and W2 the ICL method is again the best.
The real-world datasets show a similar picture except for scenario 4 where C2ST and MMD indicate that VI with inverse autoregressive flows performs best. The W2 metric, however exhibits a relatively large variance in those cases and does not yield significant results regarding the best performance.
Table 13: Factor Analysis: Evaluation on 50 synthetic and 17 real-world datasets for six different scenarios. All results within two standard errors of the best average result for each scenario are marked in bold.
Synthetic Evaluation
Real-World Evaluation
Scenario
Model
C2ST (1)
MMD ())
W/ (
C2ST (J)
MMD ())
W/ (L)
I Laplace Approximation
1.000 (#: 0.000)
3.459 (+: 1.553)
1,987 (+: 1.363)
1.000 (#: 0.000)
2.487 (+: 0.454)
0.875 (# 0.036)
VI: DiagonalNormal
1.000 (#: 0.001)
4.695 (+: 1.488)
2.865 (#: 1.681)
0.979 (#: 0.008)
1.283 (#: 0.225)
0.625 (= 0.058)
VI: MultivariateNormal
0.998 (+: 0.003)
4.163 (+: 1.473)
2.603 (+ 1.959)
0.966 (+: 0.010)
1,213 (+ 0.260)
0.608 (# 0.047)
Scenario 1
VI: Structured Normal
0.997 (t: 0.004)
4.655 (+:: 1.189)
2,700 (:: 1.333)
0.979 (t: 0.010)
1.231 (#: 0.132)
0.611 (# 0.041)
VI: IAF
0.953 (+: 0.104)
3,992 (+ 2,089)
2.750 (+: 1.838)
0.849 (+ 0.075)
0.772 (+ 0.335)
0.503 (+ 0.063)
ICL (ours)
0.552 (#: 0.028)
0.034 (1: 0.034)
0.289 0.083)
0.606 (#: 0.038)
0.068 (4: 0.069)
0.265 (# 0.078)
Laplace Approximation
1.000 (+: 0.000)
3.687 (+: 1.661)
1.954 (+: 1.129)
1.000 (4: 0.000)
1.690 (+: 0.182)
0.5%8 (# 0.058)
VI: DiagonalNormal
0.998 (#: 0.002)
3.135 (1: 1.482)
1.629 0.938)
0.975 (#: 0.010)
1.156 (#: 0.068)
0.496 (= 0.052)
VI: MultivariateNomma
0.989 (+ 0.009)
2,945 (+ 1.019)
1.482 0.683)
0,951 (+: 0.025)
0.764 [+ 0.053)
0.421 (+ 0.052)
Scenario 2
VI: Structured Normal
0,984 (#: 0.031)
3.790 (+: 1.572)
2.106 1.429)
0.958 (#: 0.025)
1.001 (#: 0.126)
0.465 0.056)
VI: IAF
0,966 (#: 0.066)
3.523 (+ 1.340)
2.153 (+: 0.968)
0.799 (+: 0.058)
0.462 (+: 0.226)
0.342 0.070)
ICL (ours)
0.542 (#: 0.006)
0.017 (+ 0,006)
0.244 (= 0.033)
0.622 (+: 0.032)
0.098 ( 0,039)
0.287 (= 0.046)
Laplace Approximation
1.000 (+: 0.000)
4.137 (+ 0,932)
2.188 (# 1,011)
1.000 (#: 0.000)
3.653 (+ 0.183)
0.473 (= 0.026)
VI: DiagonalNormal
0.999 (+: 0.002)
3.339 (+ 0,985)
1.722 (# 0.870)
0.951 (+: 0.007)
1.114 ( 0,080)
0.245 0.016)
VI: Multivariate
0.994 (+ 0.007)
3.189 (+ 0.960)
1.644 (+ 0.859)
0.945 (+ 0.007)
1.085 (+ 0.082)
0.242 (+ 0.015)
Scenario 3
VI: Structured Normal
0.997 (+: 0.003)
3.159 (1 0.968)
1.614 (#: 0.793)
0.942 (+: 0.009)
1.084 ( 0.071)
0.242 == 0.018)
VI: IAF
0.990 (+: 0,01
3.145 (+: 1 1,203)
1.705 (# 0,990)
0.928 (+: 0.015)
1.022 (=: 0.093)
0.235 (= 0.018)
ICL (ours)
0.537 (#: 0.023)
0.024 (1: 0.021)
0.259 0.088)
0.609 (#: 0.019)
0.124 (=: 0.037)
0.179 (= 0.018)
Luplace Approximation
1.000 (+: 0.000)
4.354 (1: 0.572)
3.339 (+: 0.932)
1.000 (+: 0.000)
6.617 (+: 0.259)
0.598 (= 0.135)
VI: DiagonalNormal
1.000 (#: 0.000)
3.396 (1 0.591)
2.420 0.720)
0.977 (+: 0.003)
1.499 (#: 0.066)
0.096 (= 0.003)
Mulrivariate
0,599 (+ 0.001)
3.477 (+ 0.567)
2.479 (+ 0.848)
0.973 (+ 0.008)
1.484 [+ (1097)
0.096 (+ 0.005)
Scenario 4
VI: Structured Normal
1.000 (#: 0.000)
3.421 (1 0.610)
2.481 0.884)
0.973 (#: 0.007)
1.474 (#: 0.078)
0,095 0.004)
VI: IAF
0.999 (#: 0.001)
3.269 (1 0.552)
2.307 0.779)
0,961 (+: 0.018)
1.337 (# 0.142)
0,092 0.005)
ICL (ours)
0.684 (+ 0.060)
0.198 (+ 0.141)
0.918 (+ 0.246)
0.988 (+ 0.003)
1.764 (+ 0.026)
1,248 (# 0.008)
Luplace Approximation
1,000 (t: 0.000)
4,456 (+: 0,785)
2. 2.608 (#: 0.946)
1.000 (+: 0.000)
4.559 [+: 0,494)
0.663 (# 0.127)
VI: DiagonalNormal
0.999 (+: 0.002)
3.520 (+: 1.073)
2.012 (# 0.886)
0.944 (+: 0.010)
1.007 ( 0.129)
0.261 (= 0.036)
VI: MultivariateNommo
0.995 (+ 0.007)
3.422 (+ 1.021)
1.982 (+ 0.814)
0.930 (+ 0.017)
0.964 [+ 0.111)
0.255 (+ 0.038)
Scenario 5
VI: Structured Normal
0.998 (1: 0.005)
3.369 (+: 1.044)
1.916 (1: 0.852)
0.934 0.01
0.996 ( 0.133)
0.259 (= 0.035)
VI: IAF
0.992 (+: 0.012)
3.166 (1: 0.967)
1.761 (# 0.671)
0.910 0.011)
0.892 ( 0.094)
0.247 (= 0.037)
ICL (ours)
0.535 (+: 0.016)
0.021 (+ 0.011)
0.279 (+ 0.060)
0.886 (+: 0.017)
1.207 (+ 0.101)
1.002 (+ 0.042)
Laplace Approximation
1.000 (+: 0.000)
3.942 (4: 0.971)
2.624 (+: 1 1.682)
1.000 (+: 0.000)
3.319 (+: 0.196)
0.377 (= 0.020)
VI: DiagonalNormal
0.998 (1: 0.002)
3.214 (+: 1.072)
2.209 (+: 1.543)
0.949 (:: 0.008)
1.196 (+: 0.093)
0.210 (= 0.01
Multivariate Normall
0,991 (+ 0.013)
3.056 (+ 1,237)
2.189 (+ 1.698)
0,938 (+ 0.009)
[+ (.075)
0.205 0.012)
Scenario 6
VI: Structured Normal
0.997 (1: 0.005)
3.279 (1 1.07))
2.276 L.787)
0,944 (1: 0.006)
1.661 (1 0.066)
0.208 0.012)
VI: IAF
0.989 (+: 0.029)
3.027 (4: 0,910)
1.936 1.060)
0.865 (+: 0.027)
0.822 (+: 0.106)
0.179 0.015)
ICL (ours)
0.543 (4: 0.021)
0.023 (4 0.015)
0.345 (#: 0.173)
0.666 (+: 0.020)
0.200 (+: 0.034)
0.224 0.014)
26

---

Can Transformers Learn Full Bayesian Inference In Context?
I.3. Gaussian Mixture Models
We summarize the results of the ICL approach and the different VI methods regarding the GMM scenarios in Table 47. First, one can note that on the synthetic data, the ICL approach has much lower C2ST score for scenario I and scenario 2 than the other methods. However, for scenarios 3 and 4, C2ST saturates, at least almost saturates for all approaches. The MMD metric, however, shows that ICL not only has high agreement with HMC in scenarios and 2, but that it attains the significantly best result in scenarios and 4 as well. This is supported by the W2 metric, which has the significantly lowest values for ICL in scenarios 2,3 and 4.
Analogously, on the real-world data, MMD shows that ICL is the best approach in all four scenarios without any other model coming into the two standard-deviation range. While the C2ST score is the lowest in scenario and scenario 2 for ICI., it saturates for cases 3 and 4.
Table 14: Gaussian Mixture Models: Evaluation on 50 synthetic and 17 real-world datasets for six different scenarios. All results within two standard errors of the best average result for each scenario are marked in bold.
Synthetic Evaluation
Real- World Evaluation
Scenario
Model
C2ST (L)
MMD (J)
W2 (L)
C2ST (1)
MMD (L)
W2 (L)
Laplace Approximation
1.000 (+: 0.000)
3,367 (+ 1.030)
4.341 2.018)
1,000 (#: 0.000)
3,374 (+ 0,941)
6,440 1,994)
VI: DiagonaiNormal
0.988 (# 0.013)
1.175 (# 1.189)
2.961 1.669)
0.995 0.006)
1.919 1.217)
5.145 2.489)
VI: CaltivariateNormaa
0.988 (+ 0.013)
1.135 (+ 1.149)
2.926 1.651)
0,994 0.007)
2.007 (4 1.367)
5.379 2.845)
Scenario 1
VI: Structured Normal
0.987 (+: 0.015)
1.126 (+ 1.145)
2.944 (#: 1.663)
0.993 (+ 0.009)
1.943 (4 1.359)
5,313 (# 2.737)
VI: IAF
0.989 (#: 0.013)
1.017 (# 1.036)
3,104 (#: 1.523)
0.995 (#: 0.010)
1.888 (+: 1.051)
5,402 (# 2.310)
ICI. (ours)
0.760 (+ 0.092)
0.303 (+ 0.548)
2.095 (+ 1.692)
0.847 (+ 0.082)
0.486 (+ 0.623)
4.054 (+ 2.782)
Laplace Approximation
1.000 (:: 0.000)
2.864 (1: 0.607)
5,407 (4: 2.320)
1,000 (:: 0.000)
2.928 (4: 0.438)
7.228 (:: 1.323)
VI: DiagonalNormal
0.989 (# 0.024)
1.425 (= 0.829)
4.933 2.379)
0,998 (#: 0.003)
1,525 (+: 0.356)
6.091 0,931)
VL MulivariiaeNormal
0.991 (#: 0.021)
1.532 (= 0,940)
5.119 2.521)
0.999 0.002)
1.619 (1: 0.269)
6.258 0.872)
Scenario 2
VI: Structured Normal
0.992 (1: 0.017)
1.487 (+: 0.899)
5.085 (1: 2.530)
0.999 (:: 0.002)
1,580 (4: 0.337)
6.241 (1: 0.960)
VI: IAF
0.992 (#: 0.021)
1,319 (+ 0,854)
5.265 (# 2.534)
0.998 (#: 0.004)
1.256 (+: 0.320)
6,201 (= 0.892)
ICL (ours)
0.812 (#: 0.061)
0.159 (= 0.154)
2.314 (# 0.926)
0.937 (= 0.041)
0.282 ( 0.131)
3,947 (= 1.055)
Laplace Approximation
1.000 (1: 0.000)
3.631 (+: 1,362)
16.387 (+: 19.604)
1,000 (:: 0.000)
3,009 (.1 0.768)
37,034 ( 7.178)
VI: NiagonaiNormal
0.99 0.011)
2.127 (+ 1.479)
16.864 (+ 19.301)
0.992 (+ 0.018)
2,429 (+ 0.516)
35.355 (+ 6.608)
VI: MuttivariateNNoma
0.997 (+: 0.009)
2.076 (+ 1.388)
16.938 (+: 19.636)
0,993 (+: 0.016)
2.427 (+ 0.510)
35,312 (+: 6.655)
Scenario
VI: Structured Normal
0.995 (1: 0.017)
2.049 (1: 1.462)
16.723 (1: 19.093)
0.9 0.016)
2.301 0.549)
34,217 5.461)
VI: IAF
0.994 (4: 0.018)
1.675 (+: 1.049)
14.311 (4: 9.266)
0.993 ( 0.017)
2.148 (4 0.528)
34,336 5.398)
ICL (ours)
1.000 (#: 0.000)
0.582 (= 0,280)
8.708 (#: 4.945)
1.000 (1: 0.000)
1.869 (+: 0.342)
33.230 (2: 8.095)
Laplace Approximation
1.000 (#: 0.000)
6,260 (# 1.427)
13.497 (#: 29.702)
1,000 (=: 0.000)
5.924 (+: 1.145)
12,400 (+: 4.313)
VI: DiagonalNormal
1.000 (1: 0.002)
3.958 (= 1.641)
12.068 (1: 21.301)
1.000 (= 0.000)
3,879 (1: 1.061)
11,080 ( 3,341)
VI: MultivariateNormll
1.000 0.002)
3,875 (+ 1.691)
12. (+: 22.198)
1. 1.000 (+ 0.000)
3.896 (+: 1.057)
11.112 (t: 3.321)
Scenario 4
VI: Structured Normal
1.000 (+: 0.001)
3.661 (# 1.717)
12.195 (#: 22.874)
0.996 (=: 0.016)
3,822 (# 1.302)
11.368 (# 4,266)
VI: IAF
(+ 0.002)
3.536 (+ 1.597)
12.015 (+ 20.884)
1,000 (+ 0.000)
3,471 1.036)
3.233)
ICL (ours)
1.000 (+: 0.000)
2.451 (+ 0.868)
8,333 4,202)
1.000 0.000)
2.518 (+ 0.694)
11,938 2.956)
27

---

Can Transformers Learn Full Bayesian Inference In Context?
J. Evaluating Predictive Performance
In this section, we discuss the results our ICL approach terms of predictive performance. In scenarios one to four, TabPEN gives the best overall performance, which expected since is not limited to the GLM structure. Besides that, the MAP approach obtains consistently the best results, while our ICL method performs on par (scenarios 1,2,3) or better than (scenario 4) compared to the fully Bayesian methods the real-world data. On the synthetic data there is significant difference to the other fully Bayesian methods, except for the real-world data in scenario where HMC is clearly the best method. In scenario (gamma prior on the regression coefficients), the in-context learner performs significantly worse than all other methods, while this difference is less pronounced in scenario 7. In scenario 6, TabPFN also has substantially better performance than all other methods. The MAP approach performs on average better than all fully Bayesian methods, which themselves do not differ significantly.
Table 15: Evaluating the predictive performance across 50 synthetic and 17 real-world datasets for scenarios 1-4 in terms of Root Mean Squared Error (RMSE). The best result among all fully Bayesian methods is marked bold. For the fully Bayesian approaches, we use the posterior mean as point estimate for the response. MAP denotes the predictive performance of the model with the maximum posteriori estimate for the latents.
Scenario
Model
RMSE Real-World (L)
RMSE Synthetic (1)
HMC
0,591 (#: 0,023)
0.510 (#: 0.040)
Laplace Approximation
0.594 (+ 0.023)
0.510 0.040)
VI: DiagonalNormal
0.591 (#: 0.023)
0,509 (#: 0.040)
VI: MulrivariateNrrmal
0.591 (+ 0.023)
0.510 (+: 0.040)
Scenario
VI: Structured Normal
0.629 (#: 0.017)
0.555 (#: 0.039)
VI: IAF
0.593 (+ 0.023)
0.510 (+- 0.040)
ICL (ours)
0.593 (=: 0.020)
0.524 (#: 0.038)
MAP
0.555 (=: 0.024)
0.491 (#: 0.038)
TabPFN
0.483 (+ 0.036)
0.453 (+: 0.036)
HMC
0.559 (#: 0.023)
0.556 (#: 0.049)
Laplace Approximation
0.561 (#: 0.022)
0.557 (#: 0.049)
VI: DiagomalNormal
0.560 (=: 0,023)
0.557 (#: 0.049)
VI: NulrivariateNormal
0.559 (#: 0.023)
0.556 (+: 0.049)
Scenario 2
VI: Structured Normal
0.604 (#: 0.016)
0.685 (1: 0.054)
VI: IAF
0.563 (+ 0.023)
0.557 (+: 0.049)
ICL (ours)
0.561 (#: 0.019)
0.653 (#: 0.049)
MAP
0.513 (#: 0.023)
0.522 (1: 0.048)
TabPFN
0.449 (:: 0.034)
0.498 (t: 0.047)
HмC
0.684 (:: 0.027)
0.512 (:: 0.040)
Laplace Approximation
0.688 (::: 0.026)
0,516 (+: 0.040)
VI: DiagonalNormal
0.686 (= 0.027)
0.513 (:: 0.040)
VI: MultivariateNormal
0.685 (#: 0.027)
0.512 (#: 0.040)
Scenario 3
VI: Structured Normal
0.733 (=: 0.016)
0.607 (#: 0.043)
VI: IAF
0.686 (=: 0.027)
0.512 (:: 0.040)
ICL (ours)
0.690 (::: 0.023)
0,588 (1: 0.045)
MAP
0.646 (#: 0.028)
0.495 (:: 0.039)
TabPFN
0.556 (=: 0.041)
0,462 (t: 0.037)
HМC
0.642 (:: 0.027)
0.559 (+: 0.051)
Laplace Approximation
0.737 (=: 0.048)
2.457 (#: 0.493)
VI: DiagonalNormal
0.751 (#: 0.038)
2.046 (:: 0.399)
VI: MultivariateNormal
0.690 (::: 0.037)
2.155 (+: 0.454)
Scenario 4
VI: Structured Normal
0.686 (=: 0.015)
3.019 (:: 0.545)
VI: IAF
0.643 (#: 0.027)
1.751 (+: 0.422)
ICL (ours)
0.649 (=: 0.023)
1,464 (+: 0.151)
MAP
0.626 (:: 0.038)
2,377 (#: 0.529)
TabPFN
0.522 (+ 0.037)
0.496 (+: 0.047)
28

---

Can Transformers Learn Full Bayesian Inference In Context?
Table 16: Evaluating the predictive performance across 50 synthetic and 17 real-world datasets for scenarios 5 and 7 in terms of Root Mean Squared Error (RMSE). The best result among all fully Bayesian methods marked in bold. For the fully Bayesian approaches, we use the posterior mean as point estimate for the response. MAP denotes the predictive performance of the model with the maximum posteriori estimate for the latents.
Scenario
Model
RMSE Real-World (L)
RMSE Synthetic (1)
HMC
0.699 (#: 0,022)
0,490 (#: 0.036)
Laplace Approximation
0.699 (+ 0.022)
0.491 0.036)
VI: DiagonalNormal
0.702 (#: 0.022)
0.491 (#: 0.036)
VI: MultivariateNormal
0.698 (+ 0.021)
0.491 (+ 0.036)
Scenario
VI: Structured Normal
1.507 (#: 0.089)
0.741 (#: 0.053)
VI: IAF
0.699 (+ 0.022)
0.490 (+ 0.036)
ICL (ours)
0.769 (=: 0.020)
0.701 (#: 0.049)
MAP
0.658 (#: 0,022)
0.471 (#: 0.035)
TabPFN
0.534 (+ 0.040)
0.442 0.035)
HMC
0.953 (=: 0.015)
0.719 (#: 0.041)
Laplace Approximation
0.950 (#: 0.016)
0.719 (#: 0.041)
VI: DiagonalNormal
0.954 (= 0.015)
0.718 (#: 0.041)
VI: MultivariateNormal
0.953 (+ 0.015)
0.718 (+: 0.041)
Scenario 7
VI: Structured Normal
1.082 (:: 0.026)
1.028 (#: 0.118)
VI: IAF
0.954 (+ 0.014)
0.720 (+: 0.041)
ICL (ours)
1.019 ( 0.017)
0.765 (#: 0.041)
MAP
0.945 (# 0.017)
0.686 (1: 0.048)
TabPFN
0.817 (#: 0.040)
0.654 (+: 0.039)
Table 17: Evaluating the predictive performance across 50 synthetic and 17 real-world datasets for scenarios. 5 and in terms of accuracy (Acc.). The best result among all fully Bayesian methods is marked in bold. For the fully Bayesian approaches, we use the posterior mean as point estimate for the response. MAP denotes the predictive performance of the model with the maximum posteriori estimate for the latents.
Scenario
Model
Acc. Real-World (f)
Acc. Synthetic (f)
HMC
0.694 (+: 0.028)
0.546 (+: 0.015)
Laplace Approximation
0.692 (=: 0.027)
0.547 (1: 0.015)
VI: DiagonalNormal
0.700 (+ 0.028)
0.546 (#: 0.015)
VI: MultivariateNrral
0.691 (:: 0.029)
0,546 (:: 0.015)
Scenario 6
VI: Structured Normal
0.686 (= 0.028)
0.546 (#: 0.015)
VI: IAF
0.689 (:: 0.029)
0.545 (:: 0.015)
ICL (ours)
0.688 (= 0.027)
0,545 (#: 0.015)
MAP
0.723 (+ 0.025)
0.610 (+ 0.016)
TabPEN
0.862 (=: 0.021)
0.673 (#: 0.011)
29

---

Can Transformers Learn Full Bayesian Inference In Context?
K. Ablation: Using Gaussian Approximation
In this section, we present results on using a Gaussian approximation instead of Flow Matching to parameterize the approximation of the posterior.
The key takeaway from these results is that the in-context learning approach performs substantially better with Flow Matching (Lipman et al., 2022) than when Gaussian approximation of the posterior is employed.
Table 18: Generalized Linear Models: Comparing the in-context learner with a Gaussian approximation, fitted via the forward KL--divergence, the proposed flow matching method. Evaluation on 50 synthetic and 17 real-world datasets for seven different scenarios. If one method is by more than two standard errors better than the other, it is marked in bold. Overall, the ICI. + Flow Matching method clearly outperforms the Gaussian approximation, fitted via the forward KL-divergence,; it yields significantly better results (according to the two-standard-erry criterion) in 6 out of 7 scenarios on synthetic datasets and in all 7 scenarios on real-world datasets, across at least two of the three considered metrics (C2ST, MMD, or W2). In addition, the flow matching method consistently achieves lower or comparable standard errors, indicating more stable and reliable performance across datasets.
Synthetic Evaluation
Real-World Evaluation
Scenario
Model
C2ST (Q)
MMD (3)
(2)
C2ST (L)
MMD (U)
W1/ (L)
ICL+ + Gaussiun
0.845 (#: 0.213)
1.601 (+: 1.213)
2.024 0.874)
0.980 (#: 0.007)
1.715 (# 0.295)
1.976 0.238)
Scenario
ICI. + Flow Matching
0.765 (+ 0.123)
0.767 (+ 0,727)
0.585 (+ 0,301)
0.614 (+ 0.074)
0.175 [+ 0.219)
0.310 (# 0.138)
ICL+ Gaussiun
0.941 (#: 0.056)
1,000 (+: 0,953)
1.943 (# 0.657)
0.969 (#: 0.013)
1.490 (# 0.310)
2,068 (# 0.259)
Scenario
ICL. + Flow Matching
0,839 (#: 0.072)
0.707 (+: 0.658)
1.111 (#: 0,300)
0.768 (#: 0.033)
0.143 (+ 0.089)
0.411 (# 0.094)
ICL- + Gaussiun
0.907 (+: 0. 0.138)
1.779 (+: 1.363)
4.713 (# 1.560)
0.985 (+: 0.006)
1.526 (+ 0,198)
4.144 (#: 0,438)
Scenario
ICL + Flow Matching
0.611 (#: 0.070)
0.089 (#: 0.114)
0.423 (#: 0.348)
0.576 (+: 0.027)
0.037 (=: 0.026)
0.257 (= 0.044)
ICI- + Gaussian
0.989 (+ 0.011)
3.544 (+ 0,343)
23.035 (+ 6.549)
0.990 (+ 0.003)
3.858 [+ 0.061)
13,601 (+ 0.427)
Scenario
ICL + Flow Matching
0.753 (t: 0.049)
0.171 (#: 0.153)
0.631 (+: 0.294)
0.762 (+: 0.015)
0.105 ( 0.046)
0.597 (= 0.104)
ICI- + Gaussian
0.962 (+ 0.037)
1.444 (+ 1.640)
3.299 (+ 1.614)
0..911 (+ 0.005)
1.666 [+ 0.387)
2.963 (+ 0.239)
Scenario 5
ICL + Flow Matching
0.621 (+: 0.063)
0.067 (+:: 0.080)
0.299 (#: 0.195)
0.610 (+: 0.045)
0.046 (+: 0.020)
0.242 (# 0.038)
ICL- + Gaussian
0.909 (+: 0.048)
1.020 (+t 0,505)
1.515 (#: 0.358)
0.939 (:: 0.047)
1.799
(+ 0.751)
1.904 (# 0.541)
Scenario
ICI. + Flow Matching
0.532 (+- 0.019)
0.016 (+ 0.008)
0.590 (+ 0.066)
0.556 (++ 0.017)
0.035 [+ 0.015)
0.504 (+ 0.038)
ICL- + Gaussian
0.970 (:: 0.030)
2.169 (+t: 1.473)
1,707 (:: 0.480)
0.993 (:: 0.006)
2,390
(:: 0.414)
1.362 (# 0.152)
Scenario 7
ICI. + Flow Matching
0,700 (+ 0.116)
0.317 (+ 0.355)
0.400 (+ 0.286)
0.773 (+ 0.048)
0.294
(+ 0.457)
0.559 (+ 0.256)
30

---

Can Transformers Learn Full Bayesian Inference In Context?
Table 19: Factor Analysis: Comparing the in-context learner with Gaussian approximation, fitted via the forward KL- divergence, to the proposed flow matching method. Evaluation on 50 synthetic and 17 real-world datasets for seven different scenarios. If one method is by more than two standard errors better than the other, it is marked in bold. The flow matching approach shows favorable performance in the majority of cases. Specifically, it achieves statistically significant improvements in all 6 scenarios on synthetic data and in 5 out of 6 scenarios on real-world data. Notably, often reduces discrepancy measures such as MMD and Wasserstein-2 distance by a large margin. In addition, the variability of the flow matching estimates is generally lower, leading to more reliable and consistent results across different datasets. In scenario 4, the Gaussian in-context learner learned singular covariance matrix.
Synthetic Evaluation
Real-World Evaluation
Sccnario
Model
C2ST (4)
MMD ())
Wy (
C2ST (J)
MMD ())
Wy ())
Scenario 1
ICI + Gausssian
0.974 (+ 0.028)
1.838 (+ 0.778)
1.450 (+ 0.607)
0.589 (+ 0.015)
0.080 [+ 0.010)
0.459 (+ 0.017)
ICL+ + Flow Matching
0.552 (#: 0.028)
0.034 (# 0.034)
0.289 (4: 0.083)
0.606 (#: 0.038)
0.068 (#: 0.069)
0.265 (# 0.078)
ICI + Gaussian
0.835 (+: 0.040)
0.813 (+ 0.276)
1.250 (+: 0,316)
0.889 (+: 0.027)
0.778 (+ 0.109)
1.074 (+ 0.073)
Scenario 2
ICL+ + Flow Matching
0.542 (#: 0.006)
0.017 (#: 0.006)
0.244 (# 0.033)
0.622 (#: 0.032)
0.098 (# 0.039)
0.287 (= 0.046)
ICL- + Gaussian
0.826 (:: 0.035)
0.826 (4: 0.226)
1.210 (#: 0,239)
0,942 (t: 0.008)
1.466 (+: 0.078)
1.317 (# 0.038)
Scenario 3
ICL+ + Flow Matching
0.537 (#: 0.023)
0.024 (#: 0.021)
0.259 0.088)
0.609 (#: 0.019)
0.124 (=: 0.037)
0.179 (= 0.018)
ICL+ + Gaussian
0,870 (+: 0.043)
0.706 (+: 0.218)
1.635 (#: 0.297)
0.999 (:: 0.001)
2.025 (#: 0.017)
2.013 (= 0.019)
Scenario 4
ICL + Flow Matching
0.684 (#: 0.060)
0.198 (#: 0.141)
0.918 (# 0,246)
0.988 (#: 0.003)
1.764 (#: 0.026)
1.248 (= 0.008)
ICL+ Gaussian
0,838 (+: 0.029)
0.831 (+: 0.219)
1.248 (#: 0.249)
0,944 (+: 0.009)
1.477 (+: 0.073)
1.316 (= 0.031)
Scenario 5
ICL + Flow Matching
0.535 (#: 0.016)
0.021 (#: 0.011)
0.279 (# 0.060)
0.886 (+: 0.017)
1.207 (+: 0.101)
1.002 (= 0.042)
ICL+ Gaussian
0.837 (+: 0.030)
0.831 (1: 0.219)
1.248 (#: 0,249)
0.944 (+: 0.008)
1.477 (1: 0.073)
1.316 (= 0.031)
Scenario 6
ICL + Flow Matching
0.543 (t: 0.021)
0.023 (+ 0.015)
0.345 (# 0.173)
0.666 (+: 0.020)
0.200 (+ 0.034)
0.224 (= 0.014)
Table 20: Gaussian Mixture Models: Comparing the in-context learner with a Gaussian approximation, fitted via the forward KL-divergence, the proposed flow matching method. Evaluation on 50 synthetic and 17 real-world datasets for seven different scenarios. If one method is by more than two standard errors better than the other, it is marked in bold. While the differences are less clear-cut than in the previous models, ICL + Flow Matching demonstrates favorable performance in several scenarios, particularly for the Wasserstein-2 distance and MMD. Notably, its advantage is most visible in lower-dimensional settings (Scenario I and 2), where it consistently improves upon the Gaussian approximation, fitted via the forward KL-divergence, across most metrics. However, as the dimensionality increases, the performance gap tends to narrow, and in some cases, the inherent variability of the datasets, especially for the Gaussian approximation, fitted via the forward KL-divergence, makes it difficult to conclusively determine clear winner. Nonetheless, the flow matching approach often achieves smaller standard errors and lower discrepancy measures, underlining its potential for more stable modeling.
Synthetic Evaluation
Real- Real-World World Evaluation
Scenario
Model
C2ST (L)
MMD ())
W2 (1)
C2ST (L)
MMD (L)
W2 (L)
ICL+ + Gaussian
0.926 (+: 0.029)
0.555 (# 0.452)
2.586 (+: 0,560)
0.957 (#: 0.034)
0.765 (+: 0,958)
3,717 (+: 1.709)
Scenario 1
ICL+ Flow Matching
0.760 (1: 0.092)
0.303 (+ 0.548)
2.095 1.692)
0.847 0.082)
0.486 (+: 0.623)
4.054 (2: 2.782)
ICL Gaussian
0.985 (4: 0.010)
0.761 (+ 0,227)
5.022 (t: 0,945)
0.999 (:: 0.001)
0.801 (4: 0.256)
7.525 (#: 1.513)
Scenario 2
ICI. Flow Matching
0.812 0.061)
0.159 (+ 0.154)
2.314 0,926)
0.937 0.041)
0.282 (+ 0.131)
3.947 1.055)
ICL+ Gaussian
0.998 (+: 0.002)
0.829 (#: 0.241)
11.536 (#: 2.365)
1.000 0.000)
1.500 (+t: 0.251)
26.242 4.171)
Scenario 3
ICL+ Flow Matching
1.000 (1: 0.000)
0.582 (+: 0,280)
8.708 (#: 4.945)
1.000 (:: 0.000)
1.869 (1: 0,342)
33.230 (1: 8.095)
ICL Gaussian
0.998 (1:: 0.001)
6.314 (1: 0.449)
13.404 (d: 0.609)
0.997 (:): 0.001)
2.770 (4:: 1,201)
22.596 (1: 5.717)
Scenario 4
ICI. + Flow Matching
1.000 (++ 0.000)
2.451
(+ 0,868)
8,333 (+ 4,202)
1,000 (+ 0.000)
2.518 (+ 0.694)
11.938 (+ 2.956)
31

---

Can Transformers Learn Full Bayesian Inference In Context?
L. Ablation: Using a Diffusion Objective
To validate choosing the flow matching objective with optimal transport (OT) paths resulting in the objective in equation Equation (7), we also conduct experiments using a ceffiusion-objccttvi with variance preserving paths introduced by Song al. (2020). We choose three selected GLM, FA and GMM scenarios with the same 50 synthetic and 17 real-world datasets for each scenario in the other benchmarks.
L.1. Diffusion with Flow-Matching
First, we use the diffusion objective learned via flow matching, as described in (Lipman et al., 2022), where we choose the same hyperparameters (Lipman al., 2022).
Table 21: GLMs: Comparison of the OT flow matching and the VP diffusion objective on 50 synthetic and 17 real-world datasets for three different scenarios. All results within two standard errors of the best average result for each scenario are marked in bold.
Synthetic Evaluation
Woal-World&& Evaluation
Scenario
Model
C2ST ())
MMD (1)
(1)
C2ST (L)
MMD (
W/ (4)
Diffusion paths + FM
0,961 (+: 0.040)
1.525 (+ 0.777)
3.354 (# 1.333)
0.961 (#: 0.016)
1.347 (+ 0.365)
2.025 (+: 0.270)
Sccnario 2
OT paths
0.839 (+ 0.072)
0.707 (+ 0.658)
1.111 (+ 0.300)
0.768 (+ 0.033)
(+ 0.089)
0.411 (+ 0.094)
Diffusion paths + FM OT paths
0,903 (1: 0.111)
1.000 (1: 0.564)
1.733 (1: 0.408)
0.936 (1: 0.013)
1.002 (1. 0.203)
1.422 (1: 0.103)
Socnario 3
0.611 (+ 0.070)
0.089 (+ 0.114)
0.423 (+ 0.348)
0.576 (+ 0.027)
0.037 (+ 0.026)
0.257 (+ 0.044)
Diffusion paths FM OT paths + FM
0.691 (+: 0.074)
0.211 (+: 0.143)
0.708 (+: 0.233)
0.681 (+: 0.038)
0.182 (+: 0.093)
0.554 (+: 0.090) 0.242 (#: 0.038)
Scenario 5
0.621 (#: 0,063)
0,067 (= 0,080)
0.299 (#: 0,195)
0.610 (#: 0,045)
0.046 (#: 0,020)
In summary, the empirical results demonstrate that using the OT paths consistently outperforms the VP diffusion objective across all scenarios for both GLMs and FAs. For GLMs, OT paths achieve significantly lower C2ST values in all scenarios. In Scenario 2, OT paths reduce C2ST from 0.961 to 0.839 on synthetic data and from 0.961 to 0.768 on real-world data. Similarly, in Scenario 3, OT paths achieve substantial improvements, with C2ST dropping from 0.903 to 0.611 on synthetic data and from 0.936 to 0.576 on real-worle data. This trend is complemented by consistent improvements in other metrics such as W2, where OT paths often achieve reductions by over 50%.
Table 22: FA: Comparison of the OT flow matching and the VP diffusion objective on 50 synthetic and 17 real-world datasets for three different scenarios. All results within two standard errors of the best average result for each scenario are marked in bold.
Synthetic Evaluation
Real-World World Evaluation
Scenario
Model
C2ST (J)
MMD (1)
(2)
C2ST (L)
MMD ())
(L)
Diffusion paths + FM OT paths+ + FM
0.622 (#: 0,043)
0.207 (+ 0.121)
0.692 (1: 0.192)
0.595 (#: 0.012)
0.089 [+ 0.011)
0.475 (1: 0.019)
Scenario
0.552 (1: 0.028)
0.034 (1 0.034)
0.289 (1: 0.083)
0.606 (::: 0.038)
0.068 (11 0.069)
0.265 (1: 0.078)
Diffusion paths + FM OT paths + FM
0.826 (+ n.036)
0.768 (+ 0.238)
1.219 (+ 0.276)
0.878 (+ 0.028)
0.793 (+ 0.154)
1.056 (+ 0.084)
Scenatio 2
0,542 (+: 0.006)
0.017 (+ 0.006)
0.244 (+: 0.033)
0.622 (+: 0.032)
0.098 (+: 0.039)
0.287 (+: 0.046)
Diffusion paths + FM OT paths + FM
0.751 (+: 0.048)
0.387 (+ 0.216)
0.834 (1: 0.163)
0.944 (1: 0.008)
1.514 (1: 0.056)
1.332 (1: 0.028) 0.179 (4: 0.018)
Scenario 3
0.537 (1: 0.023)
0.024 (:: 0.021)
0.259 (1: 0.088)
0.609 (:: 0.019)
0.124 (1: 0.037)
For FA, the performance gap in C2ST remains notable. In Scenario 1, OT paths achieve the best results on synthetic data, reducing C2ST from 0.622 to 0.552, while also delivering improvements in W/2 (0.289 compared to 0.692). On real-world datasets, OT paths maintain competitive results, matching or exceeding the performance of diffusion paths. The advantage even more pronounced in Scenario 2, where OT paths consistently lead across all metrics, with a particularly striking reduction in MMD on synthetic data (0.017 compared to 0.768) and strong results for C2ST on real-world data (0.622 vs. 0.878). Similarly, in Scenario 3, OT paths achieve the lowest C2ST values, with synthetic results improving from 0.751 to 0.537 and real-world results from 0.944 to 0.609.
In the case of Gaussian Mixture Models (GMMs), the empirical results indicate that the OT paths generally outperform the VP diffusion objective across most scenarios and metrics, though the differences are not always statistically significant in pair-wiss wise comparisons. For example, in Scenario 1, OT paths achieve notably better results for C2ST on both synthetic and real-world datasets, with reductions from 0.924 to 0.760 and from 0.958 to 0.847, respectively. Similarly, for W2, OT paths
32

---

Can Transformers Learn Full Bayesian Inference In Context?
Table 23: GMMs; Comparison of the OT flow matching and the VP diffusion objective on 50 synthetic and 17 real-world datasets for three different scenarios. All results within two standard errors of the best average result for each scenario are marked in bold.
Synthetic Evaluation
Real-World Evaluation
Scenario
Model
C2ST (J)
MMD (1)
(2)
C2ST (
MMD (
W/ (L)
Diffusion paths + FM OT paths + FM
0.924 (+: 0.024)
0.241 (+ 0,381)
2.195 (t: 1.431)
0.958 (#: 0.030)
0.890 (+ 0.912)
5.328 (1: 2,544)
Sccnario
0.760 (+: 0.092)
0.303 (+ 0.548)
2,095 (+: 1.692)
0.847 (+ 0.082)
0.486 (+ 0.623)
4,054 (+: 2.782)
Diffusion paths + FM OT paths + FM
0,942 (+: 0.020)
0.213 (+ 0.187)
2.748 (#: 0.659)
0.984 (#: 0.012)
0.411 (# 0,162)
5,397 (+: 1,458)
Scenario 2
0.812 (4: 0.061)
0.159 (#: 0.154)
2.314 (#: 0.926)
0.937 (#: 0.041)
0.282 (+: 0.131)
3,947 (d: 1.055)
Diffusion paths + FM OT paths FM
1.000 (+: 0.000)
0.582 (+ 0.280)
8.708 (:: 4.945)
1.000 (#: 0.000)
1.869 (+: 0.342)
33.230 (# 8.095)
Scenario 3
0.999 (+ 0.001)
0.267 (+ 0.154)
7.234 (+ 2,974)
1,000 (+ 0.000)
1.155 (+ 0.258)
26,956 (+ 3.114)
Dimension 2
Dimension 7
2.5
2.5
2.0
2.0
ICL with Diffusian via SM
Gold Standard (HMC)
1.5-
Vi: Diagonal
Value
Value
1.5
Vi: Full
VI: Structured
VI: IAF
0.5
0.5
0.0
0.0
2
4
-2
4
Density
Density
Figure& 8: Marginal distribution for GL.M scenario (left) and GMM scenario (right). The in-context learner trained with diffusion objective using VP paths.
exhibit better performance on real-world data (4.054 vs. 5.328). In Scenario 2. OT paths maintain a consistent advantage in metrics such as C2ST and W/. For instance, synthetic data shows C2ST improvement from 0.942 to 0.812, while real-world data improves from 0.984 to 0.937. The OT paths also achieve lower MMD on synthetic data (0.159 vs. 0.213), supporting their effectiveness in this scenario. For Scenario 3, OT paths achieve better results for W2 on both synthetic and real-world data, reducing it from 8.708 3,708 to 7.234 and from 33.230 to 26.956, respectively.
L.2. Diffusion with Score-Matching
Second, we compare the results of using OT paths with flow matching to the results obtained when using VP paths and score matching. We use the score matching objective introduced by Song & Ermon (2019) and maintain the VP hyperparameters from Lipman et al. (2022) that we previously used for the diffusion objective with flow matching.
We find that, across all three considered GLM scenarios, using OT paths and flow matching yields substantially better results than using Diffusion VP paths and score matching, where the score-matching objective sometimes yields results comparable to those obtained using Laplace approximation. We observe similar overall results for FA and GMMs, although the effect is less pronounced. Note that the inferiority of score matching compared to flow matching is consistent with findings by Lipman et al. (2022) and Dax et al. (2024), who also report that flow matching produces more stable and less noisy training trajectories.
The large quantity of noise in the diffusion objective might prevent the model from learning complex conditioning on datasets æ, which is arguably the main challenge for performing in-context learning for the posteriors of latent variable
33

---

Can Transformers Learn Full Bayesian Inference In Context?
models. We find visually that using the diffusion objective leads to form of collapse where the model only learns constant posterior distribution Q that has a relatively large variance and is centered around zero, while largely ignoring the conditioning on æ (Please refer to figure 8).
Table 24: GLMs: Comparison of the OT flow matching and the VP diffusion objective with score matching on 50 synthetic and 17 real-world datasets for three different scenarios. All results within two standard errors of the best average result for each scenario are marked in bold.
Synthetic Evaluation
Real-World World Evaluation
Scenario
Model
C2ST ())
MMD (1)
(1)
C2ST (L)
MMD ())
W/ (l)
Diffusion paths + SM OT paths + FM
0.996 (+ 0.01 I)
4,121 (+ 1.625)
8.761 (t 4,415)
0.998 (+ 0.002)
1,574 (+ 0.906)
8.483 (t: 1,580)
Scenario
0.839 (+: 0.072)
0.707 (+ 0.658)
1.111 (+: 0,300)
0.768 (#: 0.033)
0.143 (+: 0.089)
0,411 (+: 0.094)
Scenario 3
Diffusion paths + SM OT paths + FM
0.965 (+: 0.075)
2.466 (+ 1.224)
3,947 (+: 1.323)
0.994 (+: 0.002)
2.018 (+ 0.206)
3.301 (+: 0.260)
0.611 (1: 0.070)
0.089 (1: 0.114)
0.423 (1: 0,348)
0.576 (1: 0.027)
0.037 (1: 0.026)
0.257 (d: 0.044)
Diffusion paths + SM OT paths FM
0.998 (#: 0.002)
3.163 (#: 0.651)
8.684 (1: 1.135)
0.999 (#: 0.001)
3.004 (+: 0.056)
8.547 (d: 0.177) 0.242 (4: 0.038)
Scenario 5
0.621 (1: 0.063)
0.067 (+: 0.080)
0.299 (:: 0.195)
0.610 0.045)
0.046 (1: 0.020)
Table 25: FA: Comparison of the OT flow matching and the VP diffusion objective with score matching 50 synthetic and 17 real-world datasets for three different scenarios. All results within two standard errors of the best average result for each scenario are marked in bold.
Synthetie Evaluation
C2ST Real-World Evaluation
Scenario
Model
C2ST (J)
MMD (1)
W; (L)
MMD ())
W/2 (1)
Diffusion paths + SM OT paths + FM
0.880 (+: 0.024)
0.875 (= 0.134)
1.787 (#: 0.155)
0.906 (#: 0.007)
0.845 (+ 0.026)
1.723 (+: 0.029)
Scenario
0.552 (#: 0.028)
0.034 (+ 0.034)
0.289 (1: 0.083)
0.606 (#: 0.038)
0.068 (#: 0.069)
0.265 (1: 0.078)
Diffusion paths SM OT paths + FM
0.932 (+: 0.022)
1.459 (+ 0,128)
2.798 (#: 0,141)
0,980 (# 0.008)
1.772 (+ 0,065)
2.927 (+: 0.085)
Scenario
0.542 (+ 0.006)
0.017 (+ 0.006)
0.244 (+ 0.033)
0.622 (+ 0.032)
0.098 (+ 0.039)
0.287 (+ 0.046)
Diffusion paths + SM OT paths + FM
0.925 (1: 0.021)
1.747 (1: 0,382)
3.028 (1: 0.646)
0.989 (.1: 0.003)
2.101 (.1: 0.050)
2.882 (1: 0.050)
Scenario
0.537 (+ 0.023)
0.024 (+ 0.021)
0.259 (+ 0.088)
0.609 (+ 0.019)
0.124 (+ 0.037)
0.179 (4 0.018)
Table 26: GMMs:
Comparison of the OT flow matching and the VP diffusion objective with score matching on 50 synthetic and 17 real-world datasets for three different scenarios. All results within two standard errors of the best average result for each scenario are marked in bold.
Synthetle Evaluation
Real- World Evaluation
Scenario
Model
C2ST (1)
MMD (1)
W2 (1)
C2ST (L)
MMD (J)
W22 (1)
Diffusion paths + SM
1.000 (+: 0.001)
1.412 (+ 0.365)
7.038 (t: 0.655)
0.998 (+: 0.002)
1.574 (+: 0.906)
8.483 (+: 1,580)
Scenario 1
OT paths FM
0.760 (1: 0.092)
0.303 (1: 0.548)
2.095 (1: 1.692)
0.847 (1: 0.082)
0.486 (11 0.623)
4.054 (1: 2.782)
Diffusion paths SM
1.000 (1: 0.000)
1,275 (1: 0.240)
6.621 (1: 1.091)
1.000 (1: 0.000)
1.032 (1: 0.163)
7,931 (1: 0.748)
Scenario 2
OT paths+ + FM
0.812 (1: 0,061)
0.159 (1: 0.154)
2.314 (1 0,926)
0.937 (1: 0.041)
0.282 (1. 0.131)
3.947 (1: 1.055)
Diffusion paths SM
1,000 (+ 0.000)
1.337 (+ 0.476)
10.877 5.262)
1,000 (++ 0.000)
2.277 (+ 0.245)
24.269 (+ 3.841)
Scenario
OT paths + FM
0.999 (+: 0.001)
0.267 (+ 0,154)
7.234 (+: 2,974)
1,000 (+: 0.000)
1.155 (+: 0,258)
26.956 (t: 3,114)
34

---

Can Transformers Learn Full Bayesian Inference In Context?
M. Ablation: Using an MLP-hased Encoder
To further justify choosing a transformer encoder in our ICL approach, we conduct an ablation study comparing the performance of our original ICL method with the performance obtained when the transformer encoder is replaced by an MLP with batch normalization (Ioffe, 2015) and skip-connections. To ensure a fair comparison, we use an MLP encoder with a hidden dimension of 1250 to give the overall model approximately the same number of parameters in the transformer-basee approach. Concretely, our MLP-approach has 43.3 million parameters compared to 43.1 million parameters with the transformer encoder. We choose three selected GLM, FA and GMM scenarios with 50 synthetic and 17 real-world datasets for each scenario.
In summary, we find that the transformer encoder yields consistently better, results than the mlp encoder across all scenarios. While the difference especially pronounced for the GLM scenarios, the difference become smaller for FA and GMMs.
Table 27: GLMs: Comparison when using an MLPP-based encoder and a transformer encoder on 50 synthetic and 17 real-world datasets for three different scenarios.
Synthetic Evaluation
Real-World Evaluation
Scenario
Type of Encoder
C2ST ())
MMD (1)
W2 (L)
C2ST (L)
MMD (J)
W/ (t)
MLP
0,942 (1: 0.093)
1,783 (#: 1.048)
2,503 (1: 0.814)
0.968 (#: 0.012)
1.528 (1: 0,394)
2.271 (1: 0.315)
Scenario 2
Transformer
0.839 (1: 0.072)
0.707 (1: 0.658)
1.111 (1: 0.300)
0.768 (1: 0.033)
0.143 (1. 0.089)
0.411 (t: 0.094)
MLP
0.957 (+ 0.075)
2.236 (+ 1.218)
2.681 (+: 1.130)
0.972 (+: 0.012)
1.658 (+ 0.450)
2.076 (+ 0.427)
Scenario 3
Transformer
0.611 (+: 0.070)
0.089 (= 0.114)
0.423 (+ 0,348)
0.576 (+: 0.027)
0.037 (+ 0.026)
0.257 (+: 0.044)
MLP Transformer
0.845 (1: 0.115)
1.066 (#: 0.859)
1.166 (1: 0.996)
0.890 (#: 0.055)
1.223 (1 0.791)
1.102 (#: 0,383) 0.242 (4: 0.038)
Scenario 5
0.621 (4: 0.063)
0.067 (#: 0,080)
0.299 (1: 0.195)
0.610 (1: 0,045)
0.046 (1 0.020)
In Table 27, the transformer encoder consistently outperforms the MLP encoder across all metrics and scenarios. In Scenario 2, C2ST drops from 0.942 (MLP) to 0.839 (Transformer) on synthetic data and from 0.968 to 0.768 on real-world data. Similarly, W2 improves significantly, decreasing from 2.503 to 1.111 on synthetic data and from 2.271 to 0,411 on real-world data. In Scenario 3, transformers achieve substantial improvements, reducing C2ST from 0.957 (MLP) to 0.611 on synthetic data and from 0.972 to 0.576 on real-world data. W2 also sees notable reductions, dropping from 2.681 to 0.423 on synthetic data and from 2.076 0.257 on real-world data. Finally, in Scenario 5, transformers maintain their superiority, achieving reductions in C2ST from 0.845 (MLP) to 0.621 on synthetic data and from 0.890 0.610 on real-world data. Improvements in W2 are similarly remarkable, with reductions from 1.166 to 0.299 on synthetic data and from 1.102 to 0.242 on real-world data.
Table 28: FA: Comparison when using an MLP-based encoder and transformer encoder on 50 synthetic and 17 real-world datasets for three different scenarios.
Synthetic Evaluation
Worl- Evaluation
Scenario
Type of Encuder
C2ST (J)
MMD (1)
(2
C2ST (L)
MMD (J)
W/ (L)
MLP
0.579 (+ 0.015)
0.017 (+ 0.006)
0.364 (#: 0.029)
0.634 (#: 0.014)
0.013 (+: 0.004)
0.331 (+: 0.010)
Scenario
Transformer
0.552 (1: 0.028)
0.034 (1: 0.034)
0.289 (1: 0.083)
0.606 (1: 0.038)
0.068 (1: 0.069)
0.265 (4: 0.078)
ML.
0.562 (1: 0.038)
0.037 (1: 0.042)
0.308 (1: 0.097)
0.632 (4: 0.068)
0,182 (4: 0.407)
0.339 (1: 0.174)
Socnario 2
Transformer
0.542 (+ 0.006)
0.017 (+ 0.006)
0,244 (+ 0.033)
0.622 (+ 0.032)
0.098 (+ 0.039)
0.287 (+ 0.046)
ML. Transformer
0.539 (1: 0.025)
0.023 (1: 0.022)
0.278 (1: 061
0.680 (1: 0.019)
0.268 (1 0.044)
0.253 (1: 0.017)
Scenario
0.537 (#: 0.023)
0.024 (+ 0.021)
0.259 (1: 0.088)
0.609 (1: 0.019)
0.124 (+: 0,037)
0.179 (#: 0.018)
For the factor analysis cases (Table 28), the transformer encoder still has better average performances even though the differences are substantially less pronounced than for the GLMs, In Scenario I, transformers slightly outperform MLPs, reducing C2ST from 0.579 to 0.552 on synthetic data and from 0.634 to 0.606 on real-world data. W2 also sees moderate improvements, dropping from 0.364 0.289 on synthetic data and from 0.331 to 0.265 on real-world data. In Scenario 2, the advantage of the transformer encoder remains consistent, with C2ST decreasing from 0.562 (MLP) to 0.542 on synthetic data and from 0.632 to 0.622 on real-world data. W2 also improves slightly, dropping from 0.308 to 0.244 on synthetic data and from 0.339 to 0.287 on real-world data. Scenario 3 shows the smallest differences, where transformers marginally improve C2ST from 0.539 (MLP) to 0.537 on synthetic data and from 0.680 to 0.609 on real-world data. For W3, the reductions are minor but consistent, dropping from 0.278 to 0.259 on synthetic data and from 0.253 to 0.1 on real-world data.
35

---

Can Transformers Learn Full Bayesian Inference In Context?
Table 29: GMMs:
Comparison when using an MLP-based encoder and transformer encoder on 50 synthetic and 17 real-world datasets for three different scenarios.
Synthetic Evaluation
Real-World Evaluation
Sccnario
Type of Encoder
C2ST ())
MMD ())
(2)
C2ST (L)
MMD (
W/2 (l)
MLP
0.873 (# 0.045)
0.242 (# 0,363)
2.203 (#: 1.098)
0.917 (#: 0.067)
0.891 [ 1.150)
4.528 (#: 2.701)
Scenario
Transformer
0.760 (1: 0.092)
0.303 (#: 0,548)
2.095 (1: 1.692)
0.847 (#: 0.082)
0.486 (11 0.623)
4.054 (1: 2.782)
MLP
0.921 (#: 0.035)
0.291 (#: 0.205)
2,870 (#: 0.710)
0.992 (#: 0.005)
0,399 (1: 0.127)
5.505 (#: 1.144)
Scenario
Transformer
0.812 (+: 0.061)
0.159 (+: 0,154)
2.314 (+: 0.926)
0.937 (+: 0.041)
0.282 (+: 0.131)
3.947 (t: 1.055)
Scenario 3
MI.P
0,999 (+ 0.000)
0.438 (+ 0.181)
11,502 (+ 9.719)
1.000 (+ 0.000)
1.001 (+ 0.149)
26.282 (+ 3.731)
Transformer
0.999 (4: 0.001)
0.267 (+: 0.154)
7.234 (+: 2,974)
1,000 (4: 0.000)
1.155 (+: 0.258)
26.956 (+: 3.114)
For the Gaussian Mixture Models (GMMs), the results indicate more mixed performance where the transformer still performs slightly better (Table 29): In Scenario transformer encoders slightly outperform MLPs on synthetic data, with C2ST improving from 0.873 (MLP) to 0.760 and W/ decreasing slightly from 2.203 2.095. However, real-world data, MLPs perform marginally better in terms of MMD, reducing it from 0.486 to 0.242, while transformers show minor improvements in W2 from 4.528 to 4.054. In Scenario 2, transformers show a more noticeable advantage. On synthetic data, C2ST improves from 0.921 (MLP) to 0.812, and W2 decreases significantly from 2.870 to 2.314. On real-world data, transformers reduce C2ST from 0.992 to 0.937 and MMD from 0.399 to 0.282, along with considerable improvement in W/2 from 5.505 to 3.947. In Scenario the differences between the two encoders are relatively small but still favor the transformers on synthetic data, with W22 decreasing from 11.502 (MLP) to 7.234. For real-world data, the results are nearly identical for C2ST (1.000 for both) but show slight increase in W2 for the transformer from 26.282 26.956. Overall, for the GMMs, the transformer encoders demonstrate consistent improvements across scenarios for synthetic data, particularly in Scenarios and 2. However, for real-works data, the performance differences are less pronounced.
36

---

Can Transformers Learn Full Bayesian Inference In Context?
1.00
C2ST
MMD
W2
0.85
0.8
0.80
0.6
0.4
0.75
0.56 0.60 0.64 0.68 0,72 0.76 0.80 0.84
0.88
0.56 0.60 0.64 0.60 0.72 0.76 0.80 0.84 0.88
0.56 0.60 0.64 0.68 0.72 0.76 0.80 0.84 0.88
Difference in data distribution (C2ST) Difference data distribution (C2ST)
Difference data distribution (C2ST)
Figure 9: Out-of-distribution (OOD) performance of the ICL method in GLM Scenario 2. The z-axis shows the distribution shift between training and test distributions, quantified by C2ST. The y-axis displays the performance of the in-context learner, evaluated via C2ST, MMD, and W2 distances against HMC samples where higher values indicate worse performance. OOD data is generated by gradually increasing the variance of the prior the regression coefficients from an initial value of 1.0 to 22, 1.7, 2, 2.5, and 3.5.
N. Robustness to Out-of-distribution Data
To investigate how our ICL approach behaves under mismatches between the distribution of synthetic training data and the data used infer the posterior, we conduct an ablation study by changing aspects of the distribution of training and testing data.
In summary, the results in Tables 31, 33 and 35 show that ICL approach is, in most cases, capable of robustly generalizing beyond its specific pre-training distribution when various aspects of this distribution are changed. While the performance sometimes decreases when a mismatch between training and testing data occurs, the drops in performance are almost always modest and, in many cases, almost negligible.
N.1. GLM Scenarios
For scenario 2, we change the variance of the prior on the covariates from value of V(Br,) = 1 to V(Bi,) = 2 for scenario 2.B and V(B.,) 44 4 for scenario 2.C. In scenarios 2.D and 2.E we change the scale parameter of the prior on the
variance 2 of the noise- thereby changing its mean from E[o²] = 0.5 to a value of E[o?] = 0.7071 for 2.D and [[o = 1
for 2.E. The variance changed from Via²] 0.0833 to V[o²] 0.1667 and V[o?] ~ 0.333.
For scenarios 3.B and 3.C, the variance the coefficients is doubled from scenario scenario 3.B and from 3.B to 3.C again, analogously scenarios 2.B and 2.C.0
For scenario 5, the rate parameter of the gamma distribution is changed. This leads to decrease in the variance from v(B,,) = to V( 0.5 for scenario 5.B and V(Bk,) = 0.25 for scenario 5.C. Notably, we also change the mean in the distribution of the covariates from mean from E[G = to a value of B(Gr, zs 0.7071 for 2.D and E[G.,] | = 0.5 for 2.E.
Table 30 shows that our ICL approach only exhibits modest degradation in performance when the variance of the coefficients is doubled quadruple while the mean stays the same (Scenarios 2.B, 2.C and 3.B, 3.C). Increasing the variance of the noise term by factor of two only has small effect while multiplying it by four causes drop in C2ST by 9.3%. However, decreasing the variance of the gamma prior in scenario 5, combined with decreasing the mean, leads to notable drop in performance across all metrics.
N.2. FA Scenarios
To construct the mismatch between training and test distribution, we vary the variance of the factor loading W.j.k for scenarios 1, 2 and 3. Concretely, the variance is doubled and quadrupled.
For the FA cases (refer to Table 33), there a notable drop in performance in the first scenario when OOD data is used. Please note that even in the most misspecified scenario (1.C), the performance, as measured in C2ST still around
37

---

Can Transformers Learn Full Bayesian Inference In Context?
Table 30: Distribution of variables for the OOD analysis on GLM scenarios.
Scenario
Bi.
3,,0 of
Y/. Mu.jjBi... B0,i, 07)
Scenario 2
(o0, 1)
(0,9)
IG 2)
N(u B1, of
Scenario 2.B
(0,)
(0,
1G
Scenario 2.C
400,4)
NN
IG(5,2)
N(u,,
Scenario 2.D
N(0,
N(0,9)
(G(5,2
N(u,
Scenario 2.E
1)
N(0,9)
IG(5,4)
N(
Scenario 3
L0place(O, 1)
1G(5,2)
N(u S.
Scenario 3.B
Laplace(O, 22)
1G
Scenario 3.C
Laplace(O, 2)
IG
V(,,,,,..
Scenario 5
1)
1G
Scenario 5.B
Ga(1, 22)
(G
3..
Scenario 5.C
Ga(1,2)
(5,
Table 31: OOD Performance: Evaluation on 50 synthetic datasets for 8 different GLM scenarios. All results within two standard errors of the non-OOD result for each scenario are marked in bold.
Scenario
C2ST (4)
MMD (4)
W2 (4)
Scenario 2
0.839 (#: 0.072)
0.707 (#: 0.658)
1.111 (#: 0.300)
Scenario 2.B
0.809 (#: 0.055)
0.410 (#: 0.095)
2.250 (#: 0.916)
Scenario 2.C
0.857 (#: 0.105)
0.634 (+ 0.318)
3.067 (+: 1.759)
Scenario 2
0.839 (=: 0.072)
0.707 (+: 0.658)
1.111 (#: 0.300)
Scenario 2.D
0.840 (=: 0.109)
0.916
(#: 1.123)
4.007 (#: 3.261)
Scenario 2.E
0.932 (= 0.120)
1.556 (# 1.127)
4.850 (# 2.261)
Scenario
0.611 (=: 0.070)
0.089 (# 0.114)
0.423 0.348)
Scenario 3.B
0.667 (= 0.080)
0.210 (= 0.117)
1.172 (#: 0.258)
Scenario 3.C
0.720 (= 0.108)
0.362 (= 0.248)
1.891 (#: 0.678)
Scenario
0.621 (# 0.063)
0.067 (# 0.080)
0.299 (#: 0.195)
Scenario 5.B
0.831 (# 0.121)
0.479 (+ 0.200)
1.762 (#: 0.541)
Scenario 5.C
0.920 (=: 0.064)
0.753 (# 0.424)
3.159 (:: 1.254)
percent better than the best VI method in this scenario (Table 46). While the absolute difference between performance the training distribution and the test distribution is very small for scenarios 2 and 3, the difference is still not within two standard errors of the non-OOD performance because the standard error itself is quite small. The performance the OOD data is still better than all other VI methods (see Table 3).
N.3. GMM Scenarios
To generate several distinct OOD scenarios based on the generative processes of GMMs, we vary scenario 2 in various ways. Note that the structure of the distributions is the same for all GMM foccario--fouusing on this specific scenario thus makes sense when considering OOD generalization. First, scenario 2.B, we decrease the symmetric parameter of the Dirichlet prior on the assignments from to 0.5 causing larger discrepancy the number of points per cluster. scenario 2.C we make the opposite change.
In scenarios 2.D and 2.E we first double and then quadruple the variance of the prior on the per-component variances dim,, Finally, in scenarios 2.F and 2.G, the prior on the mean is made more dispersed compared to the training data.
On the GMM scenarios (Table 35), the sample quality obtained via ICL is surprisingly stable under various changes to the lata-generating process. It is relatively unsurprising that changing the Dirichlet prior, i.e., making the cluster more or less uniform in their number of samples, might lead to cases the ICL method can generalize to relatively easily, as demonstrated scenarios 2.B and 2.C. The most pronounced drop performance results from increasing the variance of the prior on the
38

---

Can Transformers Learn Full Bayesian Inference In Context?
Table 32: Distribution of variables for the OOD analysis on the FA scenarios.
Scenario
K
P
P4,
Vi.j
W..j.k
Zi.j
Zdim
Scenario
50
3
(0
[G(5, 1)
(0,
(00, 1)
3
Scenario 1.B
50
3
1)
IG)
(O 2)
W(0,1 1)
3
Scenario 1.C
50
3
1)
1G)
((0,4)
((0, 1)
3
Scenario 2
50
NO 0.1)
1G
Laplace(O, 10)
1)
3
Scenario 2.B
50
N/O,, 0.1)
(G
Laplace(O, 10 v2)
N)0,1)
Scenario 2.C
50
3
NV 0.1)
(G,
Laplace
N(0,1
3
Scenario
25
W(0,0.1
(G,
(O 3)
N(0,1)
Scenario
25
N 0.1)
(G
(0, 3 v2)
N(0,1)
3
Scenario
25
N 0.1)
(G(
AO 6)
N(0.
Table 33: OOD Performance: Evaluation on 50 synthetic datasets for 6 different FA scenarios. All results within two standard errors of the non-OOD result for each scenario are marked in bold.
Scenario
C2ST (4)
MMD (L)
W2 (L)
Scenario
0.552 (+ 0.028)
0.034 (+ 0.034)
0.289 (# 0.083)
Scenario I.B
0,826 (# 0.066)
0.656 (# 0.384)
0.929 (# 0,321)
Scenario 1.C
0.855 (#: 0.060)
0.837 (#: 0.494)
1.135 (#: 0.461)
Scenario 2
0.542 (+= 0.006)
0.017 (+: 0.006)
0.244 (# 0.033)
Scenario 2.B
0.580 (#: 0.069)
0.087 (=: 0.191)
0.393 (#: 0.291)
Scenario 2.C
0.589 (#: 0.076)
0.089 (+ 0.113)
0,446 (# 0.233)
Scenario
0.537 (= 0.023)
0.024 (= 0.021)
0.259 0.088)
Scenario 3.B
0.544 (+: 0.028)
0.030 (+ 0.021)
0.285 (+ 0.094)
Scenario 3.C 0.533 (= 0.025)
0.021 (= 0.015)
0.347 (# 0.152)
standard deviation of the components of the mixture model (scenario 2.E), while increasing the variance of the mean vector relative to the standard deviation of the components has less pronounced effect.
39

---

Can Transformers Learn Full Bayesian Inference In Context?
Table 34: Distribution for the OOD analysis of the GMM scenarios.
Scenario
K
M
L
of Lmm IG
pi.m.. j.m.
Scenario
25
Dir(1)
30,300 1,m,1)
Scenario 2.B
25
3
3
Dir(0.5)
IG 2)
20,30 3.m.1))
Scenario 2.C
25
3
Dir(2)
IG(5,2) 2)
30, 30....)))
Scenario 2.D
25
3
3
Dir(1)
IG 2 v2) IG(5,4)
(0, s.m.1)
Scenario 2.E
25
3
Dir(1)
(0,300
Scenario 2.F
25
3
Dir(1)
IG(5, 2)
N(0, 4a (.m.t)
Scenario 2.G
25
3
Dir(1)
1G
N(0, 50 _..mm.))
Table 35: OOD Performance: Evaluation on 50 synthetic datasets for different GMM scenarios. All results within two standard errors of the non-OOD result for each scenario are marked bold.
Scenario
C2ST (1)
MMD (1)
W2 (()
Scenario
0.812 0.061)
0.159
(+ 0.154)
2.314 (#: 0.926)
Scenario 2.B
0.829 (#: 0.050)
0.233
(4: 0.161)
2.595 (#: 0.998)
Scenario 2.C
0.816 (= 0.057)
0.149
(= 0.135)
2.272 (#: 0.654)
Scenario
0.812 (= 0.061)
0.159 (#: 0.154)
2.314 (#: 0,926)
Scenario 2.D
0.812 (=: 0.076)
0.148 (# 0.091)
2.557 (#: 0.837)
Scenario 2.E
0.880 (=: 0.057)
0.231 ( 0.109)
3.535 (#: 1.003)
Scenario
0.812 (=: 0.061)
0.159 ( 0.154)
2.314 (#: 0.926)
Scenario 2.F
0.821 (= 0.076)
0.216
( 0.214)
2.700 (# 1.044
Scenario 2.G
0.844 (+ 0.046)
0.197
(+ 0.124)
2.675 (#: 0.552)
40

---

Can Transformers Learn Full Bayesian Inference In Context?
O. Ablation: Dimensionality of the Problem
In this section, the effect of the dimensionality of the latent variable 2 E ER for the GLM scenarios investigated.
In summary, our results show that forward-KL based VI and in particular MAP solutions perform strongly in terms of predictive performance, which is in line with results first presented by Mittal et al. (2025b:a).
In table 36, we find that the advantages of the in-context learning approach to deteriorate for higher dimensionalities, with the variational inference methods using Gaussian approximation performing well for 20 dimensions. This finding is line with work by Mittal et al. (2025b:a). For 50 dimensions we find that in many cases the used metrics do not allow to significantly discriminate the performance the different approaches. Note that the randomness in the results, especially for higher dimensionalities, can in rare cases lead to better mean values. This is most likely not significant when taking the standard error into account.
Similar to scenarios 1,2 and 3, we find that in scenarios 4,5 and 6 (Table 37) the advantages of the in-context learning approach to deteriorate for higher dimensionalities, with the variational inference methods using Gaussian approximation performing well for 20 dimensions. For 50 dimensions we find that in many cases the used metrics do not allow to significantly discriminate the performance of the different approaches. Note that the randomness in the results, especially for higher dimensionalities, can in rare cases lead to better mean values. This is most likely not significant when taking the standard error into account.
Finally, the results in Table 38 show that also for this scenario 7, the advantages of the in-context learning approach deteriorate for higher dimensionalities. However, in this specific scenario the in-context learner is not significantly different from the VI methods in terms of C2ST and MMD for 20 dimensions. For 50 dimensions we find that the VI method using IAF performs well, together with the in-context learning approach in terms of MMD while the C2ST score does not indicate clear winner and W2 favors the other methods. Note that the randomness in the results, especially for higher dimensionalities, can in rare cases lead to better mean values. This most likely not significant when taking the standard error into account.

---

Can Transformers Learn Full Bayesian Inference In Context?
Table 36: Generalized Linear Models; Ablation with respect to the dimensionality of the problem on 50 synthetic and 17 real-world datasets for scenarios 1,2 and 3. All results within two standard errors of the best average result for each scenario are marked in bold. Due to the limitations of the number of features in the real-world data, we can only use 5 datasets for 20 and one dataset for 50 dimensions.
Synthetic Evaluation
Real-World Evaluation
Scenario
Dim.
Model
C2ST ())
MMD ())
W/ (1)
C2ST (L)
MMD (1)
Laplace Approximation
1.000 (+: 0.000)
2.738 (4: 0.721)
0.825 (4: 0.279)
1.000 (4: 0.000)
2.150 (+: 0.323)
0.642 0.124)
VE: DiagonaiNormal
0.904 (4: 0.076)
1445 (1: 0.984)
0.669 0.301)
0.797 (1 0.083)
0.612 [ 0.511)
0.414 0.152)
Socnario
VE: MultivariateNormal
0.750 (#: 0,128)
0.735 (+ 0.733)
0.565 (# 0.292)
0.607 (+ 0.070)
0.167 (+ 0.19)
0.301 (#: 0.123)
VI: Structured Normal
0.753 (:: 0.126)
0.736 (1: 0.737)
0.570 (2: 0.310)
0.600 (1: 0.070)
0.169 (# 0.214)
0.306 (#: 0.131)
VI: IAF
0.777 (#: 0.122)
0.864 (+: 0.844)
0.725 0.523)
0.683 (+ 0.132)
0.440 (+ 0.559)
0.503 0.383)
HMC
0.745 (+ 0.130)
0.722 (+ 0.732)
0.569 (4 0.301)
0.595 (4: 0.075)
0.173 (+ 0.213)
0.321 (+ 0.140)
ICL (ours)
0.765 (:: 0.123)
0.767 (1: 0.727)
0.585 (2: 0.301)
0.614 (1 0.074)
0.175 (4: 0.219)
0.310 (1: 0.138)
Laplace Approximation
1.000 (#: 0.000)
2.237 (= 0.024)
3.252 1.172)
1.000 (1: 0.000)
2.206 (#: 0.021)
2.792 (:: 0.339)
VI: DiagonalNormal
0,843 (+ 0.204)
1.056 (= 0.869)
2.976 (# 0.927)
0.983 (+ 0.019)
1,217 (+ 0,463)
2.406 (+: 0.348)
Socnario
20
VI: MultivariateNermal
0.789 (+: 0,140)
0.714 0.351)
2.774 (#: 0.995)
0.768 (+ 0.120)
0.180 (+ 0.159)
2.064 (+t 0.306)
VE Structured Normal
(:: 0.109)
0,708 0.153)
2.703 1.069)
0.668 0.080)
0.168 (:: 0.071)
2,052 0.275)
VI: IAF
0.832 (. 0.196)
0.847 (4: 1.016)
4.015 (4 0.415)
0.939 (1 0.024)
0.508 (1 0.200)
3,140 (-. 0.290)
ICL (ours)
0.849 (:: 0.171)
0.844 (== 0.905)
4.564 (#: 0.622)
0.970 (1 0.020)
0.724 (:: 0.287)
4,250 (:: 0.312)
Laplace Approximation
1,000 (#: 0.000)
2.401 (= 0.282)
5.152 2.268)
1.000 (:: nan)
2.339 (#: nan)
6.642 (:: nan)
VI: DiagonalNormal
0.812 (+ 0.197)
0.956 (= 1.016)
5.541 (+ 2.356)
0.915 (+ nan)
0.508 (+ nan)
6,200 (+: nan)
Scenario I
50
VI: MullivarriatNeemal
0.839 (.i 0,148)
0.926 (:: 0.682)
5.514 (1) 2.370)
0.905 (1: nun)
0.790 (.) nan)
6,258 (.1 nan)
VE: Structured Normal
0.823 (# 0.160))
0,480)
5.752 2.098)
0.910 (1: nan)
1.122 (#: nan)
nan)
VI: IAF
0.820 (# 0,182)
0,814 (= 0.987)
6.696 (+ 1.207)
0.938 (+ nan)
0,256 (+t nan)
6,369 (+ nan)
ICL (ours)
0.787 (= 0.217)
1015 (= 1.255)
8,278 (#: 0.821)
0.979 (:: nan)
0.413 (#: nan)
8,368 (:: nan)
Laplace Approximation
1,000 (1 0.000)
4.853 (.L. 2.333)
5.770 5.946)
1.000 (1 0.000)
2.572 (1. 0.206)
0,809 0.149)
VI: DiagonslNormal
0,957 (# 0.091)
3,906 (+: 2.679)
5.628 ( 6.092)
0.892 (:: 0.044)
0.847 [+ 0.389)
0,530 (:: 0.175)
VI: MullivariateNormal
0.910 (.L 0,131)
3.407 (11 2.781)
5. 5.584 (1) 6.104)
0,820 (1 0.031)
0.243 [1 0.148)
0.408 (1 0.118)
Secnario 2
VL Structured Normal
0.908 (# 0.119)
3.139 (#: 2.763)
5,480 6.164)
0,824 0.023)
0.215 (# 0.110)
0.392 0.109)
VI: IAF
0.968 (+ 0.063)
4,416 (+ 2.473)
7.474 (+ 6,235)
0,888 (+: 0.067)
0,921 (+ 0.860)
0,942 (+: 0,733)
ICI. (ours)
0.839 (.. 0.072)
0.707 (1. 0.658)
1.111 (1: 0.300)
0.768 (1 0.033)
0.143 (4. 0.089)
0.411 (1: 0.094)
Luplace Approximation
1.000 (1 0.000)
2.314 (L. 0.237)
3.069 1.168)
1.000 (1 0.000)
2.222 (1 0.018)
2.847 (1 0.305)
VI: DiagonaiNormal
0,904 (#: 0,168)
1,292 (= 0.937)
2.863
0.919)
0.990 (:: 0.009)
1,277 (#: 0.452)
2.483 (#: 0.318)
Secnario 2 20
VI: MultivariateNormal
0.851 (+ 0.134)
0.492 (= 0.547)
2.694 (#: 0.916)
0.843 (+ 0.069)
0.243 (+t 0,170)
2.166 (+: 0,266)
VI: Structured Normal
(. 0.065)
0.099)
2.497 0.993)
0.685 (1 0.031)
0.029 (1 0.025)
2.191 0.271)
VI: IAF
0.916 (+ 0.1 10)
1.062 (= 1.076)
4.191 (+ 0.623)
0.952 (+: 0.025)
0,515 (+ 0,242)
3,331 (+ 0.371)
ICL. (ours)
0.955 (.L 0.057)
1.131 (L 1.035)
4.945 (1L 0.836)
0.968 (1 0.020)
0.724 (.1 0,278)
4.356 (.L. 0,302)
Laplace Approximation
1,000 (+ 0.000)
2.477 (= 0.271)
5.728 (+: 1.358)
1.000 (+: nan)
2.350 (+ nan)
5.620 (+: nan)
VI DiagonalNormal
0,853 (# 0.182)
0,787 (= 0.687)
6.224 (# 1.225)
0.996 (#: nan)
1,080 (#: nan)
5,426 (#: nan)
VI: MultivariateNorral
0.878 (+ 0,150)
0.688 (- 0.620)
6.206 (+: 1.244)
0.994 (+ nan)
0.791 (+ nan)
5,305 (+ nan)
Scenario 2
50
VI: Structured Normal
0.865 (L 0.081)
0.186 0.169)
5.874 1.233)
0,819 (11 nan)
0.093 (1 nan)
nam)
VE IAF
0.909 (#: 0.130)
0.649 (= 0.650)
7,465 (#: 0,335)
0.985 (:: nan)
0,426 (#: nan)
6,426 (#: nan)
ICL tours)
0.972 (+ 0.039)
0.741 (= 0.713)
8.313 (#: 0.608)
0.971 (+ nan)
0.405 (+t nan)
7,718 (+: nan)
Laplace Approximation
1.000 (+ 0.000)
2,203 (- 0.997)
1.170 0.949)
1.000 (+ 0.000)
1.841 (+ 0.185)
0.729 (+ 0.175)
VI: DiugonalNormal
0,866 (1L 0,101)
1.069 1.150)
0.846 (1) 0.747)
0.797 (11 0.083)
0.526 (.L 0.361)
0,480 (1 0.207)
VI: MultivariateNnrmal
0.656 (++ 0,131)
0.445 (- 1.061)
0.600 (+ 0.737)
0.560 (+ 0.035)
0.032 (+ 0.028)
0.249 (+ 0.069)
Scenario
VI: Structured Normal
(1 0.125)
0.421 0.993)
0.659 0.736)
0.552 (1 0.028)
0.027 (1 0.015)
0.055)
VI: IAF
0,751 (#: 0,148)
0.939 (= 1.349)
0.964 (#: 0.924)
0.673 (:: 0,141)
0.399 (#: 0.543)
0,563 (#: 0.433)
ICL tours)
0.611 (+ 0.070)
0.089 (= 0.1
0.423 (+: 0.348)
0.576 (+ 0.027)
0.037 (+ 0.026)
0.257 (+ 0.044)
Laplace Approximation
1.000 0.000)
2.726 (- 1.116)
4.127 (+ 1.927)
1.000 (+ 0.000)
2.234 (+ 0.092)
3,589 (+ 0.519)
VE: DiugonalNormal
0,912 (:1 0.134)
1.704 (2: 1.467)
3,933 (1) 1.574)
0.983 (11 0.014)
1.298 (1 0.443)
3,147 (1 0.557)
VE MultivariatcNormal
0.863 (=: 0.113)
0.937 (= 1.174)
3,754 (#: 1.650)
0.796 (:: 0.099)
0.268 (#: 0.226)
2.645 (#: 0,466)
Scenario 3
20
VI: Structured Normal
0.768 (+ 0.109)
0.302 (- 0.518)
3.151 1.663)
0.722 (+: 0.073)
0.131 (+ 0.141)
2.579 (+ 0.399)
VE: IAF
0,908 (# 0.133)
1.657 (= 1.476)
5.543 (# 1.120)
0.936 (#: 0.041)
0,548 (#: 0.341)
3.678 (#: 0.670)
ICI. (nurs)
0.902 (+ 0.076)
LO53 (- 0.782)
6.206 (+ 0.783)
0.932 (+ 0.019)
0,635 (+ 0.183)
5.281 (+ 0.317)
Laplace Approximation
1.000 (# 0.000)
2,700 (= 0.789)
8.841 (# 1.691)
1.000 (:: nan)
2.348 (#: nan)
7.049 (#: nan)
VE: DiagonslNormal
0,870 (:1 0.127)
1.154 (= 1.321)
9.180 (1: 1.513)
0.997 (11 nan)
1.393 (:2 nan)
6.791 (::: nan)
VE MultivariateNormal
0.896 (=: 0.101)
1.27 (= 1.157)
9.175 (# 1.555)
0.998 (+t nan)
1.092 (#t nan)
6.667 (+: nan)
Scenario
50
VI: Structured Normal
(+ 0.1
0.539 (- 0.667)
9.118 1.538)
0.958 (+ nan)
0.420 (+ nan)
6.665 (+ nan)
VE: IAF
0,869 (.1 0.124)
0.751 0.939)
9.917 (1L 0.870)
0.971 (11 nan)
0.417 (1 nan)
7.411 (1 nan)
ICL (ours)
0.931 (#: 0.062)
0.784 (= 0.884)
10.063 (= 0.930)
0.965 (#: nan)
0,347 (#: nan)
8.482 (=: nan)
42

---

Can Transformers Learn Full Bayesian Inference In Context?
Table 37: Generalized Linear Models: Ablation with respect to the dimensionality of the problem on 50 synthetic and 17 real-world datasets for scenarios 4, 5 and 6. All results within two standard errors of the best average result for each scenario are marked in bold. Due to the limitations of the number of features in the real-world data, we can only use 5 datasets for 20 and one dataset for 50 dimensions.
Synthetic Evaluntion
Real-World Evaluntion
Sceuario
Dim.
Model
C2ST (1)
MMD (
W/
C2ST (1)
MMD ())
W2 (L)
Laplace Approxination
1,000 (# 0.000)
3.511 (+ 2.025)
2.166 1.722)
1.000 (#: 0.000)
2.011 (+ 0.058)
0.993 (# 0.144)
VI: DiagonaNNormal
0.968 (+ 0.036)
2.798 (+ 2.255)
2,065 1.745)
0.916 (+ 0.040)
0,928 (+ 0.339)
0.732 0.181)
VI: Mulivariiate
0.855 0.123)
16448 (1: 2.052)
1,853 (:: 1.745)
0.771 (4: 0.017)
0.087 (1: 0.030)
0,539 (1: 0.070)
Scenario
VI: Stractured Normal
0.847 (1: 0.116)
1.505 (4 1.978)
1,889 (4: 1.883)
0.769 0.012)
0.083 0.018)
0.543 (1: 0.070)
VI: IAF
0.942 (#: 0,077)
3.029 (+: 2.210)
3,554 (c: 2.715)
0.833 (2: 0.069)
0.636 (+: 0.756)
0.978 (#: 0.600)
ICL (ours)
0,753 (#= 0.049)
0.171 (+ 0.153)
0.631 (+ 0.294)
0.762 (#: 0.015)
0.105 (+ 0.046)
0.597 (# 0.104)
Laplace Approximation
1.00 (2: 0.900)
4.929 (1: 1.611)
8.863 (:: 3,796)
1.000 (2: 0.000)
3,196 (1: 0.841)
5.186 (1: 1.533)
VI DiagonalNormal
0,988 (= 0.060)
4,418 (+ 2.013)
9.364 4.281)
0.997 0.007)
3.095 1.417)
6.098 (# 2.435)
VI: MulivarritteNrmaa
0.986 (2 0.054)
3.388 (+ 1.907)
7,910 (:: 4.070)
0.893 (#: 0.087)
0.534 (+: 0.446)
3.175 (1 0.751)
Socnario
20
VI Stractured Normal
0.954 (= 0.076)
2.254 (+ 1.515)
7.475 (+: 4.224)
0.727 (#: 0.034)
0.074 (+: 0.070)
2.877 (#: 0.379)
VI: IAF
0,987 (+ 0.059)
3,258 (+ 1.415)
9.865 3,515)
0.955 0.030)
0.629 0-308)
4.098 (+ 0,341)
ICL (ours)
0.978 (1: 0.038)
1,885 (1 0.720)
11.335 (:L 1.378)
0.972 (1: 0.018)
0.668 (1: 0.199)
9.937 (11 0.466)
I Laplace Approxination
1,000 (+ 0.000)
6.695 (+ 1.329)
12.323 (+ 4.091)
1.000 (+ nan)
5,491 (+ nan)
7,518 (+ nan)
VI. DingonalNormal
0.965 (= 0.084)
2.395 (# 1.958)
12.022 3.673)
0.996 nan)
4.368 man)
6.951 (+ nan)
VI: Multivariate Normaal
0.984 (+ 0.054)
5.395 (+ 1.847)
12.141 (+ 3.079)
1.000 (+ nan)
5.146 (+ nan)
9.002 (+ nan)
Scenario
50
VI: Stractured Normal
0.982 (1: 0.026)
4,261 (:: 1.191)
11.126 (:L 3,396)
0.869 (1: man)
3,181 (:1 nam)
7.065 (1 nun)
VI IAF
0,981 0.048)
4.609 (#: 1.412)
12.567 3.131)
0.988 nan)
3,558 man)
7,849 (# nan)
ICL (ours)
0.960 (+ 0.045)
3.792 (+ 0.758)
14.071 (+ 0.894)
0.974 (+ nan)
3.443 (+ nan)
12,546 (+ nan)
I applace Approxination
I. 1,000 (+ 0.000)
2,060 (+ 0.472)
0.797 (+ 0.577)
1.000 (+: 0.000)
1.982 (+ 0.1.6))
0.623 (+ 0.084)
VI: DiagonalNermal
0.866 (1: 0.085)
(1: L.022)
0.651 0.549)
0.810 0.016)
0.441 0.252)
0.384 (1: 0.089)
VI MutivariateNormal
0.765 (# 0,100)
0,537 (# 1.019)
0.633 (+: 1.067)
0.711 (=: 0.038)
0.148 (+: 0.093)
0.279 (# 0.056)
Scenario
VI Stractured Normal
0.758 (+ 0.098)
0.447 (+ 0.818)
0.572 (+ 0.816)
0.705 (+ 0.032)
0.140 (+ 0.081)
0.269 (+ 0.045)
VI: IAF
0.814 (1 0.105)
(1 1.165)
0.881 1.067)
0.777 0.106)
0.684 0.939)
0.625 (1 0.525)
ICL (ours)
0,621 (# 0.063)
0,067 (# 0.080)
0,299 (#: 0.195)
0.610 (#: 0.045)
0,046 (#: 0.020)
0.242 (# 0.038)
Laplacc Approxination
1,000 (== 0.000)
2.367 (# 0.555)
2,780 (+: 1,271)
1.000 (=: 0.000)
2,200 (#: 0.041)
2.444 (=: 0.619)
VI: DiagomilNrmal
0.938 (+ 0.098)
1.153 (+ 0.954)
2.552 (+ 1.147)
0.967 0.012)
0.547 (+ 0.233)
1.973 (+ 0.452)
Scenario
20
VI: MuliivariateNormu
0,929 (1 0,882)
0,710 (.L 0.768)
2.473 (1 1.145)
0.928 (1: 0.016)
0.250 (1) 0.079)
1.776 (11 0.399)
VL Stuctured Normal
0,909 (= 0.082)
0.397 (2 0.442)
2.246 (#: 1,244)
0.924 (=: 0.018)
0.202 (#: 0.094)
1.775 (=: 0.430)
VI: IAF
0.934 (+ 0.092)
1.325 (+ 1.16I)
4.899 (+: 1.320)
0.980 (+: 0.016)
0.892 (+: 0.404)
3,593 (+ 0.597)
ICL (ours)
0,961 (L 0.046)
1,330 (1 1.125)
5,084 (1 1.297)
0.981 0.014)
1.162 (1) 0.461)
4,804 (1 0.578)
Laplace Approximation
1.000 (1: 0.000)
2.582 (11 0.606)
5.765 (1L 1.540)
1.000 (1: nan)
2.322 (1) nam)
3.485 (11 nan)
VI. DiagonalNormal
0.925 (= 0.074)
0.925 (# 1.056)
6,461 (#: 1.877)
0.972 (# nan)
0.186 (=: nan)
3,251 (#: nan)
Socnario 5
50
VI: Multivariate
0,934 (+ 0.064)
0.825 (+ 0,972)
6,404 (+: 1.882)
0.969 nan)
0.165 nan)
3.223 (+ nan)
VI: Structured Normal
0.927 (1. 0.068)
0.481 (.1 0.588)
6.420 (1. 1.970)
0.961 (1. nan)
0.072 (1) nam)
3.324 (.). nan)
VI: IAF
0,925 (= 0.069)
0.792 (# 0.975)
8,458 (:: 0,864)
0.996 (#: man)
0.519 (:: nan)
4.645 (:: nan)
ICL (ours)
0.998 (#: 0.002)
0.762 (= 0.987)
8.195 0.820)
1.000 (=: nan)
0.984 nan)
7,288 (+: non)
Laplace Approximation
1.000 (+ 0.000)
2.026 0.027)
1.612 (+: 0.162)
1.000 (+: 0.000)
1.993 (+ 0.032)
1.299 (+ 0,106)
VI: DiagonalNormal
0.724 (1. 0.060)
0.185 (1 0.082)
0.787 (11 0.078)
0.703 (1. 0.039)
0.147 (1. 0.063)
0.637 (1 0.089)
Scemario
VL Multivariateeormal
0,534 (= 0.018)
(# 0.006)
0.581 0.074)
0.538 0.019)
0.016 0.007)
0.466 0.029)
VI: Stractured Normal
0.536 (= 0.016)
0.014 (+ 0.005)
0,583 (+: 0.071)
0.536 (+: 0.019)
0.017 (+: 0.009)
0.469 (+ 0.033)
VI: IAF
0,542 (= 0.026)
0.031 (# 0.031)
0.613 (#: 0.092)
0.535 (#: 0.015)
0.015 (+: 0.006)
0.467 (# 0.031)
ICL (ours)
0.532 (+: 0.019)
0.016 (+ 0.008)
0.590 0.066)
0.556 0.017)
0.035 0.015)
0.504 (+ 0.038)
Laplace Approximation
1.000 (=: 0,000)
2.247 (2: 0.006)
4.158 (2: 0.243)
1.000 (2: 0.000)
2.240 (=: 0.007)
3.714 (# 0.127)
VI. DongonalNormal
0.747 (= 0.138)
0.136 (+ 0.123)
3.460 (+: 0.361)
0.836 (+: 0.053)
0.203 (=: 0.086)
2.977 (+t 0.1
Scenario 6
20
VI: Multivariate Normal
0.621 (1: 0.016)
0.016 (: 0.002)
3,564 0.290)
0.608 0.017)
0.015 0.003)
3.101 (+: 0.115)
VI: Stractured Normal
0.599 (+= 0.015)
0.012 (+: 0.002)
3.592 (+ 0.267)
0.584 (+: 0.028)
0.012 (+ 0.002)
3.120 (+: 0,1)
VI: IAF
0.625 (1. 0.040)
0.019 (.L 0.009)
3.572 (LL 0,266)
0.636 (1. 0.021)
0.020 (.1 0.005)
3.106 (1 0.128)
(ours)
0.747 (2: 0.148)
0.163 (# 0.144)
4.063 (1: 0.184)
0.928 0.030)
0.463 0.162)
4.425 (2 0.314)
Laplace Approxinationn
1.000 (1: 0.000)
2.291 (4 0.003)
6.742 (1: 0.362)
1.000 (1:
2.293 (:): nan)
6.587 (1: nan)
VI: DiagonalNormal
0.761 (1: 0.138)
0.087 (:2 0.083)
6.909 (:1: 0.743)
0.905 (1: nan)
0.175 (1: nan)
6.403 (:1 nan)
Sccnario 6
50
VI: MuliivariateNorma
0.797 (1. 0.100)
(1 0.055)
6.956 (1L 0.736)
0.891 nan)
0.110 nam)
6.473 (11 nan)
VI Stractured Normal
0.647 (: 0.017)
0,013 (= 0.002)
7,218 (1: 0.506)
0.654 (#: nan)
0.013 (# nan)
6.890 (:: nan)
VI. IAF
0.639 (=z 0.038)
0.014 (=: 0.006)
7,204 (+: 0. 0.463)
0.692 (+: nan)
0.024 (+t nan)
6.887 (+t nan)
ICL (ours)
0.742 (+: 0.178)
0.115 (:: 0.124)
7.713 (+ 0.
0.935 nan)
0.203 (#) nam)
7,846 (+: nan)
43

---

Can Transformers Learn Full Bayesian Inference In Context?
Table 38: Generalized Linear Models: Ablation with respect to the dimensionality of the problem on 50 synthetic and 17 weal-world datasets for scenario 7. All results within two standard errors of the best average result for each scenario are marked in bold. Due to the limitations of the number of features in the real-world data, we can only use 5 datasets for 20 and one dataset for 50 dimensions.
Synthetic Evaluation
Woul-World Evaluation
Scenario
Dim.
Model
C2ST (L)
MMD (U)
(4)
C2ST (L)
MMD ())
[/C (U)
Laplace Approximation
1.000 (#: 0.000)
3,559 (+: 1.933)
1.347 1.067)
1.000 (:: 0.000)
2.016 (# 0.080)
0.763 (#: 0.174)
VI: NiggonalNormal
0.938 (+: 0.074)
2,536 (+ 2.097)
1.142 (#: 0.993)
0.936 (+ 0.024)
1.029 [+ 0.255)
0.579 (+ 0,181)
Scenario
VI: MultivariateNormall
0.814 (4: 0.181)
1.999 (1 2.283)
1.033 0.969)
0.741 (1 0.020)
0.093 [4: 0.025)
0.391 0.074)
VE: Structured Normal
0.824 (#: 0.177)
1.891
2.127)
1,041 (2: 0,934)
0.734 (1: 0.025)
0,072 [# 0.019)
0,385 (4: 0.065)
VI: IAF
0.939 (#: 0.091)
2,707 (+ 1.712)
1 1,590 (#: 0.820)
0.864 (+ 0.093)
0.830 (+ 0.697)
1.064 (+ 0.616)
ICL (ours)
0.700 (# 0.116)
(1: 0.355)
0.400 0.286)
0.773 0.048)
0.294 (# 0,457)
0.559 0.256)
Laplace Approximation
1,000 (1 0.000)
3.581 2.147)
3,365 (1: 1.583)
1.000 (1 0.000)
2.213 (1. 0.024)
2.539 (1. 0.378)
VI: DiagonalNornal
0,887 (# 0.184)
2 (= 2.732)
3.637 (#: 1.371)
0.996 (1: 0.0015
1.734 (22 0.314)
2.348 (#:: 0.423)
Secnario 7
20
VI: MultivariateNormall
0.881 (1 0.164)
2.265 2.573)
3.524 1.392)
0.916 (1 0.085)
0.766 (11 0.535)
2,043 0.516)
VI: Structured Normal
0,850 (#: 0.162)
16677 2.266)
3,186 (#: 1.315)
0.849 (1: 0.105)
0,391 (#: 0,244)
1,800 (#: 0.367)
VI: IAF
0.867 (+ 0,184)
1.629 (= 1.584)
4.875 (#: 1.239)
0.986 (+ 0.007)
0.895 (+t 0,361)
4.096 (+: 0,319)
ICI. (ours)
(#: 0.185)
(#: 1.352)
4.836
1.032)
0.982 (1 0.010)
(:: 0.324)
(4: 0.368)
Luplace Approximation
1,000 (.L 0.000)
4.768 (1 1.171)
6.573 (1) 1.038)
1.000 (1 nun)
2.312 (L nan)
5,270 (.L. nan)
VI: DiagonalNormal
0.771 (# 0.191)
3.263 (= 1.853)
6.919 (#: 1.257)
1,000 (#: nan)
2.237 (:: nan)
5.417 (#: nan)
Sccnario 7
50
VI: MuliivariateNorall
0.816 (+ 0.154)
3.245 (= 1.793)
6.978 1.226)
0.997 (+ nan)
2.117 (+ nan)
5,781 (+ nan)
VI: Structured Normal
0.795 (#: 0.171)
3.126 (= 1.677)
6,918 (# 1.260)
1.000 (#: nan)
1,879 (# nan)
5,461 (#: nan)
VI: TAP
0.769 (+t 0.189)
2.534 (- 0.894)
7 7,895 (+: 0.843)
0.994 (+ nan)
0.584 (+ nan)
7.626 (+ nan)
(ours)
(1L 0.216)
2.451 0.790)
7.787 0.661)
0,980 (1. nan)
(1 nan)
7,461 nam)
P. Comparison to SGLD
Besides comparing the samples from our ICL approach to samples from various VI methods, we additionally compare it against samples generated via stochastic gradient Langevin dynamies (SGLD) (Welling & Tch, 2011). We run SGLD with a learning rate of 10-3 for the GLM and GMM cases and learning rate of 10-4 for FA and use 1000 gradient steps for warmup and partition the data into ten minibatches. We implement the preconditioning method introduced by (Li et al., 2016) for more stable sampling behavior. Despite the preconditioning, SGLD consistently fails for GLMs scenario 7 because the sampler diverges causing singular covariance matrices. To facilitate running SGLD for the GMMs, which also include discrete variables, we marginalize over the discrete variables.
In summary, we find that ICI. yields samples with much higher quality than SGL.D compared to the gold standard HMC samples across almost all scenarios on both synthetic and real-world data. The poor sample quality with SGLD is expected given that numerous theoretical and empirical findings confirm that, while SGLD is computationally very cheap, it is substantially outperformed by, for instance, HMC, in terms of sample quality, which is especially pronounced when the posterior distributions are complex and parameters are correlated (Chen et al., 2014; Mangoubi & Vishnoi, 2019; Izmailov et al., 2021; Brosse et al., 2018).
For GLMS (Table 42), ICI. achieves significantly better results, with notable improvements in C2ST. In Scenario 1, synthetic C2ST drops from 0.992 to 0.765 and real-world C2ST from 0.980 to 0.614. Similarly, Scenario 3 shows substantial gains, with synthetic C2ST improving from 0.997 to 0.611 and worl-world C2ST from 0.983 to 0.576. These trends extend to metrics like W/, where ICL yields consistent reductions.
For FA (Table 43), ICL also achieves superior performance, particularly in Scenarios 1 and 2. For example, in Scenario 1, synthetic C2ST decreases from 0.996 to 0.552, accompanied by improvements in W2 from 1.776 to 0.289. Scenario 2 sees further cnhancements, with synthetic MMD dropping from 2.950 to 0.017 and real-world C2ST improving from 0.995 to 0.622.
For GMMs (Table 44), ICL demonstrates clear advantage in most scenarios. In Scenario 1, ICL reduces synthetic C2ST from 1.000 to 0.760 and real-world W22 from 6.510 to 4.054. Scenario 2 shows synthetic C2ST improving from 1.000 to 0.812, and MMD from 3.046 to 0.159. While in scenarios 3, ICL has a singificantly lower MMD score on the synthetic data, the other differences are not significant.
44

---

Can Transformers Learn Full Bayesian Inference In Context?
Table 39: Evaluating the predictive performance different dimensionalities. All results within two standard emrors of the best average result for each scenario are marked in bold. Due to the limitations of the number of features in the real-world data, we can only use 5 datasets for 20 and one dataset for 50 dimensions. We find that the quality of the samples by the in-context learner, when evaluated based on predictive performance, decreases consistently with an increase in the dimensionality of the problem. Note that the randomness in the results, especially for higher dimensionalities, can in rare cases lead to better mean values. This most likely not significant when taking the standard error into account.
50 synthetic and 17 real-world datasets in GLM scenario 2 for
Scenario
Dim.
Model
RMSE Real-World
RMSE Synthetic (L)
HMC
0.559 (1 0.023)
0.556 (1 0.049)
Laplace Approximation
0.561 (1 0.022)
0,557 (# 0.049)
VI: DiagonalNormal
0.560 (+ 0.023)
0.557 (+ 0.049)
VT: MultivariateNormal
0.559 (+ 0.023)
0.556 (+ 0.049)
Scenario 2
VI: Structured Normal
0.604 (#: 0.016)
0.685 (# 0.054)
VI: IAF
0.563 (#: 0.023)
0.557 (#: 0.049)
ICL (ours)
0,561 (1 0.019)
0.653 (1 0.049)
MAP
0.513 (1 0.023)
0.522 (1 0.048)
TabPFN
0.449 (# 0.034)
0.498 (# 0.047)
HMC
0.682 (# 0.029)
0.536 0.041)
Laplace Approximation
0.682 (1 0.030)
0,538 (1 0.040)
VI: DiagonalNormal
0.680 (+ 0.029)
0.539 (+ 0.041)
VI: MultivariateNormal
0.685 (+ 0.029)
0.537 (+ 0.041)
Scenario 2
20
VI: Structured Normal
0.746 (#: 0.019)
0.681 (# 0.041)
VI: IAF
0.683 (#: 0.029)
0.539 (#: 0.041)
ICL (ours)
0.777 (1 0.011)
1.122 (11 0.078)
MAP
0.578 (.1: 0.025)
0.472 (+1: 0.039)
TabPFN
0.470 (#: 0.044)
0.446 0.038)
HMC
0.669 (# nan)
0.713 (= 0.060)
Laplace Approximation
0.594 (1 nan)
0.878 (1 0.068)
VI: DiagonalNormal
0.582 (+ nam)
0.870 (+ 0.065)
VI: MultivariateNormal
0.729 (+ nan)
0.764 (+ 0.066)
Scenario 2
50
VI: Structured Normal
0,922 (#: nan)
1.116 (= 0.074)
VI: IAF
0.695 (# nan)
0.770 (= 0.060)
ICL (ours)
1,256 (1: nam)
2.343 (+: 0.230)
MAP
0.301 (1: nam)
0.398 (+: 0.047)
TabPFN
0.235 (+ nan)
0.570 (# 0.053)
45

---

Can Transformers Learn Full Bayesian Inference In Context?
Table 40: Evaluating the predictive performance across 50 synthetic and 17 real-world datasets in GLM scenario 2 for different dimensionalities. All results within two standard errors of the best average result for each scenario are marked in bold. Due to the limitations of the number of features in the real-world data, we can only use datasets for 20 and one dataset for 50 dimensions. We find that the quality the samples by the in-context learner, when evaluated based on predictive performance, decreases consistently with an increase in the dimensionality of the problem.
Scenario
Dim.
Model
RMSE Real-World (L)
RMSE Synthetic (L)
HMC
0.684 (1: 0.027)
0,512 (#: 0.040)
Laplace Approximation
0.688 (# 0.026)
0.516 (= 0.040)
VI: DiagonalNormal
0.686 (1 0.027)
0,513 (1 0.040)
VI: MultivariateNormal
0.685 (1 0.027)
0.512 (1 0.040)
Scenario 3
VI: Structured Normal
0.733 (+ 0.016)
0.607 (+ 0.043)
VI: IAF
0.686 (+ 0.027)
0.512 (+ 0.040)
ICL (ours)
0.690 (#: 0.023)
0.588 (= 0.045)
MAP
0.646 (#: 0.028)
0.495 (= 0.039)
TabPFN
0.556 (4: 0.041)
0.462 (+ 0.037)
HМC
1.030 (4: 0.045)
0.621 (# 0.046)
Laplace Approximation
1.053 (# 0.047)
0.755 (= 0.052)
VI: DiagonalNormal
1.035 (# 0.043)
0.734 (= 0.053)
VI: MultivariateNormal
1.033 (1 0.039)
0.705 (1 0.055)
Scenario 3
20
VI: Structured Normal
1.095 (+ 0.045)
1.033 (+ 0.063)
VI: TAF
1.026 (+ 0.045)
0.653 (+ 0.047)
ICL (ours)
1.770 (#: 0.048)
2.160 (= 0.217)
MAP
0,861 (#: 0.038)
0,581 (# 0.050)
TabPFN
0.654 (1: 0.062)
0.475 (+ 0.039)
HMC
0.858 (+: nam)
0.645 (+ 0.051)
Laplace Approximation
0.866 (# nan)
0.865 (= 0.083)
VI: DiagonalNormal
0.788 (=: nan)
0.870 (= 0.084)
VI: MultivariateNormal
0.819 (1 nan)
0.778 (1 0.066)
Scenario 3
50
VI: Structured Normal
0,812 (+: nam)
1.040 (+ 0.103)
VI: IAF
0.802 (+ nan)
0.846 (+ 0.078)
ICL (ours)
1.686 (1: nan)
3.477 (= 0.604)
MAP
0.539 (1: nan)
0,618 (# 0.054)
TabPFN
0.322 (#: nan)
0.534 (# 0.038)
46

---

Can Transformers Learn Full Bayesian Inference In Context?
Table 41: Evaluating the predictive performance across 50 synthetic and 17 real-world datasets in GLM scenario 2 for different dimensionalities. All results within two standard errors of the best average result for each scenario are marked in bold. Due to the limitations of the number of features in the real-world data, we can only use 5 datasets for 20 and one dataset for 50 dimensions. We find that the quality of the samples by the in-context learner, when evaluated based on predictive performance, decreases consistently with an increase in the dimensionality of the problem.
Scenario
Dim.
Model
RMSE Real-World (1)
RMSE Synthetic (L)
HMC
0.699 (+ 0.022)
0.490 (+ 0.036)
Laplace Approximation
0.699 (# 0.022)
0.491 (# 0.036)
VI: DiagonalNormal
0.702 (#: 0.022)
0.491 (#: 0.036)
VI: MultivariateNormal
0.698 (1: 0.021)
0.491 (#: 0.036)
Scenario 5
5
VI: Structured Normal
1.507 (# 0.089)
0.741 (= 0.053)
VI: IAF
0.699 (1 0.022)
0.490 (1 0.036)
ICL (ours)
0.769 (1 0.020)
0.701 (= 0.049)
MAP
0.658 (1 0.022)
0,471 (+ 0.035)
TabPFN
0.534 (+ 0.040)
0.442 (+ 0.035)
HMC
1,527 (+ 0.055)
0,553 (+ 0.044)
Laplace Approximation
1.585 (#: 0.065)
0.586 (= 0.043)
VI: DiagonalNormal
1.554 (#: 0.058)
0.586 (= 0.042)
VI: MultivariateNormal
1.530 (4: 0.058)
0.564 (# 0.043)
Scenario 5
20
VI: Structured Normal
2.109 (# 0.156)
1.054 (= 0.067)
VI: IAF
1.548 (#: 0.057)
0.562 (= 0.043)
ICL (ours)
3.545 (1 0.288)
1.626 (1 0.140)
MAP
1.254 (1 0.027) 0.668 (+ 0.064)
0.464 (1 0.035)
TabPFN
0.413 (+ 0.032)
HMC
1.626 (+ nam)
0,521 (+ 0.028)
Laplace Approximation
1.541 (#: nan)
0.655 (= 0.040)
VI: DiagonalNormal
1.576 (#: nan)
0,639 (= 0.041)
VI: MultivariateNormal
1.659 (4: nan)
0.592 (+: 0.035)
Scenario 5
50
VI: Structured Normal
2.076 (# nan)
1.018 (= 0.102)
VI: IAF
1,706 (#: nam)
0.627 (= 0.040)
ICL (ours)
10.319 (1 nan)
1.458 (1 0.193)
MAP
1.318 (1 nan)
0.416 (1 0.018)
TabPFN
0.330 (1: nan)
0.443 (1 0.024)
Table 42: SGLD vs. ICL: Evaluation on 50 synthetic and 17 real-world datasets for six different GLM scenarios. All results within two standard errors of the best average result for each scenario are marked in bold.
Synthetic Evaluation
Real- World Evaluation
Scenario
Model
C2ST (L)
MMD ())
Wi (L)
C2ST (L)
MMD (L)
W, (U)
SGLD
0.992 (=: 0.015)
2,846 (# 1.41))
1951 (# 0.917)
0.980 (#: 0.013)
2.191 (# L.183)
0.865 (#: 0.438)
Scenario
ICL (ours)
0.765 (=: 0.123)
0.767 (#: 0.727)
0,585 (#: 0.301)
0.614 (#: 0.074)
0.175 (1: 0.219)
0.310 (=: 0.138)
SGLD
0.999 (= 0.004)
5.650 (+: 1.762)
8.295 (+: 5.629)
0.994 (+: 0.006)
2.699 (+: 1.093)
1.289 (+: 0.454)
Scenario 2
ICL (ours)
0.839 (1: 0.072)
0.707 0.658)
1.111 0.300)
0.768 (1: 0.033)
0.143 (1: 0.089)
0.411 0.094)
SGLD
0.997 (= 0.008)
3.320 (#: 1.595)
3.011 (+ 1.036)
0.983 (+: 0.013)
2.152 (:: 1.194)
0.935 (#: 0.523)
Scenario 3
ICL (ours)
0.611 (=: 0.070)
0.089 (# 0.1
0,423 (+ 0.348)
0.576 (+: 0.027)
0.037 (4: 0.026)
0.257 (#: 0.044)
SGLD
1.000 (+ 0.000)
6.626 (+ 1.215)
15.674 (+ 8.100)
0.994 (+ 0.006)
2.927 (+ 1,564)
1.606 (+ 1.022)
Scenario
ICT. (ours)
0.753 (+ 0.049)
0.171 (+ 0.153)
0.631 (+ 0.294)
0.762 (+ 0.015)
0.105 (+ 0.046)
0.597 (+ 0.104)
SGLD
0.999 (.1: 0.003)
3.308 1.728)
2.216 1.247)
1.000 (1: 0.000)
4.012 (1L 1.413)
0.996 0.406)
Scenario
ICL (ours)
0.621 (1:: 0.063)
0.067 0.080)
0.299 0.195)
0.610 (1: 0.045)
0.046 (1. 0.020)
0.242 0.038)
SGLD
0.998 (# 0.001)
2.681 0.565)
2.419 0.510)
0.998 (4: 0.002)
2.845 (1: 0.590)
1.851 (#: 0.319)
Scenario 6
ICL (ours)
0.532 (= 0.019)
0.016 (# 0.008)
0.590 (+ 0.066)
0.556 (+ 0.017)
0.035 (#: 0.015)
0.504 (#: 0.038)
47

---

Can Transformers Learn Full Bayesian Inference In Context?
Table 43: SGLD vs. ICL: Evaluation on 50 synthetic and 17 real-world datasets for six different FA scenarios. All results within two standard errors of the best average result for each scenario are marked in bold.
Synthetic Evaluation
Real- World Evaluation
Scenario
Model
C2ST ())
MMD ())
W/2 (4)
C2ST ())
MMD (L)
W2 (
SGLD
0.996 (= 0.006)
2.883 (# 1,552)
1.776 (# 0.694)
0.995 (# 0.003)
2.676 (# 0.710)
1.608 (# 0,381)
Scenario 1
ICL (ours)
0.552 (# 0.028)
0.034 (#: 0.034)
0.289 0.083)
0.606 0.038)
0.068 (#: 0.069)
0.265 (#: 0.078)
SGLD
0.997 (+: 0.003)
2.950 (+: 0.786)
1.892 (+: 0.533)
0,995 (+ 0.003)
2.517 (+: 0,583)
1,500 (+: 0.268)
Scenario 2
ICL (ours)
0.542 (1: 0.006)
0.017 (1 0.006)
0.244 (1: 0.033)
0.622 (1 0.032)
0.098 (.L. 0.039)
0.287 (1. 0.046)
SGLD
0.998 (# 0.005)
3.662 (+ 1.099)
2.086 (# 0.919)
0.956 (+ 0.025)
1,580 (# 0.819)
0.311 (+ 0.108)
Scenario 3
ICL (ours)
0.537 (= 0.023)
0.024 (#: 0.021)
0.259 0.088)
0.609 0.019)
0.124 (# 0.037)
0.179 (1: 0.018)
SGLD
1.000 (+: 0.000)
4.127 (+: 0.635)
3.047 (+: 0.972)
0.950 (+: 0.021)
1.520 (::: 0.512)
0.141 (+: 0.031)
Scenario 4
ICL (ours)
0.684 (1 0.060)
0.198 (1L 0.141)
0.918 (1 0.246)
0,988 (1 0.003)
1,764 (.L. 0.026)
1,248 (1 0.008)
SGLD
0.999 (= 0.001)
3.465 (+ 0.939)
1.981 1 (+ 0,938)
0.962 (+ 0.024)
1.945 (+ 1,383)
0.393 (+ 0.243)
Scenario 5
ICL (ours)
0.535 (# 0.016)
0.021 (#: 0.011)
0.279 0.060)
0.886 0.017)
1.207 (# 0.101)
1.002 (# 0.042)
SGLD
0.997 (+: 0.004)
3.395 (t: 1.199)
2.358 (:: 1.458)
0,950 (+: 0.040)
2.177 (t: I (.643)
0.342 (1: 0.224)
Scenario 6
ICL (ours)
0.543 (1 0.021)
0.023 (1 0.015)
0.345 (1L 0.173)
0.666 (.L 0.020)
0,200 (.L 0.034)
0.224 (1L 0.014)
Table 44: SGLD vs. ICL: Evaluation on 50 synthetic and 17 real-world datasets for four different GMM scenarios. All results within two standard errors of the best average result for each scenario are marked in bold.
Synthetic Evaluation
Real- World Evaluation
Scenario
Model
C2ST (L)
MMD ())
W/2 (L)
C2ST (L)
MMD (1)
W3 (
SGLD
1.000 (:i: 0.001)
2.629 (+L: 0.868)
3.279 (#:: 1,330)
1,000 (:L 0.000)
3,421 (dL 0,877)
6,510 (1: 1.763)
Scenario
ICL (ours)
0.760 (1: 0.092)
0.303 (1: 0.548)
2.095 (#: 1,692)
0.847 (::: 0,082)
0.486 (1: 0.623)
4.054 (#: 2.782)
SGLD
1.000 (=: 0.000)
3.046 1.041)
6.015 4.265)
1.000 (+: 0.000)
2.487 (1: 0.521)
6,858 1.618)
Scenario 2
ICL (ours)
0.812 (= 0.061)
0.159 0.154)
2.314 0.926)
0.937 (+: 0.041)
0.282 (#: 0.131)
3,947 1.055)
SGLD
1.000 (+: 0.000)
4.631 (+ 1.169)
23.247 (+: 30.646)
1.000 (+: 0.000)
2.655 (+: 0.437)
26.356 (+ 2.699)
Scenario 3
ICI. (ours)
1.000 (+ 0.000)
0.582 (+ 0.280)
8.708 (+ 4,945)
1.000 (+ 0.000)
1.869 (+ 0.342)
33,230 (+ 8.095)
SGLD
1.000 (:1: 0.000)
3.464 [ 1.098)
6.995 (dL: 5.554)
1.000 (dL: 0.000)
2.555 (1L: 0.494)
9,477 (L:: 3,432)
Scenario
ICL (ours)
1.000 (::: 0.000)
2.451 (1L: 0.868)
8,333 (1L 4,202)
1.000 (d:: 0.000)
2.518 (1: 0.694)
11.938 (#: 2.956)
48

---

Can Transformers Learn Full Bayesian Inference In Context?
Q. Evaluation the choice of classifier for the C2ST metric
In this section, we validate the choice of the classifier for the C2ST metric by comparing the ROC characteristic random forest (our default choice) and neural network in distinguishing posterior samples. In summary, we find that despite minor differences, the two metrics yield the same overall results. Across all scenarios, both Random Forest (RF) and Neural Network NN classifiers yield quite consistent rankings of model performance with only insubstantial deviations in terms of the big picture. In particular, ICL is consistently among the top-performing approaches under both evaluation metrics. Out of the 14 total scenario-domain combinations (7 scenarios x 2 dataset types), the RF and NN metrics identify the same best-performing model in 12 cases.
49

---

Can Transformers Learn Full Bayesian Inference In Context?
Table 45: Generalized L Linear Models: Comparison of C2ST scores with a Random Forest (RF) and a Neural Network (NN).
For the NN we follow the setup of Lueckmann et al., 2021. Evaluation across seven distinct scenarios on 50 synthetic and 17 real-wont world datasets. All results within two standard errors of the best average result in each scenario are marked bold.
Synthetic Evaluation
Real-World Evaluation
Scenario
Model
C2ST RF (4)
C2ST NN (J)
C2ST RF (4)
C2ST NN (1)
Laplace Approximation
1.000 0.000)
0.998 (# 0.000)
1.000 0.000)
0.998 (# 0.000)
VI: DiagonalNormal
0.904 (#: 0.076)
0.857 (#: 0.001)
0.797 (#: 0.083)
0.803 (# 0.004)
VI: MultivariateNormal
0.750 (= 0.128)
0.780 (=
0.607 (# 0.070)
0.713 (= 0.004)
Scenario 1
VI: Structured Normal
0.753 (= 0.126)
0.781 (# 0.002)
0.600 (# 0.070)
0.705 (= 0.004)
VI: IAF
0.777 (#: 0.122)
0.793 (# 0.002)
0.683 (# 0.132)
0.746 (= 0.006)
HMC
0.745 (= 0.130)
0.777 (# 0.002)
0.595 (= 0.075)
0.702 (= 0.004)
ICL (ours)
0.765 (# 0.123)
0.712 (# 0.002)
0.614 (= 0.074)
0.701 (== 0.004)
Laplace Approximation
1.000 0.000)
0.998 (#: 0.000)
1.000 (= 0.000)
0.998 (== 0.000)
VI: DiagonalNormal
0.957 (= 0.091)
0.883 (# 0.002)
0.892 0.044)
0.851 (= 0.003)
VI: MulrivariateNormal
(=:0331)
0.860 0.002)
0.820 0.031)
0.815 (== 0.003)
Scenario 2
VI: Structured Normal
0.908 (.119)
(=
0.817 (- 0.003)
VI: IAF
0.968 (+: 0.063)
0.889 (+: 0.001)
0.888 (#: 0.067)
0.849 (= 0.004)
ICL (ours)
0.839 (= 0.072)
0.824 0.001)
0.768 (= 0.033)
0.789 (= 0.003)
Laplace Approximation
1.000 (=: 0.000)
0.998 (# 0.000)
1.000 (= 0.000)
0.998 (= 0.000)
VI: DiagonalNormal
0.866 (= 0.101)
0.838 (# 0.002)
0.797 (= 0.083)
0.803 (= 0.004)
VI: MulrivariiteNormal
0.656 (+: 0.131)
0.733 (+ 0.002)
0.590 (#: 0.035)
0.685 (= 0.003)
Scenario 3
VI: Structured Normal
0.653 (# 0.125)
0.731 (# 0.002)
0.582 (= (.0088) 0.028)
0.681 (= 0.003)
VI: IAF
0.751 (:: 0.148)
0.780 (+: (+ 0.002)
0.673 (# 0.141)
0.741 (= 0.006)
ICL (ours)
0.611 (= 0.070)
0.710 (= 0.001)
0-
0.693 (= 0.003)
Laplace Approximation
1.000 (=: 0.000)
0.998 (# 0.000)
(=
0.998 (= 0.000)
VI: DiagonalNormal
0.968 0.036)
0.889 (=
0.
0.863 (= 0.003)
VI: MultivariateNormal
0.855 (::00233
(4 0.002)
0.771 (+
0.790 (= 0.002)
Scenario 4
VI: Structured Normal
0.847 0.116)
0.828 (+ 0.002)
0.769 0.01))
0.789 (= 0.002)
VI: IAF
0,942 (=: 0.077)
0.876 (#: 0.001)
0.833 0.066)
0.821 (= 0.004)
ICL (ours)
0.753 (= 0.049)
0.781 (# 0.001)
0.762 (=
0.786 (= 0.002)
Laplace Approximation
1,000 (=: 0.000)
0.998 (= 0.000)
1.000 (= 0.000)
0.998 (= 0.000)
VI: DiagonalNormal
0.866 (=: 0.085)
0.838 (+ (..002)) 0.002)
0.810 0.036)
0.810 0.003)
VI: MultivariateNormall
0.765 (= 0.100)
(1
0.760 0.003)
Scenario 5
VI: Structured Normal
0.758 (=: 0.098)
0.784 (+: 0.002)
0.705 0+03))
0.757 (= 0.003)
VI: IAF
0,814 (= 0.105)
0.812 0.002)
0.777 (= 0.106)
0.793 (= 0.005)
ICL (ours)
0.621 (=: 0.063)
0-715 (=:000) 0.001)
0.610 (= 0.045)
0.710 (= 0.003)
Laplace Approximation
1,000 (=: 0.000)
0.998 (+: 0.000)
1,000 (= 0.000)
0.998 (= 0.000)
VI: DiagonalNormal
0.724 (+= 0.060)
0.767 (+ 0.001)
0.703 (+ 0.039)
0.756 (+ 0.003)
VI: MultivariateNormal
0.534 0.018)
0.672 (# 0.001)
0.538 (=
0.674 (= 0.002)
Scenario 6
VI: Structured Normal
0.536 (:: 0.016)
0.673 (+: 0.001)
0.536 0.019)
0.673 (+ 0.002)
VI: IAF
0.542 (= 0.026)
0.676 (#
0.535 (=
0,672 (= 0.002)
ICL (ours)
0.532 (=: 0.019)
0.671 (# 0.001)
0.556 (= 0.017)
0.653 (= 0.002)
Laplace Approximation
1,000 (= 0.000)
0.998 (# 0.000)
1.000 0.000)
0.998 (= 0.000)
VI: DiagonalNormal
0.938 (=: 0.074)
4874
0.936 (= 0.024)
0.873 (= 0.003)
VI: MultivariateNormal
0.814 0.181)
0.812 0.000)
0.741 (# 0.020)
0.775 (= 0.003)
Scenario 7
VI: Structured Normal
0.824 0.177
0.817 (1 0.002)
0.772 (= 0.003)
VI: IAF
0.939 (= 0.091)
0.874 ( 0.002)
0.864 ( 0.093)
0.837 (= 0.005)
ICL. (ours)
0.700 (+ 0.116)
0.721 (+ 0.002)
0.773 (+ 0.048)
0.751 (= 0.003)
50

---

Can Transformers Learn Full Bayesian Inference In Context?
Table 46: Factor Analysis: Comparison of C2ST scores using Random Forest (RF) and Neural Network (NN) classifier across six different scenarios on 50 synthetic and 17 real-world datasets. For the NN we follow the setup of Lueckmann et al., 2021. All results within two standard errors of the best average result in each scenario are marked in bold.
Synthetic Evaluation
Real-World Evaluation
Scenario
Model
C2ST RF (4)
C2ST NN ())
C2ST RF (1)
C2ST NN (1)
Laplace Approximation
1.000 (= 0.000)
0.997 (# 0.000)
1.000 (= 0.000)
0.997 (# 0,000)
VI: DiagonalNormal
1.000 (#: 0.001)
0.997 (# 0.000)
0.979 (# 0.008)
0.950 (= 0.001)
VI: MultivariateNormall
0.998 (#: 0.003)
0.960 (=00000))
0.966 (#
0.944 0.001)
Scenario 1
VI: Structured Normal
0.997 (#: 0.004)
0.959 (+: 0.000)
0.979 (+ 0.010)
0.950 0.001)
VI: IAF
0.953 (= 0.104)
0.937 0.001)
0.849 (= 0.075)
0.885 (= 0.003)
ICL (ours)
0.552 (=: 0.028)
0.737 (#: 0.000)
0.606 (= 0.038)
0.764 (== 0.001)
Laplace Approximation
1.000 (#: 0.000)
0.997 (#: 0.000)
1,000 0.000)
0.997 (# 0.000)
VI: DiagonalNormal
0.998 (=: 0.002)
0.960 (# (+0000) 0.000)
0.975 (=
0.948 (= 0.001)
VI: MultivariateNormal
0.989 (=: 0.009)
0.955 (= (=:000))) 0.000)
0.951 (=
0.936 0.001)
Scenario 2
VI: Structured Normal
0.984 (:: 0.031)
0.953 0.000)
0.958
0.940 (= 0.001)
VI: IAF
0.966 (= 0.066)
0.944 0.001)
0.799 (= 0.058)
0.860 (= 0.002)
ICL (ours)
0.542 (=: 0.006)
0.732 (# 0.000)
0.622 (= 0.032)
0.772 (= 0.001)
Laplace Approximation
1.000 (::: 0.000)
0.997 (#:: 0.000)
1.000 (#:: 0.000)
0.997 (= 0.000)
VI: DiagonalNormal
0.999 (= 0.002)
0.960 (= 0.000)
0.951 (= 0.007)
0.936 (= 0.001)
VI: MultivariateNormal
0.994 (= 0.007)
0.958 (+ 0.000)
0.945 (= 0.007)
0.933 (= 0.001)
Scenario 3
VI: Structured Normal
0.997 (=: 0.003)
0.959 (+: 0.000)
0.942 (# 0.009)
0.932 (= 0.001)
VI: IAF
0.990 (:: 0.011)
0.987 (+: 0.000)
0.928 (+ 0.015)
0.925 (= 0.001)
ICL (ours)
0.537 (= 0.023)
0.729 (# 0.000)
0.609 (= 0.019)
0.765 (= 0.001)
Laplace Approximation
1,000 (=: 0.000)
0.997 (#: 0.000)
1,000 (= 0.000)
0.997 (= 0.000)
VI: DiagonalNormal
1,000 (= 0.000)
0.997 (# 0.000)
0.977 (= 0.003)
0.949 (= 0.000)
VI: NuriivariateNormal
0.999 (+ 0.001)
0.960 (+ 0.000)
0.973 0+00))
0.947 (= 0.001)
Scenario 4
VI: Structured Normal
1.000 (= 0.000)
0.997 (=
0.973 (= (=007)
0.947 (= 0.001)
VI: IAF
0.999 (+:: 0.001)
0.960 (+ 0.000)
0.961 (# 0.018)
0.941 (= 0.001)
ICL (ours)
0.684 (= 0.060)
0.803 (1 0.001)
0.988 (= 0.003)
0.955 0.000)
Laplace Approximation
1,000 (=: 0.000)
0.997 (#: 0.000)
1.000 (= 0.000)
0.997 (= 0.000)
VI: DiagonalNormal
0.999 (=: 0.002)
0.960 (=: 0.000)
0.944 (= 0.010)
0.933 (= 0.001)
VI: MultivariateNormal
0.995 (= 0.007)
0.958 ( 0.000)
0.930 (=
0.926 (= 0.001)
Scenario 5
VI: Structured Normal
0.998 (=: 0.005)
0.960 0.000
0.934 0.01)
0.928 (= 0.001)
VI: IAF
0.992 (#: 0.012)
0.957 0.000)
0.916 (..00)) 0.001)
ICL (ours)
0.535 (= 0.016)
0.728 (+ 0.000)
0.886 ( 0.017
0.904 (= 0.001)
Laplace Approximation
1.000 (= 0.000)
0.997 (#: 0.000)
1.000 (= 0.000)
0.997 (= 0,000)
VI: DiagonalNormal
0.998 (=: 0.002)
0.960 (#: 0.000)
0.949 (# 0.008)
0.935 (= 0.001)
VI: MultivariateNormal
0.991 (= 0.013)
0.938 0.00)
0.930 0.001)
Scenario 6
VI: Structured Normal
0.997 (+= 0.005)
0.959 (+ 0.000)
0.944 0.006)
0.933 (= 0.001)
VI: IAF
0.989 (=: 0.029)
0.955 (+ 0.000)
0.865 (= 0.027)
0.893 (= 0.001)
ICL (ours)
0.543 (= 0.021)
0.732 (# 0.000)
0.666 (= 0.020)
0.794 (=94(=00) 0.001)
51

---

Can Transformers Learn Full Bayesian Inference In Context?
Table 47: Gaussian Mixture Models: Comparison of C2ST scores using Random Forest (RF) and a Neural Network (NN) classifier across six distinct scenarios on 50 synthetic and 17 real-world datasets. All results within two standard errors of the best average result in each scenario are marked in bold. For the NN we follow the setup of Lueckmann al., 2021. Both RF and NN classifiers yield consistent rankings, with ICL emerging as the top method in scenarios with more pronounced model mismatch.
Synthetic Evaluation
Real-World Evaluation
Scenario
Model
C2ST RF (L)
C2ST NN (L)
C2ST RF (L)
C2ST NN (J)
Laplace Approximation
1.000 (+ 0.000)
0.997 (# 0.000)
1,000 (+ 0.000)
0.997 (# 0.000)
VI: DiagonalNormal
0.988 (#: 0.013)
1.012 (#: 0.000)
0.995 (#: 0.006)
0.996 (# 0,001)
VI: MultivariateNormal
0,988 (#: 0.013)
1.012 (= 0.000)
0.994 (#: 0.007)
0.993 (# 0.001)
Scenario
VI: Structured Normal
0.987 (+: 0.015)
0.982 (+: 0.000)
0.993 (+: 0.009)
0.992 (+ 0.001)
VI: IAF
0.989 (#: 0.013)
0.983 0.000)
0.995 (#: 0.010)
0.996 (#: 0.001)
ICL (ours)
0.760 (# 0.092)
0.825 (= 0.001)
0.847 (# 0.082)
0.869 (# 0.003)
Laplace Approximation
1.000 (+ 0.000)
0.997 (+ 0.000)
1.000 (+= 0.000)
0.997 (+ 0.000)
VI: DiagonalNormal
0.989 (+ 0.024)
0.983 0.000)
0.998 (# 0.003)
0.997 (#: 0.001)
VI: MultivariateNormal
0.991 (#: 0.021)
0.991 (= 0.000)
0.999 (# 0.002)
1.002 (# 0.001)
Scenario 2
VI: Structured Normal
0.992 (+: 0.017)
0.988 (+: 0.000)
0.999 (+: 0.002)
1.002 (+ 0.001)
VI: IAF
0,992 (#: 0.021)
0.988 (# 0.000)
0.998 (#: 0.004)
0.997 (#: 0.001)
ICL (ours)
0,812 (#: 0.061)
0,851 (= 0.001)
0.937 (# 0.041)
0.915 (# 0.002)
Laplace Approximation
1.000 (+: 0.000)
0.997 (#: 0.000)
1.000 (+: 0.000)
0.997 (+: 0.000)
VI: DiagonalNormal
0.996 (=: 0.011)
1.004 (# 0.000)
0.992 (#: 0.018)
0.988 (# 0.001)
VI: Nurriaa
0.997 (+: 0.009)
1.007 (+ 0.000)
0.993 (+: 0.016)
0.992 (+ 0.001)
Scenario 3
VI: Structured Normal
0.995 (#: 0.017)
0.996 (# 0.000)
0.993 (#: 0.016)
0.992 (#: 0.001)
VI: IAF
0.994 (= 0.018)
0.993 0.000)
0.993 (+ 0.017)
0.992 (#: 0.001)
ICL (ours)
1.000 (+= 0.000)
0.997 (+ 0.000)
1.000 (+ 0.000)
0.997 (+ 0.000)
Laplace Approximation
1.000 (#:: 0.000)
0.997 (# 0.000)
1.000 (:E: 0.000)
0.997 (#: 0.000)
VI: DiagonalNormal
1.000 (#: 0.002)
0.997 0.000)
1.000 (= 0.000)
0.997 (#: 0.000)
VI: NulriaariateNorraall
1.000 (+: 0.002)
0.997 (+: 0.000)
1.000 (+: 0.000)
0.997 (+ 0.000)
Scenario 4
VI: Structured Normal
1.000 (#: 0.001)
0.997 (#: 0.000)
0.996 (=: 0.016)
1.004 (# 0.001)
VI: IAF
1.000 (= 0,002)
0.997 (# 0.000)
1.000 (+ 0.000)
0.997 (# 0.000)
ICL (ours)
1.000 (+: 0.000)
0.997 (+ 0.000)
1.000 (+: 0.000)
0.997 (+ 0.000)
52


Reviews:
[{"review_id": "7vaW4fWRYs", "reviewer": "Reviewer_PknZ", "summary": "The authors present an approach for (approximate) Bayesian inference, based on in-context learning with a combined transformer/flow model. The approach relies on access to the generative model, and generates data to learn the inverse model. The method is reasonable and performs well relative to baselines.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to read our manuscript and for providing detailed feedback.\n\n> The paper is missing an application that motivates a reader, and it is currently not clear that one really exists. \n\nWe would like to point out that the question addressed in our paper is “can transformers learn full Bayesian inference in context?”, which we see and treat as a fundamentally conceptual question. Our work thus stands in line with papers such as [1-2] that investigate different aspects of in-context learning from a conceptual perspective. However, unlike the aforementioned papers [1-2], we also present results on real-world data. We see this as the next, substantially more difficult step to benchmark for the conceptual capabilities of the proposed in-context learning approach.\n\nHowever, we think that learning full Bayesian inference in context has **conceptually** great promise to alleviate important issues with traditional Bayesian inference, including \n\n(a) slow inference time of methods like MCMC. Once pre-trained, an in-context learner can provide samples quickly. \n\n(b) inferior predictive performance compared to non-Bayesian approaches: TabPFN [3] and related approaches show that carefully designing a realistic synthetic distribution to train an in-context can lead to Bayesian methods with excellent predictive performance. \n\n> The authors only compare to simple VI methods and Laplace approximation. \n\nThe main concern of our benchmarks is to demonstrate that transformers are capable of in-context learning full Bayesian inference akin to other recent in-context learning works analyzing whether transformer can learn optimization routines or decision boundaries of other methods in-context, e.g. [1-2]. We thus compare our method against popular and commonly used methods in variational inference as well as HMC for the latent variable models we analyze. \n\nPlease also note that the Laplace approximation, while we fully agree is certainly an outdated method for the considered problems, serves as a simple baseline.\n\n> There is an existing literature on Bayesian methods for inverse problems that would provide better methods\n\nFirst, we would like to point out that approaches that also operate in the flow-matching framework are indeed state-of-the-art methods for Bayesian inference in inverse problems [5]. We further compare against the recently proposed score-matching-posterior-estimation [3] via our ablations in appendix E. Our [new results](https://anonymous.4open.science/r/Extra-B820/FwdKL_Gaussian.pdf) are a comparison further method that can also be used for simulation-based inference.\n\nPlease note that our goal is not to propose a new method for inverse Bayesian problems but to show that in-context learning can be effective, even compared to well-established methods for Bayesian inference that do not operate in context, which present an arguably even stronger baseline. \n\n>While there is some discussion of robustness to OOD in appendix H, the model still feels quite sensitive to the data generation process [...] \n\nWe fully agree that in our setup, the data-generating process we use to train the in-context learner determines what datasets the in-context learner can provide meaningful inference on. Please note, however, that this is expected as we consider controlled and simple setups (GLM, FA, and GMM scenarios) to perform full Bayesian inference. We think that having a somewhat artificial setup of well-established latent variable models where strong traditional baselines exist (HMC, VI) is crucial for exploring whether transformers can do full Bayesian inference in context. \n\nAs you correctly point out, however, this comes at the cost of inherent limitations in terms of OOD performance, which we confirm in our ablation study in appendix E. In principle, however, our in-context learner could be applied to compley, more realistic setups similar to what TabPFN does; but then we would lose the ability to do meaningful evaluations. \n\n> The authors may find \"General-purpose in-context learning by meta-learning transformers\" interesting [...]\n\nThank you for pointing out this relevant and interesting reference. We will discuss its relation to our work in the revised related work section.\n\n> The authors could mention what their approach is [...] earlier in the paper\n\nThank you for this very helpful remark! We will add a short explanation to the introduction. \n\n## References\n\n[1] Garg et al. What can transformers learn in-context? NeurIPS 2022.\n\n[2] Panwar, Ahuja & Goyal. In-Context Learning through the Bayesian Prism. ICLR 2024.\n\n[3] Gloeckler, Manuel, et al. All-in-one simulation-based inference. ICML 2024.\n\n[3] Hollmann et al. Accurate predictions on small data with a tabular foundation model. Nature, 2025.\n\n[3] Ahuja et al. In-Context Learning under Distribution Shifts. Workshop on Efficient Systems for Foundation Models @ ICML 2023.", "claims_and_evidence": "Overall, the claims are supported, although the baselines and experimental evaluation are relatively weak (discussed below).", "methods_and_evaluation_criteria": "The authors should better justify applications in which the this degree of amortized inference is necessary/useful. There are many inverse problems, for which simulation-based data generation is done, that could perhaps provide better motivation than what is currently provided.", "theoretical_claims": "The mathematical results appear to be correct, and are primarily direct application of previous results.", "experimental_designs_or_analyses": "The major weakness of this paper is the experimental evaluation: \n- The paper is missing an application that motivates a reader, and it is currently not clear that one really exists. I could imagine that this approach would be useful for situations in which some form of parameter identification is desirable and must be done repeatedly, but the authors really must add an experiment with a real application. \n- The authors only compare to simple VI methods and Laplace approximation. The use of LA for this problem seems somewhat strange, it is not clear to me why this would be a sensible approach to this problem. There is an existing literature on Bayesian methods for inverse problems that would provide better methods, and the authors could also compare to other amortized variational methods beyond the IAF. This is partially addressed in the diffusion objective ablation, which should probably be presented in the paper body. \n- While there is some discussion of robustness to OOD in appendix H, the model still feels quite sensitive to the data generation process and a more thorough characterization of robustness to model misspecification is likely necessary. This is especially pertinent for real world data.", "relation_to_broader_scientific_literature": "The work exists within the literature of approximate/amortized Bayesian inference. It also overlaps with the literature on Bayesian inverse problems. The authors discuss approximate methods in Bayesian inference, and discuss TabPFN in detail. The authors could provide better motivation for why their approach may be useful than they currently do.", "essential_references_not_discussed": "The authors may find \"General-purpose in-context learning by meta-learning transformers\" interesting as another example of ICL on (partially) synthetically generated data, but citing this paper isn't critical.", "other_strengths_and_weaknesses": "Overall I found the paper to be fairly well presented, although the authors could mention what their approach is (transformer encoder + flow matching) earlier in the paper for people who are skimming the paper.", "other_comments_or_suggestions": "None", "questions_for_authors": "Feel free to respond to the weaknesses I stated, but I have no other questions.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1ZL5MOYCuP", "reviewer": "Reviewer_Y4eg", "summary": "The authors leverage transformers architecture to amortize Bayesian posterior estimation based on training data / observations fed in-context to the model. They conduct analysis on generalized linear models, factor analysis and mixture models and highlight that the proposed method discovers the true posterior distribution quite well, measured through classifier 2-sample test (C2ST), maximum mean discrepancy (MMD) and 2-Wasserstein ($\\mathcal{W}_2$) metrics. Training such an amortized model is done through the lens of forward-KL minimization, which becomes tractable when amortized (i.e. the outer expectation w.r.t observations) and the parameterization of the approximate density is accomplished using flow matching. The paper shows that across a suite of synthetic and real world tasks, the proposed neural posterior estimation method stays competitive and outperforms reverse KL Variational inference when the metrics are computed through MCMC samples as proxy.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the valuable suggestions — we extended the experiments as proposed and now prominently highlight Mittal et al. as particularly relevant related work.\n\n> The key contributions of the work lie in their experimental setup [...].\n\nWe fully agree that our core contribution lies in the experimental setup. However, we also (a) introduce a new architecture combining the PFN-type encoder and a modified version of diffusion transformers, (b) point out a sufficient condition that is central for many related amortized inference setups but has, to the best of our knowledge, not been made explicit in this form. Unlike your references [2] and [3], we (c) directly evaluate the quality of the posterior distribution. \n\n> Training such an amortized model is done through the lens of forward-KL minimization \n\nPlease note that we do not use a KL-based objective, but train the in-context learner within the Flow Matching framework. \n\n> The authors claim that their method yields samples from the posterior distribution without parameter updates or parametric assumptions about the posterior. \n\nWe use the term “nonparametric” in the sense that we make very weak assumptions about the underlying distribution of the data.\n\nIn particular, we obtain a posterior distribution only implicitly defined through flow matching.\n\n> The authors do not provide a general framework to analyze the circumstances that enable learning purely through samples from the joint.\n\nWe use the terminology “general framework” to refer to (a) our architectural framework in combination with Flow Matching, which arguably is a generic framework for learning distributions, and (b) Proposition (1) that, although a direct corollary of the law of total expectations, is to the best of our knowledge the first sufficient condition explicitly presented to describe  Neural Posterior Estimation [1], PFNs, Score Matching Posterior Estimation [2] and Flow Matching Posterior Estimation [3]. \n\n\n>In addition, it is claimed in Section 3 that the proposed method does not suffer from overly or insufficiently flexible distribution assumptions as in VI. This is untrue, as one could also leverage continuous-time methods for reverse KL minimization, yielding the same family of distributions as the proposed work.\n\nA known benefit of in-context learners in the form of PFNs, and especially TabPFNs, is that they can flexibly adapt to the complexity of the presented data and thus do not require extensive hyperparameter tuning. This is what we mean by that claim. We will clarify this in the manuscript. \n\n> The paper does not contain any evaluation based on predictive performance. \n\nPlease refer to our [new results](https://anonymous.4open.science/r/Extra-B820/Pred_Performance.pdf) regarding the predictive performance in the GLM scenarios. Those results show that forward-KL based VI and in particular MAP solutions perform strongly in terms of predictive performance, similar to results first discovered by your references [2] and [3]. \n\n> [...] it would be useful to get a comparative analysis in terms of predictive performance from TabPFN.\n\nPlease see the previous answer for our new results that also include a comparison to TabPFN. \n\n> [...] An identical formulation can be framed from the reverse KL minimization perspective, and should be considered as one of the baselines. \n\nWe also conducted [comprehensive experiments](https://anonymous.4open.science/r/Extra-B820/FwdKL_Gaussian.pdf) investigating the performance of using a Gaussian approximation in the forward-KL setup.\n\n>Details regarding the dimensionality of tasks considered is missing and is extremely relevant. \n\nPlease refer to our [new results](https://anonymous.4open.science/r/Extra-B820/Dim_Ablation.pdf). We will also rigorously adapt the claims and limitations section of our paper in light of the new results here. \n\n> Unfortunately, the descriptions of the experimental designs were unclear and / or potentially riddled with typos. \n\nWe thank the reviewer for pointing out those typos! We fixed all the issues you mentioned. \n\n> The authors should consider citing [...]\nWe thank the reviewer for pointing out those missing and clearly important references. We will include a discussion of those two papers and think that especially the work by Mittal et al. will nicely complement our findings while analyzing further interesting aspects of in-context learning. \n> The authors claim in line 146 that VI problems typically consider a simplified factorization of the variational density, which is not always true. \nThank you for this remark. We will expand the section on related work on amortized inference to include your comment. \n\n### Questions For Authors\nWe thank the reviewer for the very helpful suggestions regarding different conventions in our notation and for pointing out typos. We will follow the reviewer's recommendations.", "claims_and_evidence": "While the authors claim that they propose a general-purpose machinery for Bayesian posterior estimation, I will consider the contribution of the work as the application of existing approaches towards a suite of interesting and useful tasks, with a focus on studying its generalization capability to real world tasks. This is because the framework is quite similar to some of the existing works [1, 2] and is primarily the application of neural posterior estimation in simulation-based inference on the class of tasks described in the paper using flow matching and the transformer architecture.\n\nIn addition, the following claims made by the authors in their contributions section are not well supported -\n\n- The authors claim that their method yields samples from the posterior distribution without parameter updates or parametric assumptions about the posterior. This is untrue as they do make parametric assumptions about the posterior, in the sense that the posterior is modeled through an ordinary differential equation with explicit parameters. \n\n- The authors *do not provide a general framework to* analyze the circumstances that enable learning $P^{z|x}$ purely through samples from the joint. In particular, they do not provide any theory on when such a density can be reliably learned.\n\nIn addition, it is claimed in Section 3 that the proposed method does not suffer from overly or insufficiently flexible distribution assumptions as in VI. This is untrue, as one could also leverage continuous-time methods for reverse KL minimization, yielding the same family of distributions as the proposed work.\n\n[1] Wildberger, Jonas, et al. \"Flow matching for scalable simulation-based inference.\" Advances in Neural Information Processing Systems 36 (2023): 16837-16864.\n\n[2] Mittal, Sarthak, et al. \"Exploring exchangeable dataset amortization for bayesian posterior inference.\" ICML 2023 Workshop on Structured Probabilistic Inference {\\&} Generative Modeling. 2023.", "methods_and_evaluation_criteria": "Since the proposed method is aimed at posterior estimation, the benchmark datasets and evaluation criteria considered make sense and are relevant. However, there are a number of limitations in the evaluation procedure, which I detail below\n\n- The paper does not contain any evaluation based on predictive performance. In particular, for supervised learning problems (i.e. GLM experiments) at least, it is imperative to also provide performance metrics (e.g. $l_2$ loss or the likelihood) and compare it to directly learning the linear method through gradient descent or estimating the posterior through MCMC / VI. \n\n- Given that the work claims to use the data generating mechanism for TabPFN [1], it would be useful to get a comparative analysis in terms of predictive performance from TabPFN.\n\n- For the VI-style baselines, the authors consider a non-amortized setup as baseline while using an amortized model as the proposed approach. An identical formulation can be framed from the reverse KL minimization perspective, and should be considered as one of the baselines. In particular, [2, 3] show that reverse KL often outperforms forward KL methods in terms of predictive metrics especially for high-dimensional tasks.\n\n- Details regarding the dimensionality of tasks considered is missing and is extremely relevant. The authors should also provide some form of analysis into how their performance varies as a function of dimensionality of the task, at least for the GLM experiments. This is because with increasing dimensionality, it becomes harder and harder to maintain similarity to real-world tasks in the simulated data, and could lead to worsening performance on them.\n\n[1] Hollmann, Noah, et al. \"Tabpfn: A transformer that solves small tabular classification problems in a second.\" arXiv preprint arXiv:2207.01848 (2022).\n\n[2] Mittal, Sarthak, et al. \"In-Context Parametric Inference: Point or Distribution Estimators?.\" arXiv preprint arXiv:2502.11617 (2025).\n\n[3] Mittal, Sarthak, et al. \"Amortized In-Context Bayesian Posterior Estimation.\" arXiv preprint arXiv:2502.06601 (2025).", "theoretical_claims": "The authors do not make any theoretical claims, except for Proposition 1 which is well known in literature and is the basis for neural posterior estimation methods within the framework of simulation-based inference. Please note that not making any theoretical claims is not a critique of the paper.", "experimental_designs_or_analyses": "Unfortunately, the descriptions of the experimental designs were unclear and / or potentially riddled with typos. Appendix A.1 in the write-up often confuses between $\\mathbf{u}$, $\\mathbf{z}$ and $\\mathbf{x}$. For example, the authors claim that $\\mathbf{x} := (\\mathbf{z}, y)$ while also claiming that $\\mathbf{z} = \\mathbf{\\beta}$, which should not be the case. Similarly, for Factor Analysis in Algorithm 3, the authors do not include $\\mathbf{W_i}, \\mathbf{\\psi_i}, \\mathbf{\\mu_i}$ in the latent. Additionally, line 6 should be $\\mathbf{z}_{i,j}$.", "relation_to_broader_scientific_literature": "The key contributions of the work lie in their experimental setup, as prior works have leveraged transformers for amortization, forward KL minimization as the training signal for posterior estimation and flow matching as the parameterization of density. I think the work is well positioned and shows interesting experiments and outcomes, but needs more rigorous testing and benchmarking, as well as additional baselines and ablations.", "essential_references_not_discussed": "The authors should consider citing [1] which essentially looks at the problem of amortized posterior inference in known likelihood models like linear models, Bayesian neural networks and Gaussian Mixture Models through the lens of both forward and reverse KL methods using a Gaussian distribution or normalizing flows as the approximate density.\n\nThey should also cite [2] which, among other SBI methods, describes how forward KL can be used as a measure of divergence for training amortized posterior estimators.\n\n[1] Mittal, Sarthak, et al. \"Exploring exchangeable dataset amortization for bayesian posterior inference.\" ICML 2023 Workshop on Structured Probabilistic Inference {&} Generative Modeling. 2023.\n\n[2] Radev, Stefan T., et al. \"BayesFlow: Learning complex stochastic models with invertible neural networks.\" IEEE transactions on neural networks and learning systems 33.4 (2020): 1452-1466.", "other_strengths_and_weaknesses": "**Strengths**\n\n- The problem considered in this work is important and relevant and the authors conduct analysis on a variety of synthetic and real-world tasks which is useful.\n- The work successfully shows that through posterior estimation metrics, amortization for posterior estimation uncovers the true posterior quite well.\n\n**Weaknesses**\n\n- The novelty of the work lies solely in their experimental results as the use of forward KL minimization, flow matching and amortization on transformers has been already studied considerably in related works.\n- Benchmarking and evaluation of the method is limited as the authors do not provide predictive metrics, or compare to amortized VI methods. \n- The work provides limited ablations; it could consider benchmarking the methods with different assumptions about the modeling distribution: flow matching, score-based diffusion, Gaussian approximation and discrete normalizing flows. Only two of the mentioned methods are tested.\n- The writing needs more work; currently the mathematical formalism in the main paper and the data generation outline in the Appendix are not consistent.", "other_comments_or_suggestions": "- The authors claim in line 146 that VI problems typically consider a simplified factorization of the variational density, which is not always true. The works cited (e.g. VAEs) perform this approximation as they consider latent variables corresponding to every observation (i.e. only local latent variables) while other works like Neural Processes [1,2] consider global latent variables which are shared across observations for a particular dataset. The authors should make it clear that they do not provide a more general assumption.\n\n- The writing of the draft can be significantly improved. The authors often use the notation of $\\mathbf{x}$ to either mean a dataset (e.g. line 187) or a meta-dataset (dataset of datasets; line 177).\n\n- The authors claim that they do not make any assumptions about the latents $z \\in \\mathcal{Z}$ in footnote 2, regarding its decomposition into $z_i$. However, immediately afterwards in line 189 they consider a mapping $f_0$ which maps $\\chi$ to $\\mathcal{M}(\\mathcal{Z})$ implying that $x_i$ maps to a density over $z$, which does imply that $z$ is local to each dataset.\n\n- In light of the above two points, I would highly recommend the authors to provide a clearer write-up in their draft and make sure that the mathematical objects that they are working with are consistent. In particular, they can either describe everything in the general framework of $z \\in \\mathcal{Z}$ or work under the factorization $P^{z_i | x_i}$, but they should be consistent.\n\n[1] Garnelo, Marta, et al. \"Neural processes.\" arXiv preprint arXiv:1807.01622 (2018).\n\n[2] Kim, Hyunjik, et al. \"Attentive neural processes.\" arXiv preprint arXiv:1901.05761 (2019).", "questions_for_authors": "I found the following details to be incorrect, could the authors clarify if I am missing something?\n- Equation 5 appears to be wrong. The objective for learning the vector field should be $z^{(1)} - \\omega z^{(0)}$ (note the lack of division).\n- Equation 6 needs to be pre-multiplied by $\\frac{1}{N}$.\n- Please use $\\gamma_t(z^{(1)}, z^{(0)})$ (functional notation) as it is not a density, hence should not use the conditional operator.\n- Line 300 introduces an additional notation of $\\psi$ which is exactly same as the $\\gamma$ considered before.\n- Have the authors considered using a nonlinear neural network for computing C2ST? Could the authors do so and highlight if they get similar performance?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "x3kGWqdQRs", "reviewer": "Reviewer_RMVg", "summary": "The paper proposes an in-context learning (ICL) approach for performing Bayesian inference over three classes of models: Generalized Linear Models (GLMs), Gaussian Mixture Models, and Factor Analysis (FA). The paper shows that their ICL method can produce similar posterior samples to Hamiltonian Monte Carlo (HMC).", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to read our manuscript and for providing detailed feedback.\n\n> Default hyperparameters are used for the VI approaches according to the appendix (no hyperparameter optimization) \n\nWe would like to kindly point out that we investigate the role of the learning rate, which is a crucial hyperparameter for the VI methods, in appendix J of the manuscript. We furthermore decided to use standard “out-of-the-box” variational methods as baselines to allow for a direct with the in-context learner, which cannot even adapt its weights for each specific dataset. \nFurthermore, we include ablations in our paper (please see Appendix D) where we compare against strong diffusion-based baselines. \n\n> The key weakness that is preventing me from providing a higher score is in the identification of the novelty of the work in comparison to Müller et al. \n\nIn their paper on PFNs, Müller et al. [1] focus exclusively on **univariate** posterior predictive distributions. Our approach, on the other hand, targets multivariate posteriors of latent variables. The key difference is thus in a scalar-valued distribution for PFNs versus a multivariate distribution in our approach. This is a fundamental difference comparable in scope to the difference between supervised learning (PFNs) and unsupervised learning (our method). \n\nThis difference manifests itself in various aspects that include:\n\n- The tasks for which inference is learned in context:\n\n    - Müller et al: Regression with Bayesian neural networks, regression with Gaussian processes, and classification of hand-written digits.\n    - Ours: Full Bayesian inference regarding the latent variables of generalized linear models, factor analysis, and Gaussian mixture models. \n- The framework to learn the distributions\n    - Müller et al: Discretization of the distribution of interest and a cross-entropy loss.\n    - Ours: Flow matching.\n\n- The architecture\n    - Müller et al: A slightly modified standard transformer.\n    - Ours: A novel setup combining a  PFN-style transformer-encoder together with a diffusion transformer decoder.\n\n- The evaluation\n    - Müller et al: Predictive metrics and metrics for calibration\n    - Ours: Comparison of posterior samples from our method and the samples from HMC using measures for the similarity of distributions (MMD, C2ST, and W2 metrics)\n\n> how many synthetic datasets were needed to achieve the reported performances.\n\nThank you for this question. We train on 37.5 million synthetic datasets, test on 30 million datasets, and validate on 7.5 million datasets. In terms of computational resources, each of our training runs takes at most three hours on an A100 graphics card. To put that into relation, for example, TabPFN version 1 is trained for 20 hours on 8 GPUs (Nvidia RTX 2080 Ti) and TabPFN version 2 even for 2 weeks on 8 Nvidia RTX 2080 Ti.\n\nBesides the existing detailed explanation in sections A and E of the appendix, we will include this information in the paper and thank the reviewer again for this suggestion. \n\n> The experiments on the OOD performance in the appendix are interesting and probably should be promoted to the main paper. I would suggest thinking of a new way of presenting these experiments though. Perhaps as a plot where the y-axis is the metric, and the x-axis is the KL-divergence between the training and test data distributions.\n\nThank you for your comment. We agree that the OOD performance is an interesting aspect worth highlighting in the main body of our paper. Since we focus on controlled setups (GLM, FA, GMM) to enable comparisons to baselines from standard Bayesian inference (HMC and VI), limited OOD generalization is expected — a trade-off we confirm in Appendix E. We will make this clearer in the main paper.\n\nWe also followed your great recommendation and created the [following plot](https://anonymous.4open.science/r/Extra-B820/plot_ood.pdf) which we will add to the new section in the main paper explaining the situation in terms of OOD performance.\n\n## References\n\n[1] Müller, Samuel, et al. \"Transformers Can Do Bayesian Inference.\" Eleventh International Conference on Learning Representations.\n\n[2] Lipman, Yaron, et al. \"Flow Matching for Generative Modeling.\" The Eleventh International Conference on Learning Representations.", "claims_and_evidence": "* Claim 1: ICL yields posterior samples that are very similar to HMC:\n    * This does seem to be the case from the experimental results. Looking specifically at distributional metrics for measuring similarity between samples.\n* Claim 2: ICL samples are preferred over popular VI techniques across the experiments:\n\t* Default hyperparameters are used for the VI approaches according to the appendix (no hyperparameter optimization) and the VI approaches do not seem to be state-of-the-art in VI.", "methods_and_evaluation_criteria": "* The benchmarks and evaluation criteria make sense.", "theoretical_claims": "* This does not seem to be applicable here. I do not believe any theoretical claims are made.", "experimental_designs_or_analyses": "* See related comments in strengths and weaknesses.", "relation_to_broader_scientific_literature": "* The paper is focused on ICL for performing Bayesian inference. The results show there is potential to leverage transformer-based architectures for performing Bayesian inference. What is less clear is how this work’s conclusions differ from the paper “Transformers can do Bayesian inference”.", "essential_references_not_discussed": "* Not that I am aware.", "other_strengths_and_weaknesses": "### Strengths:\n* The paper is well-written and the presentation (Figure 1 etc.) is strong.\n* The results seem to support the claims.\n### Weaknesses\n* The key weakness that is preventing me from providing a higher score is in the identification of the novelty of the work in comparison to Müller et al. I think the main difference (novelty) is in the architecture (Figure 2). The loss function appears to be comparable to Müller et al. As such, it would seem like comparing to prior-data fitted networks from that paper, would really help show if the proposed ICL approach is superior. \n* Another component of the Müller et al. paper that is missing from this paper is a reference to how many synthetic datasets were needed to achieve the reported performances. For example, it would be good to know how many synthetic data sets were needed to get to the current performance, and what the behaviour of the performance is as the number of datasets trained on are increased.", "other_comments_or_suggestions": "* I could not find details of the size of the data trained on for each class of model. (If it is hiding in the appendix, I apologise for missing it.)\n* The experiments on the OOD performance in the appendix are interesting and probably should be promoted to the main paper. I would suggest thinking of a new way of presenting these experiments though. Perhaps as a plot where the y-axis is the metric, and the x-axis is the KL-divergence between the training and test data distributions. (This is just a suggestion.)", "questions_for_authors": "In addition to responding to the comments regarding the weaknesses:\n* Is it the case that the architecture is the main novelty of the paper?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lnjTKUXybm", "reviewer": "Reviewer_qsBb", "summary": "The paper introduces an innovative approach to full Bayesian inference using in-context learning (ICL) with transformers. By leveraging ideas from continuous normalizing flows and flow matching, the authors propose a framework that learns to approximate the posterior distribution P(z|x) directly from synthetic samples drawn from the joint distribution. Evaluations on generalized linear models (GLMs), factor analysis, and Gaussian mixture models demonstrate that the ICL method produces posterior samples comparable to those obtained via Hamiltonian Monte Carlo and state‐of‐the‐art variational inference techniques. The work is well-motivated and offers a promising new direction by uniting meta-learning ideas with Bayesian inference.\n\n**update after rebuttal**\n\nI thank the authors for their responses. Although it is understandable that not many new empirical evidence can be provided due to the restricted length of rebuttal this year, the empirical evaluation remains the weakness of this paper. Hence I will maintain my current score. I disagree that image datasets are less relevant, since there are many works (e.g, Bayesian neural networks, and VAE related works) trying expand Bayesian inference to the image domain. I think potential evaluations on image datasets (with Bayesian versions of vision transformers) would greatly strengthen the paper in the future.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to read our manuscript and for providing detailed feedback.\n\n>More advanced and large datasets like image datasets should be used to verify the effectiveness of the proposed frameworks. \n\nPlease note that, to the best of our knowledge, our paper presents the first thorough investigation of in-context learning for full Bayesian inference. We think that tabular data is a very natural domain for full Bayesian inference and in particular for the latent variable models considered in our work. While fully Bayesian methods for image data exist, they are arguably substantially less prominent. Furthermore, we believe that the inherent heterogeneity of the 17 considered real-world tabular datasets, spanning domains such as superconductors, environmental pollution, or wine quality, is well suited to provide a challenging and diverse benchmark. \n\nFollowing your comment, we now also include an [ablation study](https://anonymous.4open.science/r/Extra-B820/Dim_Ablation.pdf) that investigates the effect of the dimensionality of the used data.\n\n> Also, some image generation tasks (e.g., on MNIST) can be included to test the quality of the learned posterior.\n\nWe would like to argue that image generation is usually not considered to be within the scope of full Bayesian inference, which we investigate in this paper. \n\n> Somehow, what the authors claim is related to density estimation methods. Experiments on UCI datasets and related density estimation methods would better support the claims.\n\nWe would like to emphasize that in this work, we consider Bayesian inference as the task of providing samples from the posterior. This is in scope of what MCMC methods do and thus also importantly allows to compare our sampling-based method and the VI baselines to HMC as a gold standard. Even though Bayesian inference can also be accomplished via density estimation, we would argue that this is a different task. \n\n> the authors only verify the comparability to existing optimization methods, but yet its practicability to different downstream tasks.\n\nWe thank the reviewer for this comment. Exclusively comparing posterior samples themselves is, however, a common practice in amortized (simulation-based) inference [1]. We would also like to highlight that, in practical applications, the results of the latent variable models we study are arguably mostly analyzed in isolation rather than being used directly for downstream tasks, which motivates our choice of evaluation procedure.\n\nHowever, following your recommendation, we now include results for [new experiments](https://anonymous.4open.science/r/Extra-B820/Pred_Performance.pdf) where we use the posterior samples for the GLMs case for the purpose of prediction.\n\n> Perhaps the authors need to include the key assumptions by these works here for better clarity.\n\nThank you for this point. We will include the assumptions underlying flow matching in our paper besides just referring to the literature. \n\n> A comparison to related density estimation methods is missing, e.g., Salazar, Sebastian. \"VaRT: variational regression trees.\" Advances in Neural Information Processing Systems 36 (2023): 45681-45693.\n\nWe will add a more extensive discussion of density estimation methods, including the aforementioned paper, to the revised version of the manuscript and how they relate to our approach. \n\n> Have the authors explored scenarios where the generative model does not perfectly capture the true data distribution?\n\nPlease refer to Appendix H for experimental results regarding the OOD performance of our approach and to Appendix B for a more detailed discussion on model misspecification. We will expand our explanations of the situation regarding OOD performance in a new section in the revised version of the manuscript. We created a [new plot](https://anonymous.4open.science/r/Extra-B820/plot_ood.pdf) for this purpose. \n\n## References\n\n[1] Lueckmann, Jan-Matthis, et al. \"Benchmarking simulation-based inference.\" AISTATS 2021.", "claims_and_evidence": "The central claim that transformers can learn full Bayesian inference in context is well-supported by extensive experiments on GLMs, factor analysis, and GMMs—showing that the posterior samples produced by the ICL approach closely match those obtained via HMC and often outperform several VI methods on synthetic and limited real-world datasets.\n\nHowever, the claims on large-scale training is not well-backed by experiments. Although real-world data are used, they remain at tabular datasets. More advanced and large datasets like image datasets should be used to verify the effectiveness of the proposed frameworks. Also, some image generation tasks (e.g., on MNIST) can be included to test the quality of the learned posterior. \n\nSomehow, what the authors claim is related to density estimation methods. Experiments on UCI datasets and related density estimation methods would better support the claims.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria generally make sense for the problem at hand. The authors introduce a transformer-based in-context learning approach that leverages continuous normalizing flows and flow matching to approximate full posterior distributions. This method is evaluated using well-established metrics—C2ST, MMD, and Wasserstein—which are standard for comparing how close the inferred distributions are to those obtained by gold-standard methods like HMC. Furthermore, the experiments span a range of models (GLMs, factor analysis, and GMMs) and use both synthetic and curated real-world tabular datasets, which is appropriate for demonstrating feasibility across different Bayesian inference scenarios.\n\nHowever, while the chosen benchmarks and evaluation criteria are solid for controlled experiments, the reliance on synthetic data and moderate-scale real-world datasets raises questions about the scalability and robustness of the method when dealing with more complex, high-dimensional tasks. And the baselines compared are insufficient - the authors only verify the comparability to existing optimization methods, but yet its practicability to different downstream tasks. Somehow what the authors are claiming are related to density estimation methods. Experiments on UCI datasets and related density estimation methods would better support the claims.", "theoretical_claims": "Many theoretical results are only outlined rather than given in full detail. The results heavily follow previous work which are well-verified. Perhaps the authors need to include the key assumptions by these works here for better clarity.", "experimental_designs_or_analyses": "The experiments and designs are generally adequate and sound.", "relation_to_broader_scientific_literature": "As mentioned before, I think the task that the authors are more like posterior density estimation. In this sense, including comparisons to density estimation benchmark would be necessary (e.g., UCI) and even image generation tasks for higher dimensional cases.", "essential_references_not_discussed": "A comparison to related density estimation methods is missing, e.g., Salazar, Sebastian. \"VaRT: variational regression trees.\" Advances in Neural Information Processing Systems 36 (2023): 45681-45693.", "other_strengths_and_weaknesses": "Strength\n* Novelty: The paper is novel. applies transformer-based in-context learning to perform full Bayesian inference—a departure from conventional MCMC or VI methods. \n* Flexibility: The proposed approach can handle multivariate and complex posterior distributions, showing robust performance even in cases with non-standard posterior shapes (e.g., skewed or multi-modal distributions). \n* Empirical Results: Comparative evaluations indicate that the ICL approach outperforms traditional VI methods in capturing the full posterior, particularly in synthetic experiments.\n\nWeaknesses\n\n* My largest concern is the scalability issues. The authors only adopt small datasets (e.g., simulated and tabular datasets by Grinsztajn metal.). Authors should test the capability of the proposed transformer on image datasets (e.g., MNIST or ImageNet) to verify their claims on scalability.\n\n*Although the comprehensiveness of evaluation in the suggested settings, I think comparisons to density estimation methods (e.g., UCI) and image generation (for validating the quality of the posterior distribution) would be necessary. The current empirical evaluation is less sufficient.", "other_comments_or_suggestions": "N.A.", "questions_for_authors": "In practical settings where model misspecification is a concern, how robust is the ICL approach? Have the authors explored scenarios where the generative model does not perfectly capture the true data distribution?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper presents a method for using in-context learning in transformer based models to perform full Bayesian inference.  The authors use the transformer to learn a conditional continuous normalizing flow that parameterizes a vector field of an ordinary differential equation (ODE).  An ODE solver can then be used to sample from some simple base distribution to the target posterior of interest.  The paper received borderline but positive reviews (3, 2, 3, 3).  The reviewers found the paper well-written, well structured and mathematically sound.  They found the approach novel and the experimental results presented strong.  In terms of weaknesses, the reviewers asked for stronger baselines, better metrics and were concerned about scalability.  There was a bit of a sense that reviewers found it interesting that one could do Bayesian inference this way, but they didn't necessarily get why one would do Bayesian inference this way.  Perhaps the authors could have motivated the approach a bit better and given some insight into how this would be used in practice over performing standard Bayesian inference (e.g. MCMC) using the actual model.   There was some concern that the approach isn't scalable, which adds to the question of why one would want to use a giant transformer-based model to approximate a simpler model in-context.  Nevertheless, the paper is insightful and technically interesting and seems like a novel and fun way to co-opt in-context learning to do something more sophisticated.
Decision: Accept (poster)