Transformers Can Do Bayesian Inference
Authors: Samuel Müller
Noah Hollmann
Sebastian Pineda Arango
Josif Grabocka
Frank Hutter
Venue: ICLR 2022 Poster (2022)

Abstract: Currently, it is hard to reap the benefits of deep learning for Bayesian methods, which allow the explicit specification of prior knowledge and accurately capture model uncertainty. We present Prior-Data Fitted Networks (PFNs). PFNs leverage large-scale machine learning techniques to approximate a large set of posteriors. The only requirement for PFNs to work is the ability to sample from a prior distribution over supervised learning tasks (or functions). Our method restates the objective of posterior approximation as a supervised classification problem with a set-valued input: it repeatedly draws a task (or function) from the prior, draws a set of data points and their labels from it, masks one of the labels and learns to make probabilistic predictions for it based on the set-valued input of the rest of the data points. Presented with a set of samples from a new supervised learning task as input, PFNs make probabilistic predictions for arbitrary other data points in a single forward propagation, having learned to approximate Bayesian inference. We demonstrate that PFNs can near-perfectly mimic Gaussian processes and also enable efficient Bayesian inference for intractable problems, with over 200-fold speedups in multiple setups compared to current methods. We obtain strong results in very diverse areas such as Gaussian process regression, Bayesian neural networks, classification for small tabular data sets, and few-shot image classification, demonstrating the generality of PFNs. Code and trained PFNs are released at https://github.com/automl/TransformersCanDoBayesianInference.

Published as a conference paper at ICLR 2022
TRANSFORMERS CAN Do BAYESIAN INFERENCE
Samuel Müller', Noah Hollmann', Sebastian Pineda', , Josif Grabocka', Frank Hutter 'University of Freiburg, "Charité Berlin, 'Bosch Center for Artificial Intelligence Correspondence to Samuel Müller: mue muellesa@ce uni faii-freburgg... de
ABSTRACT
Currently, it is hard to reap the benefits of deep learning for Bayesian methods, which allow the explicit specification of prior knowledge and accurately capture model uncertainty. We present Prior-Data Fitted Networks (PFNs). PFNs leverage large-scale machine learning techniques to approximate a large set of posteriors. The only requirement for PFNs work is the ability sample from prior distribu- tion over supervised learning tasks (or functions). Our method restates the objective of posterior approximation as a supervised classification problem with set-valued input: it repeatedly draws a task (or function) from the prior, draws a set of data points and their labels from it, masks one of the labels and learns to make proba- bilistic predictions for it based on the set-valued input of the rest of the data points. Presented with a of samples from new supervised learning task as input, PFNs make probabilistic predictions for arbitrary other data points in single forward propagation, having learned to approximate Bayesian inference. We demonstrate that PFNs can wear-perfectly mimic Gaussian processes and also enable efficient Bayesian inference for intractable problems, with over 200-fold speedups in multi- ple setups compared to current methods. We obtain strong results in very diverse areas such as Gaussian process regression, Bayesian neural networks, classification for small tabular data sets, and few-shot image classification, demonstrating the generality of PFNs. Code and trained PFNs are released at wtthtps://// / / ccom automl Trrans www. an In
INTRODUCTION
In the last decade, supervised machine learning (ML) methods using deep learning architectures have made substantial progress on machine learning tasks where a large amount of training data is available (Vaswani et al. 2017 He et al., 2016: Krizhevsky et al. 2012). A very important problem in MI. thus to transfer these successes to smaller-scale setups with less data available. this paper, propose way to build models that approximate posteriors with flexible and replaceable priors using deep learning models. makes specifying a prior as simple as defining a sampling scheme of supervised learning tasks.
Train the PFN by minimizing
Sample prior datasets D(t) ~ P(D)
Σ K log go (yeest Feest
i=
(8) D
train)
D = -itiin U (x(ted )(er))
PPN with parameters 0
Bayesian inference via the trained PFN, with the actual training data and a test point as input:
D(K D (K)) U (1 b(e)))
Actual training datasct and test input
qe (Wrest Ftests Divein) R p(yteat Ftests Dtrain)
D (traiin Itest)
Figure 1: A visualization of Prior-Date Fitted Networks (PFNs). We sample datasets from prior and fit PFN on hold-out examples of these datasets. Given an actual dataset, we feed it and a test point to the PFN and obtain approximation to Bayesian inference in single forward propagation.

---

Published as a conference paper at ICLR 2022
While the success of deep learning on large datasets can be attributed to the capacity of neural networks to approximate any function, there is still a need for encoding prior knowledge, for example through model architecture (c.g., Convolutional Neural Networks (LeCun et al. 1989] regularizers (e.g., data augmentations (Hendryeks et al., 2019; Cubuk et al. 2020)). Otherwise, the no free lunch theorems show that there no good methods solve the class of prediction problems (Wolpert & Macready] 1997). Thus, a large number of specialized algorithms have been developed for different small-scale tasks (LeCun al. 1989; Kadra al. 2021; Chen & Guestrin 2016). Encoding prior information into a machine learning model can, however, be challenging.
well-defined way to bias a model is to use Bayesian inference. The foundation of Bayesian inference is an assumption on the distribution of the data to appear in a real-world application. This assumption gives rise to a prior belief over the probability for the data to follow a particular model. One might, e.g., implement& prior, encoding the data as created by neural network (Bayesian Neural Networks, (MacKay) 1992)), by a polynomial, the likelihood of Gaussian mixture or random code in pre-defined programming language (Solomonofi 1997). Using Bayesian inference for prediction in supervised learning has the advantages that (i) it has a theoretical foundation that makes it valid in setups where the prior p(t) fits; (ii) it can thus better account for the actual likelihood of different events; (iii) it is well calibrated and (iv) it is interpretable as the prior describes the expectations of the model. However, retrieving the posterior predictive distribution for given prior is intractable in most cases (Blei et al. 2017: MacKay 1992).
Figure I Noullines Prior-Data Fitted Networks (PFNs), for approximating such Bayesian models. We assume a given representative prior distribution over supervised learning tasks (or functions), which provides our inductive bias. To train PFN, we use supervised learning with set-valued inputs representing entire datasets: we repeatedly sample a meta-train task (or function) from the given prior, draw a set of data points and their labels from it, mask one of the labels and learn to make probabilistic predictions for it based on the set-valued input of the rest of the data points. Given an actual real-world dataset, we feed together with a test point as inputs to the PFN and that outputs its prediction distribution for the test point, conditioned on the dataset. As we will demonstrate, this distribution approximates exact Bayesian posterior prediction. We refer to this step as (Bayesian) inference, opposed to the training of the PFN itself.
Our PFNs thus allow us to approximate the posterior predictive distribution for any prior from which are able to sample data. This is very weak requirement compared to the standard assumptions of other approximations for Bayesian inference (Hoffman et al. 2014 2013) Jordan et al. 1999). This allows simple approximation of a large set of priors, including priors that very hard to approximate with currently available tools. We make the following contributions:
We present architectural changes to successfully use Transformers for posterior predictive distribution (PPD) approximation, including novel predictive distribution for regression tasks. The proposed method is simple, cheap and generally applicable to a large set of priors.
We demonstrate that PFNs can approximate the PPD of Gaussian processes and Bayesian neural networks (BNNs) orders of magnitude faster than MCMC with NUTS SVI with ba-bkkprop (Blundell et al. 2015).
We demonstrate that PFNs can have an impact on real-world tasks. (i) We implement BNNs with priors architectures PFNs that enable tuning free predictions in single forward pass and outperform all baselines on a large benchmark of small tabular datasets. (ii) Additionally, we show that simple handwriting priors enable few-shot learning on Omniglot (Lake et al. 2015)
2 BACKGROUND
This paper concerns the use of a large amount of data to later yield strong performance on tasks for which only small datasets are available. In multiple machine learning setups, one can see successes based on this approach, such as fine-tuning unsupervised models image data (Chen & He 1021
Chen et al. (020)) or large language corpora (Devlin et al. 2018). Another way to transfer know ledge from large corpora is to design prompts for language models (Wang & Komatsuzaki, 2021 Brown jet al. 2020). While we use a large amount of data during training, unlike the other approaches, all the data use is generated artificially.
2

---

Published as a conference paper at ICLR 2022
In meta-learning, or learning-to-lean (Hochreiter al. 2001 Finn et al. 2017a). generalize learning methods from a training set of datasets to a validation set of datasets. Similar to approach, there is a branch of research on building models that can learn to learn in single forward pass (Santoro et al. 2016; Hochreiter et al. 2001 Gordon et al. 2019). More recently (Conditional) Neural Processes (Garnelo et al.] 2018a b Kim et al. 2018) were proposed for meta-learning. These methods focus on architectural constructions as stochastic processes. The difference, besides the dissimilar model architectures and dissimilar prediction distributions, to previous meta-learning work is that we set up learning problems from which we sample artificial, on-the-fly on-the-fly-geneee fly-generated datasets such that we can show that our method does not only learn learn, but that it learns to perform Bayesian inference.
tries to
In this work, we are interested in Bayesian posterior prediction for supervised learning problems, i.e. we model the output y for new input based on supervised dataset of arbitrary size n, D = ( )=) where y, the output for x;, To address the supervised learning problem in a Bayesian way, we consider prior p(t) over the latent variable t, the task. We can now use Bayes' Theorem to define the posterior p(t)D). The crucial distribution for prediction, the posterior predictive
distribution (PPD), can then be inferred
plyla,, D) = J. Plulz, plull,,)p(((D)) t)p(t)D).
In some cases, one can arrive at the PPD ply|2z D) in closed form (e.g., for Gaussian Processes, (Rasmussen & Williams 2005), but in most cases the PPD can only be approximated. Traditionally, one approximates the posterior p() and based on this approximation infers an approximation of the PPD. To approximate the posterior, there are two classes of prominent methods: i) Markov Chain Monte Carlo (MCMC) (Neal 1996; Andrieu et al. 2003 Welling & Tch (011) methods, like NUTS (Hoffman al., 2014), allow accurate but sometimes very slow approximations of the
posterior; and i1) Variational Inference (VI) (Jordan et al.
1999 Wainwright & Jordan 2008 Hoffman
al. 2013) methods approximate the posterior by a tractable distribution, such as a factorized normal distribution.
One more line of research to be mentioned is amortized iimulation-based inference (Cranmer et al. 2020 Chan et al. 2018), in particular neural posterior estimation (Lueckmann et al., 2021
Papamakarios & Murray 2016). The objective here is to find a model that approximates the posterior p((]X) by training samples from p(t, X), where X can be dataset D like above, but typically is a single vector. Previous work in simulation-bast based inference is focused on simulations for which specific model underlying the data is known (Lueckmann et al. 2021). Our method is in far similar to simulation-based inference as both solely use samples from the prior. In the next section, will show how we can model the PPD directly, though, without ever instantiating the posterior, and apply this to large datasets with general priors.
3 PPD APPROXIMATION WITH PFNS
Let us consider a parameterized model qo that can accept dataset D = {z; y/ y: as well as query as input, and which predicts distribution over possible values of y for the query x. Many neural architectures can be used as such a model; in this paper, we use variant of Transformers (Vaswani et al., 2017) as they are reliable and powerful architecture. We train this model by
Algorithm 1: Training PFN model by Fitting Prior-Data
Input :A prior distribution over datasets pDD), from which samples can drawn and the number of samples K to draw
Output :A model go that will approximate the PPD Initialize the neural network qo:
for j - to K do
Sample D U {( w)) ~ p(D):
Compute stochastic loss approximation Ey >m> (- log (o D));
Update parameters with stochastic gradient descent on Volo;
end
3

---

Published as a conference paper at ICLR 2022
cross-entropy samples drawn from the prior. Our proposed loss, the Prior-Data Negative Log-Likelihood (Prior-Dace Data NLL) Eg defined as
to T. (=( log go (yla,, D)},
where DU {a, y) simply is dataset of size [D| sampled from p(D).
PFNs are trained to minimize bg like outlined in Algorithm|]! [ We draw many datasets from our prior and fit model to predict a hold-out example conrectly for these.
In contrast to VI and MCMC, we learn to approximate the PPD directly from dataset samples. For all cases consider, it is simple and cheap to sample from p(D), fulfilling the only requirement our method has. MCMC on the other hand requires access to a non- non-normalized posterior (() D, x) x p([[,=)) and VI requires access the density values of p(l. D), both of which be hard to compute.
Following similar insights Gordon al. (2019). le yields an approximation of the PPD in terms of cross-entropy and thus KL-Divergence:
show that minimizing the Prior-Data NLL
Insight 1. The proposed objective tg equal to the expectation of the cross-entropy between the PPD and its approximation qe: Lg EE,, D=p(D) [H[p(-|z, D), qo (- D))]
Proof. The above can be shown with a simple derivation. We mark transformations for easy reading.
D.w. plz y, D) log (y(12 D) =-
D.D. p(a, D) / plylæ; D) log (u(1,, D)
to
(3)
[ J
Do) D)) Ex Drop(D [H(p(11,, D), qo (-12, D))
Thus, have approximation of the PPD. We can formulate this in terms of KL-Divergence, too. Corollary 1.1. The loss to equals the expected KL-Divergence ED., (KL(p-llz, D), go(-|o D)) between p(- ((, D) and qo(-)z D) over prior data x, D, up to an additive constant. Proof App.
In the following,
consider the optimum of the given optimization problem H" E arg ming Eo- Corollary 1.2. p is a member of the distribution family qe (i.e., there is 0 such that p). we have qo (-12, D) = p(1|z, D) for all x, for which p(11x, D) is defined. Proof App.
(a)
(b)
(c)
dl|F,, D) psssss D)
Denity
Danity
0.2
0.2
(21, (11.v1)(r2)) (2ra, ya)
34 27
-3
-10
3
-3
-10
1
3
D
Figure 2: (a) visualization of the Transformer for n = 3 input pairs and m = queries. Every bar the representation of one input and the arrows show what each representation attend to. (b-c) A visualisation of the Riemann distribution, with and without bounded support.
4 ADAPTING THE TRANSFORMER FOR BAYESIAN INFERENCE
In this section, we propose an efficient architecture based on the Transformer encoder Vaswani et al. 2017) and novel regression head for regression problems.
An Efficient Architecture Our architecture, visualized in Figure 2a is based on a Transformer
encoder without positional encodings, which makes it invariant permutations in the dataset D. We
4

---

Published as a conference paper at ICLR 2022
feed the Transformer with simple linear projections of inputs and queries. Our model returns the PPD for each given query z, only depending on the dataset D and the query itself. We sample the number of imputs and queries randomly such that N
+ for some fixed N. See Appendix E. for a more detailed deseription and Appendix E.3 for an ablation the permutation invariance.
For additional practical training information see Appendix [
Riemann Distribution The modelling continuous distributions hard with neural networks. To yield strong performance in modelling PPDs, used a distribution that works particularly well with neural networks. Based on the knowledge that neural networks work well for classification and inspired by discretizations in distributional reinforcement learning (Bellemare al., 2017). we made use of a discretized continuous distribution that we call Riemann Distribution. PDFs of this distribution are bar plots, see Figure 2b We select the boundaries of the buckets B which we discretize such that they all have the same probability in prior-data: p(y E b) 1/|Bi, vb E B. We estimate this using large prior-data sample. In experiments with priors that need unbounded support, we replace the last bar on each side with an appropriately scaled half-normal distribution, see Figure 2c For a more exact definition, we refer the reader to Appendix B.2 For an ablation of the Riemann distribution, see Appendix E.3
Theorem 1. Riemann distribution QB with finite support, where B are the buckets, can model any distribution with finite, almost everywhere continuous (ieemann-integrabae density function and full support up to arbitrary precision. That is, VpVeaB.KL (B p) S Proof in App.
(a)
(b)
1.5
1.5
0.5
Given examples
confidence
PFN density
PEN mean
PEN 95% confidence
Figure 3: (a) The PFN's PPD given two evaluations of the function, highlighting the binning it its predictions (best viewed in full-screen mode). (b) The PFN's and the GP's mean and confidence intervals, which are nearly identical (red arrows mark tiny exemplary differences). For additional comparisons, see Figure in the appendix.
POSTERIOR APPROXIMATION STUDIES
In our first set of experiments, we study the capability of PFNs to perform Bayesian inference for the tractable case of Gaussian Processes (GPs) with fixed hyperparameters (where we can compare to ground truth data; Section 5.1 and the intractable cases of GPs with unknown hyperparameters (Section 5.2 and Bayesian Neural Networks (BNNs; Section 5.3
Corollary 1.1 shows that the Prior-Data NLL is equal to the KL-Divergence between the exact PPD and the approximation, up to an additive constant. We thus use in all following experiments to compare how close methods can match the correct PPD. For methods that do not approximate PPD directly, we approximate the integral of the PPD in Equation with a large Monte Carlo integration. In this section, report the 95% confidence interval over different sampled datasets of the Prior-Date NLL for each experiment as shaded areas alongside the mean. For more information of the setup of the following setup refer the reader Appendix F
5.1 GAUSSIAN PROCESS APPROXIMATION
We begin by approximating GPs to study what PFNs are capable of; GPs with fixed hyperparameters convenient for this purpose since the tractability of their PPD allows assess how close
5

---

Published as a conference paper at ICLR 2022
(a)
Negolte Log Log-Lihcillod
Negatios Logs Liciiieeloood
-2
500
1,000
1,500
2,000
10
20
30
Number of revealed data puints (s)
Number of revealed points (t)
PFN (500K dutasels)
PFN (IM datneets)
MI.F-II
NUTS w/ 32 (scrmmpp) steps
PEN (2M datasesa)
PEN (4M) datasets)
NUTS w/ 256 (warawp)
PEN (4M datssets)
GP
Figure 4: (a) Prior-Data NLL with a fixed GP. While, the PPD of the GP is exact, we can see that our approximations get very close to it. (b) Prior-Data NLL with mix over GP hyperparameters. There is no closed-form solution for this PPD, but we can see that our PFN still gets closest to the true PPD. Our PFN is also more than x faster than any of the baselines.
our approximation to the exact PPD. To create the prior for this task, we sample the N inputs x; uniformly random from the unit-cube. The data-prion of zero-mean GP with a kernel function k, given the sy's, can be described as a multivariate normal distribution /O K), where Kig = k(,,,,y) We sample from this normal distribution to yield a dataset ((x1, 11); ------ (IIN, yN)]- As described above, we then train our PFNs to predict the y's for a held-out subset of the dataset. The trained PFN can then be used to approximate the PPD for new datasets and arbitrary query points in the unit-cube.
Figure 3 visualizes the predictions of such PFN with two data points and query points ranging from 0 to 1. We visualize both the estimated PPD as heat map, as well as confidence bounds, compared to the exact PPD. The estimated confidence intervals and means are virtually indistinguishable from the ground truth, therefore we marked the most noticeable differences with red arrows. The model learns on its own, completely automatically, to generate smooth distributions without any explicit knowleder of the positions of the hars in R. We published] an interactive version of this experiment. In Figure 7 of the appendix, we show more qualitative examples of the behavior of our approximation together with the exact posterior for a different length scale. In Figure 8, we additionally compare to Attentive Neural Processes (Kim et al., 2018), showing the clear superiority of our method for Bayesian Inference. Figure 4a shows that the qualitative results above generalize to datasets with up to 2000 examples with multiple features. The tractable GP posterior, which achieves optimal performance by definition, is approximated very closely, and increasingly better so the more meta-train datasets train the PFN for.
5.2 APPROXIMATING A GP WITH HYPER-PRIORE
common practice fields applying GPs in the real world to define distributions over the hyperparameters of these GPs (Rasmussen & Williams 2006 [noek et al., 2012). so-called hyper- priors. Thus, the prior of these models considers a more diverse set functions, to model the fact that the correlation between data points, the smoothness of functions and the scale of outputs varies across applications. The downside of using syper-prions is that it is not possible to compute the PPD of the GP exactly anymore. There are two common practices to approximate it anyways, and we evaluate against both. (i) Firstly, MLE-II, II, the most common approximation. This is a simple special case of variational inference. Here, suitable hyperparameters
found by maximum posteriori (MAP) estimation (e.g., Rasmussen & Williams (2006)). ). We use the fitting setup of Bo (Balandat et al. 2020) which also inspired our tlyeer-prors (ii) Secondly, Markov Chain Monte Carlo (MCMC), method to sample from any distribution for which one can calculate non non-normalized probabilities, that is also frequently applied sample hyperparameters (e.g.. Snoek et al. (2012)) Here, we use the offee--ofthe-ttte algorithm NUTS (Hoffman et al. 2014), which uses gradients to facilitate the
httpa: httpa://ruggifrn acee.cc ca/ org.eeaeeee@@..eeeeeeeeeeeeeeeee

---

Published as a conference paper at ICLR 2022
(a)
(b)
Negative Log Lok-likelloodd
104
Negative Log-- Etalhood
10
10"
10
110
II
100
10-2
2
10
10"
10
10a
10
10
10
10
10
10a
10
Training time per
Training per
NUTS
SVI
PFN PFN Dattssets)
Figure 5: Time spent for Bayesian Inference per dataset compared to Prior-Data NLL. For NUTS we scale the number of warmup and sampling steps, for SVI, we scale the number of training steps and averaged samples. PFN is evaluated at only setting (blue dot). We show two different BNN architectures (a) a BNN with 3 features, layers and a hidden dimensionality of 5 and (b) a BNN with 8 features, 2 layers and hidden dimensionality of 64.
Hamiltonian Monte Carlo algorithm. We plot the performance of the different methods in Figure 4b Our model can approximate the PPD on the prior much closer than both of these methods, and its inference more than 200 faster than MLE-II and x000 x00x faster than NUTS. Further details can be found in Appendix
5.3 APPROXIMATING BNNS
Bayesian Neural Networks (BNNs) provide strong ability model uncertainty due their Bayesian nature, compared to traditional neural networks. Recently, posterior approximation methods, such as stochastic variational inference (SVI) and stochastic gradient MCMC methods (such as NUTS) allow for much faster convergence, which has sparked a lot of interest in the area (Izmailov et al. T02T Fortuin et al. 2021). However, computational feasibility still remains a key issue of current BNNs.
In Figure we show that our PFNs can outperform today's default methods for BNN inference using a fraction of their compute budget (PFNs are 000x faster than Buyes-by-Backpryy SVI and 10 x000 faster than NUTS to achieve the same performance). The figure shows the Prior- Prtor-Data NLL of the BNN prior that used during inference by all solvers. To generate prior-data for our PFNs, we sampled random weights for & and used cormall-ddistrbutee i.i.d. feature vectors X; to obtain y/ : (X.) with D == (Xim
Im Vi:.). For the evaluation we sampled 100 datasets, with m = 200 test data points and n 100 revealed data points per dataset. A detailed description of baselines as well as further experiments are included in Appendix
APPLICATION TO TABULAR DATASETS
We evaluate Transformer-basse classifiers with priors over Gaussian Processes and Bayesian Neural Networks on a range of small real-world tabular datasets. Surprisingly, the simple priors we describe next outperform large set of baselines, including XGB (Chen & Guestrin 2016) and Catboost (Prokhorenkova et al. 2018), while greatly surpassing all baselines in uncertainty calibration.
GP Prior for Tabular Classification We build a classification prior based on GPs with a hyper- prior, as described in Section 5.2 Before, we used this prior for regression. To make it classification prior, we find the median over all targets in the dataset y and set the class to one greater than the median and zero otherwise.
BNN Prior over Architectures We generalize our BNN setup to learn a prior over model architec- tures. Fitting neural network requires finding suitable architecture, e,g., embedding size, number of layers and activation function. Commonly, resource- resource-intensive searches need to be employed to find suitable architectures (Liu et al. 2018: Zoph et al. 2018). The result of this search is not posterior distribution over architecture choices and their respective parameters, though, but instead only point-estimate of the architecture choice. This true for Bayesian optimisation approaches as
7

---

Published as a conference paper at ICLR 2022
(a)
(b)
Figure 6: (a) A BNN. The inputs I are mapped to the output y. (b) A prior over BNN architectures. The network considers a distribu- tion over different model architec- tures at once.
Fi
well (Kandasamy et al. 2019: Zhou et al. 2019 Ru et al. 2021). Ensembling methods over multiple architectures yield a rough approximation
distribution over architectures, but this scales linearly in cost with the number of architectures considered (Zaidi et al., 2020). Using dropout can be regarded as ensemble learning but is limited to varying connectivity (Hara al. 2017). PFNs allow us to be fully Bayesian, not only about the weights of the BNN, but its architecture, too. By defining a prior probability over the space of model architectures, we remove the need for finding an optimal neural architecture and instead jointly integrate over the space of architectures and their respective weights in a single Bayesian inference. This yields a distribution over architectures the posterior. The degrees of freedom of model architectures we considered can be found in Appendix G Our prior-data sampling algorithm for learning to predict based on a prior over both weights and architectures iterates the following steps:
(1) Sample model architecture A ~ p(4)
(2) Sample model weights for the given architecture: Wa (.
(3) Sample i.i.d. features f ~ N (0, 1) for each data point and each feature f
(4) Propagate each 2; forward through Aw yielding {=,; Aw (xx)))
We binarized targets like for the GP Prior above. See Appendix G for more details.
Setup of Datasets We used a large collection of tabular datasets from the open-source OpenML AutoML Benchmark (Gijsbers et al., 2019); we first removed datasets with more than one hundred features or missing values, ending up with 20 datasets that represent diverse set of classification problems with numerical and categorical features. We then simplified each dataset to a balanced binary classification problem, and, for each dataset sampled 20 subsets, each including 100 samples. Within each subset we provide labels for the first 30 samples and evaluate on the remaining samples. All baselines and PFNs use the same datasets and subsets. We also define set of six unrelated validation datasets used for optimizing the prior distribution over architectures of the PFNs. This similar to setting the range of hyperparameters in cross-validation grid search and can be reused for all similar problems. See Appendix G for more details.
Setup of Models Our baselines comprise two Gradient Boosted Tree algorithms, XGBoost (Chen & Guestrin 2016) and Catboost (Prokhorenkova et al. 2018), as well as logistic regression, K-Nearest Neighbors, Gaussian Processes (RBF Kernel) and Bayesian Neural Networks (Bayes-by berres-by-Backkeee Backprop SVI with Pyro). Catboost additionally received information on which features are categorical. We used two PFNs, one PFN for each prior described above. We were able apply the same model across different feature dimensions. We used grid search with 5-fold cross-validation to optimize our baselines' hyperparameters for each dataset, using the hyperparameter spaces described in Table 6 in the appendix.
Experimental Results Table Daole[oomee compares the area under the receiver operating curve (ROC AUC) of PFNs based on BNN and GP priors against the various baselines. While the PFN-GP demonstrates on-par performance to the best of the evaluated baselines, the PFN-BNN performs strongest in this comparison. The Wilcoxon signed-rank test (Demsar (2006). a standard metric for comparing classifiers across multiple datasets evaluated on each of the 20 subsets of the 21 evaluation datasets separately and shows a significant performance improvement. While all baselines make use of cross-validation to find suitable hyperparameters, the PFNs naturally consider a distribution of hyperparameters and integrate cross-validation and training within the model's Bayesian inference step. As shown in Table T this makes it very fast compared to the baselines: since all it needs to do on new dataset is single forward propagation step, when on GPU (Nvidia Tesla V V100), it requires as little as 13 seconds for all 20 datasets combined. This is more than 5000 faster than the 20 hours XGBoost requires, and even faster than K-Nearest Neighbors.
8

---

T0 of the appendix, Published as a conference paper at ICLR 2022
Metric
PFN-BNN PFN-GP Log. Reg.
GP
KNN
BNN
Catboost
XGB
Mean rank ROC AUC
2.786
3.833
4.690
5.286
6.214
5.000
4.833
3.357
Loss/Tie/Win PFN-BNN
14/1/6
16/1/4
17/0/4
17/1/3
17/1/3
13/1/7
12/2/7
Wilcoxon P. VS FFN-BNN
3.6c-13
3.6c-13
3.9c-13
3.6c-13
3.6c-13
1.9c-12
1.6c-06
Expected Calibration Error
0.025
0.067
0.157
0.095
0.093
0.089
0.157
0.066
GPU: 0:00:13
Benchmark Time
0:09:56
0:24:30
0:00:34 12:04:41
2:05:20
20:59:46
CPU: 0:04:23
Table 1: Aggregated performance on subsets of datasets with 30 training samples. Our novel PFN-BNN performs strongest overall. See Table the appendix for per-dataset results.
We can also experimentally confirm that our PFNs make well-adjusted uncertainty predictions. We find that our data they have lower Expected Calibration Error (ECE, using 100 bins (Nacini et al., 2015)) even than standard BNNs and ensemble methods such as XGBoost, the two baseline methods that best capture uncertainty. We also provide confidence histograms in Figure visually demonstrating the PFN's strong calibration.
APPLICATION TO FEW-SHOT LEARNING
As shown in the previous section, PFNs can be trained with priors that are not (easily) possible to use in the traditional Bayesian setups. In this section, show another example of this, building prior hand-writter digits and letters and demonstrating strong performance on Omniglot (Lake et al. 2013) in combination with fine-tuning of the PFN on Omniglot.
As an approximation to hand-written symbols, we create a prior that generates random straight lines. In each sample from this prior, consider five different random line assortments as class prototypes. For each instance of a class, we apply noise to the positions of the lines. It required just 55 lines of readable code create this prior (see Figure 1I i the Appendix). In Figure 12 of the Appendix show a sample dataset from this prior.
For our evaluation Omniglot, 5-way classification, i.e. we consider five classes and five examples of each class. We create one task from the first five classes of each of the 30 train and 20 evaluation alphabets in Omniglot. We train our PFN on the synthetic prior and then fine-tune it on the 30 training tasks created. This practice unlike the previous sections. Our premise is show the flexibility of our model: it can even be fine-tuned on similar datasets, if these are available. We do not use task augmentation, like Rothfuss al. (202T), but apply translations each image, as we only use simple linear layers encode the images in the Transformer. In Table 2 one can see that for this very hard version of the Omniglot dataset our simple method yields results on par with the state of the art. For more information on the setup, please see Appendix H
follow the setup of Rothfuss et al. (221)) We perform 5-shot
Method
Accuracy
Method
Accuracy
Vanilla BNN Liu Wang 2016)
0.795 0.006
MLAP (Amit & Meir 2018
0.700 0.0135
MAML Finn et al. 2011/b
0.693 + 0.013
BMAML (Yoon et al. 2018
0.764 + 0.025
PACOH-NN (Rothfuss et al. 2021
0.885 0.090
PFN (this work)
0.865 + 0.019
Table 2: Comparison of meta-learning algorithms in terms of test accuracy the Omniglot dataset. Results within the confidence interval of the best performance are bold-faced.
8 CONCLUSION & FUTURE WORK
We presented Prior-Fitted Networks (PFNs), a novel way to efficiently approximate the posterior predictive distribution using deep neural networks and showed its capability on a set of diverse tasks and priors. We expect the following future work to be especially fruitful: (i) Work on finding novel priors that are now feasible to approximate using PFNs. (ii) Work on architectures that are well-fit for this task, scaling PFNs to even larger real-world problems. (iv) Work on using our model for the amortized simulation-banl based inference setting.
simply used slight adaption of current Transformer models. (ii) Work

---

Published as a conference paper at ICLR 2022
9 ACKNOWLEDGEMENTS
This research was funded by the Deutsche Forschungsgemeinschattt (DFG, German Research Foun- dation) under grant number 417962828, Robert Bosch GmbH is acknowledged for financial support. We also want to thank the maintainers of PyTorch PPaskke et al. 2019).
REFERENCES
Ron Amit and Ron Meir. Meta-learning by adjusting priors based on extended pac-bayes theory. In International Conference on Machine Learning. pp. 205-214, PMLR, 2018.
Christophe Andrieu, Nando De Freitas, Arnaud Doucet, and Michael I Jordan. An introduction to meme for machine learning. Machine learning, (0(1):5-43, 2003.
Maximilian Balandat, Brian Karrer, Daniel R. Jiang, Samuel Daulton, Benjamin Letham, An- drew Gordon Wilson, and Eytan Bakshy. BoTorch: A Framework for Efficient Monte-Carke Bayesian Optimization. In Advances in Neural Information Processing Systems 33, 2020.
Marc G Bellemare, Will Dabney, and Rémi Munos. A distributional perspective reinforcement learning. International Conference on Machine Learning, pp. 449-458, PMLR, 2017.
Eli Bingham, Jonathan P. Chen, Martin Jankowiak, Fritz Obermeyer, Neeraj Pradhan, Thcofanis Karaletsos, Rohit Singh, Paul Szerlip, Paul Horsfall, and Noah D. Goodman, Pyro: Deep Universal Probabilistic Programming. Journal of Machine Learning Research, 2018.
David M. Blei, Alp Kucukelbir, and Jon D. McAuliffe. Variational inference: A review for statisticians. Journal of the American Statistical Association, 112(518):859-877 Apr 2017. ISSN 1537-274X. doi: 00.1080/06621455 011.992017.1287777
Blundell, Cornebise, Kavukcuoglu, and D. Wierstra. Weight uncertainty in neural network. In F. Bach and D. Blei (eds.), Proceedings of the 32nd International Conference on Machine Learning (ICML') 15), volume 37, pp. 1613-1622. Omnipress, 2015.
Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et Language models few-shot learners. arXiv preprint arXiv:2005. /4165, 2020.
Jeffrey Chan, Valerio Perrone, Jeffrey P Spence, Paul A Jenkins, Sara Mathieson, and Yun $ Song. A likelihood-free inference framework for population genetic data using exchangeable neural networks. In NeurlPS, 2018.
Tianqi Chen and Carlos Guestrin. Xghoost: scalable tree boosting system. In Proceedings of the 22nd ACM SIGKDD International Conference Knowledge Discovery and Data Mining, KDD 16, pp. 785-794, New York, NY, USA, 2016. Association for Computing Machinery. ISBN 9781450342322. doi: 10. (0.11144/93396722999785
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597-1607. PMLR, 2020.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15750-15758, 2021.
Kyle Cranmer, Johann Brehmer, and Gilles Louppe. The frontier of simulation baasdd inference. Proceedings of the National Academy of Sciences, 117(49):30055-30022 2020.
Ekin Cubuk, Barret Zoph, Jonathon Shlens, and Quoc Le. Randaugment: Practical automated data augmentation with reduced search space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 702-703, 2020.
Janez Demšar. Statistical comparisons of classifiers over multiple data sets. J. Mach. Learn. Res., 1-30, December 2006. ISSN 1532-4435.
10

---

Published as a conference paper at ICLR 2022
Jacob Devlin, Ming- bidirectional transformers for language understanding. arXiv preprins arXiv: 018.800000800, 2018.
Ming-Wei Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep
C. Finn, Abbeel, and Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In D. Precup and Y. Teh (eds.), Proceedings of the 34th International Conference Machine Learning (ICML' 17), volume 70, pp. 1126-1135. Proceedings Machine Learning Research, 2017a.
C. Finn, T. Yu, T. Zhang, P. Abbeel, and S. Levine. One-shot visual imitation learning via meta- learning. In Proceedings of the 1st Annual Conference on Robot Learning (CoRL), volume 78, pp. 357-368. PMLR, 2017b.
Vincent Fortuin, Adrià Garriga-Alonoo, Florian Wenzel, Gunnar Rätsch, Richard Turner, Mark van der Wilk, and Laurence Aitchison. Bayesian neural network priors revisited, 2021.
Marta Garnelo, Dan Rosenbaum, Christopher Maddison, Tiago Ramalho, David Saxton, Murray Shanahan, Yee Whye Teh, Danilo Rezende, and SM Ali Eslami. Conditional neural processes. In International Conference on Machine Learning, pp- 1704-1713. PMIR, 8a.
Marta Garell,, Jonathan Schwarz, Dan Rosenbaum, Fabio Viola, Danilo J Rezende, SM Eslami, and Yee Whye Teh. Neural processes. arXiv preprint arXiv:1887.001622. 2018b.
Gijshers, E. LeDell, S. Poirier, J. Thomas, B. Bischl, and J. Vanschoren. An open source automl benchmark. arXiv preprint 1907.00900000000 [cs.LG], 2019. Accepted at AutoML Workshop at ICML 2019.
Jonathan Gordon, John Bronskill, Matthias Bauer, Sebastian Nowozin, and Richard Turner. Meta- learning probabilistic inference for prediction. In International Conference on Learning Represen- tations, 2019.
Kazuyuki Hara, Daisuke Saitoh, and Hayaru Shouno. Analysis of dropout learning regarded as ensemble learning. CoRR,
2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Dan Hendrycks, Norman Mu, Ekin Dogus Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshmi- narayanan. Augmix: simple data processing method improve robustness and uncertainty. In International Conference on Learning Representations, 2019.
Sepp Hochreiter, A Steven Younger, and Peter R Conwell.
Learning to learn using gradient descent. In International Conference on Artificial Neural Networks, pp. 87-94. Springer, 2001.
Matthew D. Hoffman, David M. Blei, Chong Wang, and John Paisley. Stochastic variational inference. Journal of Machine Learning Research, 14(4): 2013.
Matthew D Hoffman, Andrew Gelman, et al. The no-u-turn sampler: adaptively setting path lengths in hamiltonian monte carlo. J. Mach. Learn. Res., 15(1):1993-1633, 2014.
Pavel Izmailov, Sharad Vikram, Matthew D. Hoffman, and Andrew Gordon Wilson. What bayesian neural network posteriors really like? CoRR, abs/2104. 14421, 2021.
Michael I Jordan, Zoubin Ghahramani, Tommi Jaakkola, and Lawrence Saul. An introduction to variational methods for graphical models. Machine learning, 37(2):183-233 1999.
Arlind Kadra, Marius Lindauer, Frank Hutter, and Josif Grabocka. Regularization is all you need: Simple neural nets can excel on tabular data, 2021
Kirthevasan Kandasamy, Willie Neiswanger, Jeff Schneider, Barnabas Poczos, and Eric Xing. Neural architecture search with bayesian optimisation and optimal transport, 2019.
Hyunjik Kim, Andriy Mnih, Jonathan Schwarz, Marta Gareeo,, Ali Eslami. Dan Rosenbaum, Oriol Vinyals, and Yee Whye Teh. Attentive neural processes. In International Conference on Learning Representations, 2018
11

---

Published as a conference paper at ICLR 2022
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR (Poster), 2015.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convo- lutional neural networks. In F. Pereira, C. C. Burges, L. Bottou, and K. Q. Weinberger (eds.), Advances in Neural Information Processing Systems, volume 25, Curran Associates, Inc., 2012.
Brenden M Lake, Ruslan Salakhutdinov, and Joshua B Tenenbaum. Human-level concept learning through probabilistic program induction. Science, (13062666)133321333336: 2015.
Y. LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard, and L. D. Jackel. Backpropagation applied handwritten zip code recognition. Neural Computation, 1(4):541-551, 1989. doi: 10. 162/neco. 1989. 4.5.41.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. In International Conference on Learning Representations, 2018.
Qiang Liu and Dilin Wang- Stein variational gradient descent: A general purpose bayesian inference 2016.
algorithm. arXiv preprint 1w0iv:160..044
Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:/608. 1XXx:/668..3383, 2016.
Jan-Matthis Lueckmann, Jan Boelts, David Greenberg, Pedro Goncalves, and Jakob Macke. Bench- marking simulation- based inference. In International Conference on Artificial Intelligence and Statistics, pp. 343-3433333 351. PMLR, 2021.
David JC MacKay. A practical bayesian framework for backpropagation networks. Neural Computa- tion, 4(3):448 47244444442 1992.
Mahdi Pakdaman Naeini, Gregory Cooper, and Milos Hauskrecht. Obtaining well calibrated probabilities using bayesian binning. In Proceedings of the Twenty-Ninth AAAI Conference Artificial Intelligence, AAAI' 15, pp. 2901-2907. AAAI Press, 2015. ISBN 026251 12900.
R. Neal. Bayesian Learning for Neural Networks. 1996.
George Papamakarios and lain Murray. Fast e-free inference of simulation models with bayesian conditional density estimation. In Advances in neural information processing systems, pp. 1028- 1036, 2016
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Vito,, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. H. Wallach, H. Larochelle, Beygelzimer, F. d'Alché- Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems 32, pp. 8024 8035. Curran Associates, Inc., 2019.
Liudmila Prokhorenkova, Gleb Gusev, Aleksandr Vorobev, Anna Veronika Dorogush, and Andrey Gulin. Catboost: Unbiased boosting with categorical features. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS' 18, pp. 6639-6649 Red Hook, NY, USA, 2018. Curran Associates Inc
C. Rasmussen and C. Williams. Gaussian Processes for Machine Learning. The MIT Press, 2006.
Carl Edward Rasmussen and Christopher K. I. Williams. Gaussian Processes for Machine Learning (Adaptive Computation and Machine Learning). The MIT Press, 2005. ISBN 026218253X.
Jonas Rothfuss, Vincent Fortuin, Martin Josifoski, and Andreas Krause. Pacoh: Bayes-optimal meta- learning with pac-guarantees. In International Conference on Machine Learning, pp. 91 9116-9122. PMLR, 2021.
Binxin Ru, Xingchen Wan, Xiaowen Dong, and Michael Osbome. Interpretable neural architecture search via bayesian optimisation with weisfeiler-lehman kernels. In International Conference on Learning Representations, 2021.
12

---

Published as a conference paper at ICLR 2022
Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy Lillicrap. Meta- learning with augmeettd neural networks. In International conference on machine learning, pp. 1842-1850. PMLR, 2016.
Snoek, H. Larochelle, and R. Adams. Practical Bayesian optimization of machine learning algorithms. In P. Bartleu, F. Pereira, C. Burges, L. Bottou, and K. Weinberger (eds.), Proceedings of the 25th International Conference on Advances in Neural Information Processing Systems (NeurIPS' 12), pp. 2960-2968 2012.
Ray Solomonoff. The discovery of algorithmic probability. Journal of Computer and System Sciences, 55(1):73-88, 1997. ISSN 0022-0000. doi: https://doi.org: org. 1006/jess. 1997.1500.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, &. ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
Martin Wainwright and Michael Irwin Jordan. Graphical models, exponential families, and variational inference. Now Publishers Inc, 2008.
Ben Wang and Aran Komatsuzaki. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model. https:://iithub cow wwtp:/cgooo/oooo//oooooooooo. /wwhh transformes jax May 2021.
Max Welling and Yee W Teh. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th international conference machine learning (ICMI--I1, pp. 681-688. Citeseer, 2011.
David H Wolpert and William G Macready. No free lunch theorems for optimization. IEEE Iransactions on evolutionary computation, 1(1):67-82, 1997.
Jaesik Yoon, Taesup Kim, Ousmane Dia, Sungwoong Kim, Yoshua Bengio, and Sungjin Ahn. Bayesian model-agnostic meta-learning. In Proceedings of the 32nd International Conference Neural Information Processing Systems, pp. 7343-7353, 2018.
Sheheryar Zaidi, Arber Zela, Thomas Elsken, Chris Holmes, Frank Hutter, and Yee Whye Teh. Neural ensemble search for uncertainty estimation and dataset shift. arXiv preprint wrXiv:2006.0857. 2020.
Hongpeng Zhou, Minghao Yang. Jun Wang. and Wei Pan. Bayesnas: A bayesian approach for neural architecture search. CoRR, abss/905..m 149 2019.
Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V Le. Learning transferable architectures for scalable image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 8697-8710, 2018.
13

---

Published as a conference paper at ICLR 2022
A
PROOF OF COROLLARY 1.1
Proof.
This can easily be seen by considering the result of Insight [ for the last transformation in
line
Ex, D (e_,<KLL(((/a,, go D))]
(5)
/ p(ula, D) log (o(y)z, D)
--EaD
(6)
plylx, D)
U plyla, D) log go(y/ D) + Ez,D [/ plylz, D)
= -EEDD
-Eg, (a)(H((((-ll)) D), 20 D)) Eg,D (Ex(D[H(p(-1e)) D))I]
(8)
Eo C,
where C= - D D)] is a constant that does not depend on 0.
B PROOF OF COROLLARY 1.2
Proof. We assume a f with qo p exists and we know that the cross-entropy is minimized for equal
distributions; thus the optimum H fulfills qo- qo-(-la; (-|z, D) = ([,| D) for all x. D where the density of
the prior fulfills p(z, D) > 0, which is exactly the subset for which the conditional is defined. O
C PROOF THEOREM
Proof. Given an arbitrary distribution p, an arbitrary bound c. For arbitrary minimum (maximum) bucket borders (b), we consider the function bB that maps each position y its bucket 6 E B, where B is an arbitrary set of buckets, We define the upper bound of p for each bucket
u(y) sup plv')
(10)
yb)
Similarly, we define the lower bound
I(y) = inf p(y').
(11)
For each of these we consider their log form, too. That is ū(y) = logu(y) = sb) logp(v')
and ((y) log (vy = infyrebaty) log'(y'). We can move the log inside the sup (inf), since log is steadily increasing.
Integrals over and thus handily represent Darboux's upper and lower sums of log p. if log p is Riemann integrable and finite. We assumed that is Riemann integrable, thus log p is also integrable, since the composition of continuous functions, which log is for > O, with an almost everywhere continuous function is almost everywhere continuous and any finite value has finite value log > O. The requirement æ > is given due to the assumed full support of p. Thus, we can use FB' _ ū(v) ily)dy < c'. We define t = suPye(a.M) p(y). T = suPyER p(y) and the total probability mass of the interval P = ply E [a, b)). We use e' e log 1/ P, which fulfills e' > 0 since p has full support and thus for any borders and we have < 1, get B s.t.
a(v) -(v)dd < log 1
(12)
t(ū(g) ((y)) 1/o
(13)
1. flog T(y)
bb u(y) dy < blog 1/P
(14)
/' u(v) log u(y) dy log 1/P 1(y)
(15)
14

---

Published as a conference paper at ICLR 2022
Let be constant s.t. cu(y) yields valid probability distribution with f culy)dy Sa culy)dy =.
uly) dy < et log1/P ((v)
cu(y) log
(16)
cu(y) log cu(y) dy at log 1/P L: cu(y) loge
(17)
Iyy)
culy) log cu(y) dy < ct log 1/P P + loge
(18)
I(y)
We use e < 1/P, as 1/P would be the factor d' make Sa d'ylyddy and e is this factor for u(y).
which has u(y) 2 p(y) for any y ba,
cu(y)
culy) log
dy at log 1/P 1/P 1(y)
(19)
cu(y) log cu(y), dy < (ct + log 1/P
(20)
((y)
culy) log cu(y) dy
log P
log P
(21)
((y)
P
cu(y) log cu(y) dy < -T log P
log P
(22)
[(y)
P
cu(y) log cu(y) dy -T p(y)
log P
log P
(23)
(L(cu(v), < -T log P
log P
(24)
P
log P
(L((c(y)) < -(T' + P)
(25)
P
KL(cu(y), < (T + 1) log P
(26)
P
Here, cu(y) is valid bar distributions. Now we want to find a and b such that -(T + 1) log P which is equivalent to
6,
P log 1/P
(27)
T+1
Here, we can substitute the Lambert W function to arrive at the final result, that we should choose and such that p([a, b]) 2 e Wo' which is always possible as Wo maps to (0, oc) for positive inputs, which we have with T+T- And we have that < and -= > 0 for any
(0, oo).
Thus, we can conclude the proof. We could upper bound the KL-Divergence for any and b. Then, based on that bound we found a setting for values a and b such that this upper bounds falls below e.
D GENERAL TRANSFORMER TRAINING DETAILS
We follow the standard Transformer encoder setup. In the whole discussion one should note the crucial difference to a standard MLE training: Our optimization problem does not allow for overfitting. An improvement in training loss always means an improvement in the closeness to the exact posterior. More training just improves performance and calibration. The only exception of this is the fine-tuning done for the few-shot learning experiment in Section 7 We use Adam (Kingma & Ba 2015) and cosine decay (Loshchilov Hutter 2016) with warmup. For all experiments we used embedding size of 512, only for fhw-shot classification
used 1024. The only parameearseteers that did fine-tune for the Transformer training were the batch size and learning rate. Although, it usually was enough to choose a large enough batch size while the learning rate was more important. Learning
15

---

Published as a conference paper at ICLR 2022
rates generally where chosen based on training performance, if not specified differently. We do this as we generate new data during training and thus do not need to tune on held-out data. We saw that search over the set {.00001, .0001, enough results. Usually performance the models got better with more training, thus the number of samples used for training generally depends we could improve performance with more training. We stopped when we were satisfied with the results, We did not train longer than few days single standard GPU for any PFN.
.001} for learning rate generally yielded good
compute budget. For setups models considered
+2
-2
-2
+2
0.5
0.5
Figure We show the mean and 95% intervals of the original GP posterior green and the corresponding PFN approximations in blue. We used length scale of 0.1.
0.2
0.4
0.6
0.8
Figure 8: Like in plot 7 we show the mean and 95% confidence intervals for the exact GP prior in green and for our PFN in blue. (They are almost indistinguishable, with very small differences visible, e.g., at æ 0.4.) To further highlight what big step forward our method is, we additionally show the respective values for the relatively new Attentive Neural Processes (ANP) (Kim et al.) 2018). We trained both methods in very different environments, but ANP was clearly given the bigger budget.
16

---

Published as a conference paper at ICLR 2022
E TRANSFORMER ADAPTIONS
E. AN EFFICIENT ARCHITECTURE
The first step the model is to encode z and y, in all our experiments, we use simple linear layers for this, but could also use other application-specifes encoding layers. Our setup is equivariant to the ordering of the input dataset's examples. To achieve this, we remove positional encodings and feed (x, y) pairs together to the Transformer for an ablation of the impact of the equivariance to the ordering. In practice, we feed query points together for efficiency reasons; these are the only inputs for which position matters, and thus the output at their positions as prediction for their corresponding PPD over y. Since different query points should not depend on each other, we use an attention mask that allows attention from every input example to every other input example and allows attention from every query points to every input example. An example of the model and its attention can be seen in Figure 2a
sum of their encodings; see Appendix EE.3]
During training, we fix the total number of inputs N = n m, where n is the number of inputs and m the number of queries, and sample n from a distribution, in order to allow the Transformer to learn to work with different dataset sizes. Since for smaller we have more query points to train on, we over-sample larger n, s.t. the number of query points seen during training for each n approximately equal. That is, we sample each n E {O..... .N- 1} with a weight 1/m 1/(N n.). As prediction head we use a softmax for multi-class classification tasks, a sigmoid for binary classification tasks and the Riemann distribution for regression.
E.2 RIEMANN DISTRIBUTION DEFINITION
Given set of of buckets B, where each bucket b B describes an interval such that
U 6 = Ja, b] BEB
(28)
and for any b, E B, we have b 6 6' bound) of the lowest (highest) bucket to be I, and function B--) that maps a y E R to the unique bucket b with E b. We assume to have a function w mapping to the width of each bucket as wbb) = max y minges Lastly, we assume the model gives us probability pa for each bucket
Ø. Additionally, we define the upper-bound (lower-
In the finite case the Riemann distribution is defined as
p(v) = Po(y) Pbts)/ (bbu)))
(29)
We normalize with the width of the bucket since ps describes the probability p(y' b), but interested in the probability p(y = y').
For the case of infinite support, we have
2. fNr 1),
if 21
f(y)) = 2 [N([ Wa),
if y 2 otherwise.
(30)
pb(n.)/b(up))
Here, the Half-Normal distributions are scaled such that half the probability weight inside the bounds of the first (last) bucket.
E.3 ABLATION STUDIES
Above, we introduce novel way of doing regression with neural networks that departs from the tradition of using normal distributions. Additionally, we remove the default positional embeddings from the Transformer. We show the impact of both approaches on the Transformer in Figure 9 Notice that, while the effect of the Riemann Distribution is very pronounced, the effect of making architecture permutation invariant only slowly increases with sequence length. For each setup we searched over a set of the same four learning rates and evaluated the best performing on another sample from the distribution.
17

---

Published as a conference paper at ICLR 2022
Neemal Distriution
Ricmann Distriution
Riemann Disnibutio Positiona Finheddings
Negaiie Lag Liwelhodd
20
40
80
80
100
120
140
160
180
200
Datrset Sioe
Figure 9: In this figure we show the impact of different architectural decisions on the performance of method in fitting Gaussian Process. We the same Gaussian Process as in Section 5.1
F DETAILS FOR SECTION 5
The error intervals that can be seen in the Prior-Data NLL plots are 95% percent confidence intervals over different dataset samples from the prior.
Details for Section 5.1 Here, we generally followed default settings for GPs. That is, we used the RBF-Kernel with zero-mean, length scale of 0.6, le-4 additional noise the diagonal of the covariance matrix. The output scale was set to 1. Only for the plots the Figure 7 and Figure the length scale was decreased to 0.1. The experiments with more examples in Figure [igure44adii use input dimension, while the qualitative visualizations only used one. For all methods we sample 1000 datasets for evaluation, each of which has single validation example, for each dataset size.
Details for Section 5.2 In this section, we followed the default settings that can be found in BoTorrch's SingleTasker (Balandat et al. 2020). We only changed the distribution over the diagonal noise, as the default is very high, such that it is hard to see any improvements with any of the methods in NLL with increasing number of training samples. The GP is setup a Matérn Kernel with 2.5 and we have the following hyper-prion the noise was sampled from I'(a = 0.0001, B 1.0), the output scale from la 2.0, 3 0.15) and the length scale from I'(a 3.0, B 6.0).
NUTS. Here, we used every sampled hyper-parameter setting sample GP. We averaged the output distribution over all sampled GPs to yield an approximation to the PPD. We scaled both the warmup and the rollout steps together: with "8 (warmup) steps" we mean 8 warmup steps and then 8 sampling steps.
We used the Pyro (Bingham et al. 2018) implementation
Only the MCMC experiments were run on CPU, as this was faster than using the GPU. For all methods we sample 1000 datasets for evaluation, each of which has single validation example, for each dataset size.
For SVI (Bayes-by-Backpryg we followed the setup of (Blundell et al. 2015) and used Adam (Kingma Ba| 2015) as optimizer. We leveraged Pyro (Bingham al. (018) for optimization. We scaled both the training steps and number of samples averaged together to increase approximation quality.
Details for Section 5.3
For MCMC we used the Pyro (Bingham et al. 2018 implementation of NUTS and averaged the output distribution over all sampled BNNs to yield an approximation to the PPD. We scaled both the warmup and the rollout steps together to increase approximation performance. We used one MCMC chain. PFN and SVI are evaluated on a GPU (Nvidia Tesla V100) while MCMC is evaluated on CPU since it was faster CPU than GPU.
18

---

Published as a conference paper at ICLR 2022
(a) Bayesian Neural Network
(b) PEN-BNN
(c) XGBoost
1,0
0.8
0.8
0.8
Accunayy
Aucuncy
0.5
0.6
0.6
0.4
0.4
0.6
Confidence
Confidence
Confidence
Figure 10: Comparison of uncertainty calibration curves over all predictions on the evaluated datasets. The dotted line indicates perfect calibration.
G DETAILS FOR SECTION 6
Accomodating a variable number of features Since the evaluated datasets used variable number of features, the dimensionality of the input is variable. This can not be accommodated by the fixed size of the linear layer used our PFNs. Thus, we fixed maximum number of features (60) and extended the datasets to this maximum number by appending zeros. We did this during prior training, sampling the number of features uniformly at random alongside the model architecture and during inference. We rescale the features by factor of features during prior training and bayesian inference to keep the mean input scale constant with a varying feature number.
Degrees freedom for model architectures The evaluated BNN models consider a wide range of model architectures, that is those architectures vary in terms of: number of layers, number of hidden units per layer, layer sparsity, standard deviation of Gaussian noise applied to each unit and activa- tion function applied each unit. This list be easily extended by adapting the sampling procedure.
Tuning hyperparameter ranges for hyper-priors and cross-validation While the Transformer- based methods need no parameter tuning on each dataset individually, they still require setting the prior hyperparameters. This is similar to setting the range hyperparameters grid search and can be reused for all similar problems, e.g., all small-scale tabular classification. We therefore run a hyperparameter search on second set of validation datasets, listed in Table The hyperparameters considered in this grid search can be found in 5 this search, we find good settings for hyper-parameters
cross-validatio
our priors for the average over all validation datasets. This search results in one final hyper-parameter setting and only one single Transformer model that is reused for all datasets. We use the same setup to find cross- validation ranges for our basclines.
Standardization of datasets Each subset is standardized to have zero mean and unit variance where the statistics for the standardization are only calculated the samples considered by the model, thus without any dependence on the test set.
Runtimes of prior fitting step Fitting our transformer with GP prior was done over 100 epochs, taking 2:44h, while fitting the BNN prior took 3:14h for the same number of epochs. Crucially, this model is reused for all datasets.
H DETAILS FOR SECTION 7
We searched for the hyper-parameters specified as inputs in Figure with grid search, where we measure performance on the training tasks as validation set. We then trained our model as a PFN on samples from this prior. After this stage we fine-tuned the model on the training tasks of Omniglot. Here, we applied translations and used default batch size of 100. We used learning rate of le-5 to yield the performance shown the plot after sampling 500,000 samples from the training tasks. Unlike above, we sample over different samples of our fine-tuned model to yield the confidence interval, to follow the standard reporting on the Omniglot benchmark.
19

---

Published as a conference paper at ICLR 2022
def pirokc.prierrsum.caase size=18. min min.max.stokes=((
min, min.mox_los=(573 20/28), min.max..tari=((21 25/228)
min.max.width-(125. (/22),maa.offsetss/22. mox.cargetlooffsee222333
classes = []]
for i in raago(num_chsees
num.strokes random. taddit((min.mxx.strk
len_strokes [random. (indint min_max.Jen
[size min_max, (1]))
for in tange(aum.stroke]]]
randint(int
min.max.start]0)) int(size min.mass.strt (1)))
ranomm randint (inli (ini( size min.max.siart(01) ini(size min.max.start (1)))
for in range (num (num_strukes)]
stroke.direction:
fromarray (ap..erpeee zeros (22. diyy.eep (
draw ImageDraw Draw(i)
for i in rangel num.strokes)::
sp. length strroke.star..ooitssll len.strokes[i]
while True:
if counter =
length
can..aaadnntnniiaaaan. min.mox.lea (01),
min.max.ien|1]]
sp
random, randint (size min.max.star((0))-
iniisize randint(inl miinmatt_sarr(())
stax.ttar(]]])
intisize I [
struke start. points [il. len sirokossss length
radians random. randomi)
math.pi
A.vel raadiaiis leagth
y.vel math sia (radianns length
new.p [sp10] x.vel. spil] y-vel)
for
counter 4-
stroke directions directions.ageedd (radians)
(ppendd (Ien.strakes. stroke.start.points. stroke.directions)
generator-Iaans generator.tenctions
for in classes:
def g(c=e):
len.strokes. stroke_start.poinn stroke.directios
Image fromatray (ap.
dtypesap. winth
drow ImagelDraw Dra(ii)
width random. randint( (size min max width]01).
width
(readom: raadiat
max.offaet). int(size max-offset)),
random. random.rande candint(Intie size max offset). intisize max offset)))
for length. radians in (rexrhackstash
zipistreke.star
len_strokes, stroke.directions):
sp [sp]0] offset/0]. splll offset]ll)
veve! math cos (radiann)) length
x.vel a- random. randint(int(-aa
int(size max.target. (ax.ttrrgt.offfet).
offset))
y.vel sia(radians) length
y.vel *- random. (iat(-size + max offset) int(size max target offxet))
x,vel, sp[l] y.tel)
directions appead'
draw. line (rooud((x) for in sp new pl. fill-128, width-widhh) a_ array
a 1281 random. randiat
sha
reture Image fromarray 1). filter (ImageFilter. Ganssian (.3)) caattar_fucccion appead(g)
return generator Ceneraatorfuunction functions
Figure 11: A prior definition for prior over straight strokes in python.
20

---

Published as a conference paper at ICLR 2022
Name
Number of Features
Number Symbolic Features
kr-vs-kp
37
37
credit-g
21
14
vehicle
19
wine
14
kel
22
1
airlines
8
bank-marketing
17
10
alood-tanssissio-ssriic--eeet
5
phoneme
6
covertype
55
45
numerai28.6
22
connect-4
43
43
7
Australian
15
9
segment
20
jungle chess 2pcs raw endgame complete
7
sylvine
21
MiniBooNE
51
jannis
55
helena
28
Table 3: Tabular datasets used for evaluation. Taken from the OpenML AutoML Benchmark (filtered for features < 100, no missing values.)
Name
Number of Features
Number of Symbolic Features
haberman
4
ionosphere
35
sa-heart
10
cleve
14
four
german
24
Table 4: Tabular datasets used to find good general hyperparameters for hyper-priors as well as hyperparameters used in baseline cross-validation.
Different Samples
7
Training Examples
Query Example
Figure 12: In this figure we show different samples from our prior for Omniglot data.
21

---

Published as a conference paper at ICLR 2022
param
index
prior_activations
torch.nn. Tanh
2 prch.nn.ReLUU
prior dropout dropout
lambda: 0.0
2 lambda: 0.5
beta.sampler [(I11, 1.4)
4
eeta_sampler_[[1]] 2.0)
peta_sampler_f((1. 4.00
beta veta_sampler_f(2.0, 1.4)
beta sampler f(2.0, 4.0)
prior emsize sampler
scaled beta sampler f(1.1, 1.3, 256, max features)
scaled_beta. waapsaaammpp.tit
256, max_ max_features)
scaled beta aattt.ssmlle__((L. 4.0, 256, max_features)
httled_beta.sapplee
4.0, 100, 1+max features)
sampler
lambda:
2
lrreed_btaasaample __[11, 1.3, 4, 3)
scaled beta sampler f(1.1, 2.0, 5,
4 scaled beta sampler f(1.1, 4.0. 5,
acaled_beta_samee cheta_sampler_f(2... 4.0, 4,
prior.noise_std -at.mdddaaaaakkk..
range: [1.5, 5.0]
prior noise std gamma theta
range: [0.01, 1.0]
prior-sigma www.-sigaa-aamma..... proo.sigmaagamma.ttee gammaathmmattt..a theta
gamma.k
range: [1.5, 5.0]
range: [0.01, 1.0]
Table 5: Hyperparameters considered during grid search tuning of the PFN-BNN validation datasets. The activation functions refer to the activation function used in the data generating BNN and not the ones used in the PFN transformer. scaled_beta_sane bewpaasaamllerraaaa..a (tt.e.r.aaaammmmminn .mix. refers to sampling from a beta distribution with parameters b numppyrandom retaa,, scaling the output range from [0, 1] to [max, min] and then rounding the output the integer range. beta watussamlle..fe.. f(a, b) refers to function that samples from the beta distribution without scaling and rounding.
Log. Regr.
('solver': ['saga'], 'penalty': ['11", '12,, 'none'), 'tol': [0.01, 0.0001, 1e-10
"max iter": [500], fntteettr::pt:: [True, False], 'C": [1e-05, 0.001, 0.01, 0.1, 1.0, 2.0]] ('n_neighhors (max number of samples)': array([1, (1, 2, 3, 4, 5D)
KNN
BNN
('embed': [5, 10, 30, 64), 'Ir': [0.001. 0.0001), 'num training steps': [400]
'num samples for prediction": [400]}
GP
('params y scale": [0.05. 0.1, 0.5, 1.0, 5.0. 10.0], "params length scale': [0.1, 0.5. 1.0. 2.0]}
Catboost
("leariing_rate:: [0.1. 0.5, 1.0]. 'depth": 42. 7]. '12.leaf [0.0, 0.5, 1]
'iterations": [10, 40, 70], fonssffuntioon:: ['Logloss']]
XGB
'miinchhid_.ecigh'::: [0.5, 1.0], 'Icarningrrate: [0.02, 0.2), 'subsample': [0.5, 0.8]
"max depth": [1, "colsample bytrec': [0.8], 'eval metric': ['logloss'], 'n estimators': [100]]
Table 6: Hyperparameters used in cross-validation for each baseline method.
22

---

Published as a conference paper at ICLR 2022
PFN-BNN
PFN-GP
logistic
GP
KNN
BNN
Catboost
XGB
kr-vs-kp
0.928
0.897
0.920
0.886
0.754
0.843
0.653
1.000
credit-g
0.671
0.681
0.621
0,562
0.593
0.607
0.602
0.637
vehicle
0.562
0.501
0.601
0.506
0.527
0.562
0.525
0.522
wine
0.959
0.975
0.944
0.785
0.900
0.919
0.908
0.930
kel
0.614
0.695
0.730
0.695
0.666
0.724
0.708
0.747
airlines
0.916
0.896
0.913
0,873
0.834
0.907
0.982
0.985
bank-market..
0.878
0.877
0.811
0.857
0.810
0.849
0.908
0.885
blood-transfus..
0,588
0.576
0.558
0.577
0.588
0.555
0.565
0.531
phoneme
0.757
0.782
0.735
0.785
0.772
0.733
0.766
0.785
covertype
0.962
0.904
0.904
0.929
0.793
0.870
0.810
0.798
numerai28.6
0.449
0.454
0.460
0.502
0.502
0.454
0.502
0.464
connect-4
0.981
0.961
0,951
0.933
0.871
0.934
0.945
0.976
car
0.959
0.926
0.925
0.771
0.874
0.935
0.849
0.898
Australian
0.906
0.903
0.862
0,866
0.865
0.873
0.848
0.871
segment
1.000
1,000
1.000
0.999
0.985
0.999
1.000
1.000
gungle_chesse
0.774
0.780
0.693
0.852
0.747
0.708
0.841
0.708
sylvine
0,833
0.791
0.915
0.708
0.664
0.785
0.885
0.929
MiniBooNE
0,855
0,852
0.821
0.843
0.850
0.861
0.818
0.855
dionis
0.996
0.977
0.981
0.982
0.962
0.987
0.975
0.991
jannis
0.654
0.652
0.570
0.551
0.569
0.549
0.603
0.600
helena
0.911
0.892
0.839
0,870
0.842
0.882
0.870
0.905
Mean rank AUC
2.786
3.833
4.690
5.286
6.214
5.000
4.833
3.357
Win/Tic/LLos loss AUC vs T-BNN
0/21/0
14/1/6
16/1/4
17/0/4
17/1/3
17/1/3
13/1/7
12/2/7
Wilcoxon AUC vs T-BNN
0.0c+00
3.6c-13
3.6c-13
3.9c-13
3.6e-13
3.6c-13
1.9c-12
1.6c-06
GPU: 0:00:13
Benchmark Time
0:09:56
0:24:30
0:00:34
12:04:41
2:05:20
20:59:46
CPU: 0:04:23
ECE
0.025 0.067
0.157
0.095
0.093
0.089
0.157
0.066
Table 7: ROC AUC Comparison: Evaluation is made subsets of datasets with 30 training samples. 20 data-subsets are evaluated for each dataset. The ECE calibration measure. The novel PFN-BNN performs strongest overall.
23


Reviews:
[{"review_id": "rTc_zt0GRQI", "reviewer": "Reviewer_W5N8", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. The proposed framework is clear and sound, and it works well for some well known models (e.g., GP and BNN).\n\nWeaknesses:\n1. According to Eq(2) or Algorithm 1, the dataset $D$ can be drawn from a much larger dataset or distribution $p(\\mathcal{D})$. In practical, the sampled data size of $D$ should not be computationally reasonable, which means that $D$ may not represent $p(\\mathcal{D})$ very well. I'm wondering the scalability of this approach. In other works, for more stochastic case ($|D|$ is far smaller than $\\mathcal{D}$ or a much larger model with more number of parameters), how about the performance of the proposed method? In the experiment, only small datasets are discussed. \n2. In this paper, the authors emphasize that the transformer encoder can be adapted to fulfill the suggested PPD approximation. So does the title. However, I didn't see the necessity or justification to use transformer as the tool for PPD approximation. By observing Fig 2(a), I think any graph neural networks can achieve the same purpose. ", "summary_of_the_paper": "This paper presents the posterior inference framework named Prior-Data Fitted Networks. The theoretical background is to approximate the posterior predictive distribution (PPD) with a proposal distribution and optimize the KL divergence between the proposal and the PPD. The architecture used as approximation networks is adapting from a transformer encoder, by tweaking the attention mask to output the estimation of $y$ for queries alone. In addition, the authors mainly discussed two application cases, Gaussian process (GP) and Bayesian neural networks (BNN). In the experiment, the paper demonstrates the proposed method can be successful in few-shot learning.", "summary_of_the_review": "In summary, the paper is clearly written and proposes an interesting method for approximation of posterior predictive distribution. The author has validated their approach in the classification task with small datasets. But I would like to see the generalization on other tasks and data/model scalability. In addition, I have concerns or unclear intuition for adopting transformer encoder architecture.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "ciOO8uHTmJn", "reviewer": "Reviewer_NwoK", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I found the results of this paper pretty impressive, since it shows good performance across many different modalities of data and it is able to generalize well to new datasets. However, I find the main claim of the paper, that \"Bayesian inference\" is happening, unsubstantiated.\n\nIt is true that Fig. 3 shows that the posterior GP and the posterior obtained from the PFN are well matched. This proves that the Bayesian posterior is being recovered _for this particular case_. This is a particularly simple case, and similar results could be obtained simply taking the training data (which is 1D and confined to a small space) and plotting all the training trajectories weighted by how close they are to the two training points when traversing that particular x coordinate. The results would be very similar to those in Fig 3. As we get to more complicated cases, we see that the results from PFN and GPs start to diverge. In Fig 4.b, for 5 data points, we see that the MCMC solution and the PFN solution are different. The fact that PFN performs better in terms of the NLL means that it is a better model for the data, but not necessarily that it's closer to the true posterior. In fact, the MCMC solution, if run long enough, should be the golden standard.\n\nThus, the provided results show that PFNs perform very well in terms of NLL which means that they are well calibrated, but saying that they are doing Bayesian inference could be a stretch, since outside of the GP with fixed-hyperparameters we have no verification for that (in fact, it seems that either MCMC was not run to covergence, or PFN does not match Bayesian inference).\n\nMinor:\n\nTheorem 1 and the following Corollaries are trivial and that space could probably given a better use. The so-called \"Prior data NLL\" is just a standard leave-one-out NLL. The connection between maximum likelihood and minimum KL are also trivial and well-known.\n\nComparisons are missing the training time of the PFN. Although I understand that that time can be amortized, it would be good to have a sense for it in each of the experiments.\n\nHow does this process scale with the size of the dataset? Can we use it for GPs with 100.000 training samples for instance?\n\n\"we sample the N inputs xi uniformly at random from the unit-cube\"-> isn't x_i one dimensional? If so, it's not a unit \"cube\". What's the value of N?\n", "summary_of_the_paper": "In this paper the authors show a simple architecture (simple on top of transformer pieces) that can learn to perform supervised learning on new datasets given collections of existing datasets. The model is trained on a collection of supervised learning datasets, and, at test time a novel training dataset and the corresponding test inputs are given and a single forward pass through the network produces the predictive distributions for those test inputs. Several different types of data are tested with close to state-of-the-art results provided, but in much less time.", "summary_of_the_review": "Interesting paper and strong results. Probably mainly useful for fast inference in small datasets. The claim of doing Bayesian inference is not fully substantiated beyond 1D GPs.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "9yhI1qajoZ-", "reviewer": "Reviewer_ZRNF", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strong points:\n\n- The meta-learning algorithm itself, which proposes to draw datasets from synthetic priors, is an interesting and novel approach. It requires a leap of faith to attempt to transfer knowledge from synthetic datasets, such as ones generated from the BNN, to real-world datasets such as the tabular ones. It is surprising and exciting that this approach worked well and beat all the baselines.\n\n- The novel set-valued training approach which allows to feed entire datasets as samples, and the appropriately modified Transformer model. It fits very well with the meta-learning approach, and allows for fast inference.\n\n- The experiments are convincing and quantitative. In particular, the authors compare their approach to very solid baselines with appropriate tuning.\n\nWeak points:\n\nMostly, I would like to see a bit more details and explanations, as outlined in the questions and comments below:\n\nSection 2:\nStarting with ‘in this work’, this is not background anymore, this is your method. I would rework that into section 3.\n\nSection 4: \nHow do you choose the size of the datasets, N? Is this a hyperparameter of sorts? Is it better to have datasets of say, 100 samples each, or 10k samples?\n\nIn section 5.1, am I correct in my understanding that what you’re doing is:\n1- Choosing a given gaussian process.\n2- Sampling from the GP.\n3- Grouping the samples into buckets of size N: these are the datasets used for training your PFN.\n4- Training the PFN on the datasets, and in figure 4)a) you vary the number of datasets you use (from 500K to 4M)\n5- Using that for inference on new, unseen datasets generated from the GP (1000 datasets, according to appendix F). \n\nIf this is correct, then I don’t understand how this is different from traditional supervised learning, in which you directly train on all the datapoints without grouping into datasets. Indeed, I understand the point of meta-learning as being that the datasets are coming from different data generating processes. Section 5.2 and 5.3 make sense to me, as the data generating process changes between sample datasets due to the sampling of GP parameters.\n\nI don’t understand what does ‘number of data points’ refer to in Figure 4? Is the size of the dataset, N? \n\nSection 5.3: \n\nIn figure 5, you show the performance of a PFN with 4M datasets. However, in the text, you mention sampling K=100 BNNs with m=200 and n=100. Are those just the evaluation datasets?  \n\nSection 6: \n\nAm I correct in my understanding that, for this section, the data priors for the PFN are just the synthetic datasets generated in sections 5.2 and 5.3 (plus the architecture prior)? As in, there is no PFN training on the tabular datasets, only bayesian inference? \n\nIf it is the case, that is very impressive, and I wonder why does it work so well? In particular, why does the scale and nature of features (e.g. categorical) not impact the appropriateness of the data priors you used? Why did you not fine-tune on the tabular datasets, just like you did in section 7 for Omniglot? I’m surprised because seemingly the tasks in the tabular datasets are very different from the GP or BNN priors, since the data generating process is intuitively very different.\n", "summary_of_the_paper": "This paper describes a novel way to do bayesian inference with deep learning models by employing a meta-learning approach. This is done by creating a synthetic prior over datasets, training a deep learning model on samples from this prior, which results in an approximation of the posterior predictive distribution over the datasets. The authors demonstrate their approach on two synthetic tasks: approximating gaussian processes and bayesian neural networks, then on two real world tasks: the OpenML tabular datasets and the few-shot learning Omniglot dataset.", "summary_of_the_review": "Overall, I vote for accepting. I think the approach is very novel and the results surprisingly good – especially for the tabular data with BNN priors. This method has a lot of potential for bridging the gap between deep learning models and small data regimes, all the while adopting a Bayesian perspective. It also opens the door to very interesting future research, for instance on the question of design of priors for a particular problem. \nMy main concern is with regards to the clarity of the paper, as there is a lot of prior knowledge assumed from the reader and sometimes a lack of details and explanation. Hopefully my comments will be addressed during the rebuttal period.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a method for using transformer models to perform approximate Bayesian inference, in the sense of approximating the posterior predictive distribution for a test example.  This seems similar to doing amortized variational inference using a transformer model.  The reviewers all found the paper to be clearly written, interesting, novel and compelling.  Two of the reviewers found the results "impressive".  There is some concern of over-claiming (is it really Bayesian?, are the authors making too broad statements based on very simple case studies?).  The presented method is also not scalable O(n^2), so the setting is restricted to very small datasets and models. 
 However, the reviewers didn't seem especially concerned by this.  The reviews were mixed but leaning positive (8, 6, 5) and the positive reviews are more substantial.  Therefore the recommendation is to accept, but please incorporate the reviewer feedback and additional discussion about related methods (discussion below) into the camera ready.
Decision: Accept (Poster)